.in websites and AI crawlers

Of 229 readable .in sites in the census, 12.7% disallow GPTBot and 9.6% disallow ClaudeBot.

SitePopularityrobots.txtRead on
lnkd.inTop 1,00018 disallowed2026-10-10
google.co.inTop 10,000none disallowed2026-10-10
amazon.inTop 10,00018 disallowed2026-10-10
indiatoday.inTop 10,000none disallowed2026-10-10
blogspot.inTop 10,000none disallowed2026-10-10
teletype.inTop 10,000none disallowed2026-10-10
businesstoday.inTop 10,000none disallowed2026-10-10
tripadvisor.inTop 10,0009 disallowed2026-10-10
theprint.inTop 10,0002 disallowed2026-10-10
rbi.org.inTop 10,000no robots.txt rules2026-10-10
allevents.inTop 10,000none disallowed2026-10-10
thewire.inTop 10,000none disallowed2026-10-10
thebase.inTop 10,000none disallowed2026-10-10
scroll.inTop 10,000none disallowed2026-10-10
theweek.inTop 10,0002 disallowed2026-10-10
garv.inTop 10,000none disallowed2026-10-10
mea.gov.inTop 10,000none disallowed2026-10-10
freepressjournal.inTop 10,000none disallowed2026-10-10
downtoearth.org.inTop 10,000none disallowed2026-10-10
maharashtra.gov.inTop 10,000none disallowed2026-10-10
zapf.inTop 10,000no robots.txt rules2026-10-10
vogue.inTop 50,00016 disallowed2026-10-10
espn.inTop 50,0006 disallowed2026-10-10
ibtimes.co.inTop 50,0004 disallowed2026-10-10
glassdoor.co.inTop 50,00010 disallowed2026-10-10
airtel.inTop 50,000none disallowed2026-10-10
livelaw.inTop 50,000none disallowed2026-10-10
npci.org.inTop 50,00011 disallowed2026-10-10
huffingtonpost.inTop 50,00019 disallowed2026-10-10
airbnb.co.inTop 50,0003 disallowed2026-10-10
fyers.inTop 50,000none disallowed2026-10-10
galidesawar.inTop 50,0001 disallowed2026-10-10
plinko.inTop 50,0001 disallowed2026-10-10
aajtak.inTop 50,000none disallowed2026-10-10
canttboardjabalpur.org.inTop 50,0001 disallowed2026-10-10
gettyimages.inTop 50,000none disallowed2026-10-10
architecturaldigest.inTop 50,00016 disallowed2026-10-10
paidforarticles.inTop 50,000none disallowed2026-10-10
shortcutkeys.inTop 50,0008 disallowed2026-10-10
bits-pilani.ac.inTop 50,000none disallowed2026-10-10
iiit.ac.inTop 50,000none disallowed2026-10-10
cntraveller.inTop 50,00016 disallowed2026-10-10
goodreturns.inTop 50,000none disallowed2026-10-10
taxguru.inTop 50,000none disallowed2026-10-10
peoplematters.inTop 50,000none disallowed2026-10-10
irdai.gov.inTop 50,00022 disallowed2026-10-10
fita.inTop 50,000none disallowed2026-10-10
linkd.inTop 50,00018 disallowed2026-10-10
robotickneereplacements.inTop 50,0001 disallowed2026-10-10
shaktipay.net.inTop 50,0008 disallowed2026-10-10
newgujaratisong.inTop 50,0001 disallowed2026-10-10
riddleroom.inTop 50,0001 disallowed2026-10-10
nsdl.co.inTop 50,0001 disallowed2026-10-10
adesh.inTop 100,0001 disallowed2026-10-10
livescore.inTop 100,0006 disallowed2026-10-10
brainly.inTop 100,0002 disallowed2026-10-10
payu.inTop 100,0001 disallowed2026-10-10
femina.inTop 100,0002 disallowed2026-10-10
kylejasm.inTop 100,0001 disallowed2026-10-10
lilfeet.inTop 100,0001 disallowed2026-10-10
vit.ac.inTop 100,0001 disallowed2026-10-10
hostinger.inTop 100,0002 disallowed2026-10-10
bitcoinworld.co.inTop 100,000none disallowed2026-10-10
krisumicity.co.inTop 100,0003 disallowed2026-10-10
wapking.inTop 100,0001 disallowed2026-10-10
expedia.co.inTop 100,000none disallowed2026-10-10
shiprocket.inTop 100,000none disallowed2026-10-10
millenniumpost.inTop 100,000none disallowed2026-10-10
besonline.inTop 100,0003 disallowed2026-10-10
autocarpro.inTop 100,000none disallowed2026-10-10
weddingwire.inTop 100,0001 disallowed2026-10-10
grabon.inTop 100,000none disallowed2026-10-10
myra.ac.inTop 100,000none disallowed2026-10-10
thelittleluxurystore.inTop 100,0001 disallowed2026-10-10
chitkara.edu.inTop 100,0004 disallowed2026-10-10