.ai websites and AI crawlers

Of 309 readable .ai sites in the census, 5.5% disallow GPTBot and 3.9% disallow ClaudeBot.

SitePopularityrobots.txtRead on
claude.aiTop 1,0004 disallowed2026-10-10
perplexity.aiTop 1,000none disallowed2026-10-10
x.aiTop 10,000none disallowed2026-10-10
mistral.aiTop 10,000none disallowed2026-10-10
beacons.aiTop 10,000none disallowed2026-10-10
jasper.aiTop 10,000none disallowed2026-10-10
openclaw.aiTop 10,000none disallowed2026-10-10
openrouter.aiTop 10,000none disallowed2026-10-10
stability.aiTop 10,000none disallowed2026-10-10
otter.aiTop 10,000none disallowed2026-10-10
copy.aiTop 10,000none disallowed2026-10-10
dropcatch.aiTop 10,000none disallowed2026-10-10
leonardo.aiTop 10,000none disallowed2026-10-10
devin.aiTop 10,000none disallowed2026-10-10
google.com.aiTop 10,000none disallowed2026-10-10
artificialanalysis.aiTop 10,000none disallowed2026-10-10
meta.aiTop 10,00020 disallowed2026-10-10
z.aiTop 10,000none disallowed2026-10-10
character.aiTop 10,000none disallowed2026-10-10
deeplearning.aiTop 10,0006 disallowed2026-10-10
lmstudio.aiTop 10,000none disallowed2026-10-10
lumalabs.aiTop 10,000none disallowed2026-10-10
ideogram.aiTop 50,000none disallowed2026-10-10
kling.aiTop 50,000none disallowed2026-10-10
pictory.aiTop 50,000none disallowed2026-10-10
wandb.aiTop 50,000none disallowed2026-10-10
data.aiTop 50,000none disallowed2026-10-10
digital.aiTop 50,000none disallowed2026-10-10
together.aiTop 50,0001 disallowed2026-10-10
higgsfield.aiTop 50,000none disallowed2026-10-10
techmonitor.aiTop 50,0001 disallowed2026-10-10
resemble.aiTop 50,0003 disallowed2026-10-10
meshy.aiTop 50,0003 disallowed2026-10-10
scite.aiTop 50,0002 disallowed2026-10-10
notta.aiTop 50,000none disallowed2026-10-10
qodo.aiTop 50,000none disallowed2026-10-10
litellm.aiTop 50,000none disallowed2026-10-10
nansen.aiTop 50,000none disallowed2026-10-10
unstoppable.aiTop 50,000none disallowed2026-10-10
magichour.aiTop 50,0002 disallowed2026-10-10
soloist.aiTop 50,000none disallowed2026-10-10
tripo3d.aiTop 50,0003 disallowed2026-10-10
podcastle.aiTop 50,000none disallowed2026-10-10
fathom.aiTop 50,000none disallowed2026-10-10
plaud.aiTop 50,000none disallowed2026-10-10
dream.aiTop 50,000none disallowed2026-10-10
fin.aiTop 50,000none disallowed2026-10-10
sonix.aiTop 50,000none disallowed2026-10-10
autonomous.aiTop 50,0001 disallowed2026-10-10
energymonitor.aiTop 50,0001 disallowed2026-10-10
masto.aiTop 50,0001 disallowed2026-10-10
inworld.aiTop 100,000none disallowed2026-10-10
sapling.aiTop 100,000none disallowed2026-10-10
muse.aiTop 100,00014 disallowed2026-10-10
techbuzz.aiTop 100,00010 disallowed2026-10-10
placer.aiTop 100,0005 disallowed2026-10-10
smartlead.aiTop 100,000none disallowed2026-10-10
mindtrip.aiTop 100,0005 disallowed2026-10-10
eesel.aiTop 100,000none disallowed2026-10-10
hypotenuse.aiTop 100,0001 disallowed2026-10-10
jerry.aiTop 100,000none disallowed2026-10-10
vellum.aiTop 100,000none disallowed2026-10-10
bardeen.aiTop 100,0008 disallowed2026-10-10
investmentmonitor.aiTop 100,0001 disallowed2026-10-10
pippit.aiTop 100,000none disallowed2026-10-10
treasure.aiTop 100,000none disallowed2026-10-10
thebrief.aiTop 100,000none disallowed2026-10-10
copysmith.aiTop 100,0009 disallowed2026-10-10
sambanova.aiTop 100,000none disallowed2026-10-10
codium.aiTop 100,000none disallowed2026-10-10
glama.aiTop 100,000none disallowed2026-10-10
quso.aiTop 100,000none disallowed2026-10-10