Which websites block AI crawlers?
We read the robots.txt of the most popular websites and worked out, for each one, which of 22 AI and search crawlers it asks to stay away: GPTBot, ClaudeBot, Google-Extended, CCBot, OAI-SearchBot, PerplexityBot and more. The census covers 242,313 sites whose file we could read (last full read 2026-11-02); 18,500 of them have a page of their own here because their file says something distinctive, and every change is kept.
Named and disallowed counts a site whose file has a rule for that crawler. In effect also counts a site that disallows the crawler only because its catch-all rule (User-agent: * with Disallow: /) turns away every crawler the file does not name. That is 72.6% of the sites that disallow GPTBot here, 94.3% of it beyond the top 100,000: use the first number for AI-specific policy. See the bands below and the method.
Percentages are of the 216,442 sites with a robots.txt that was read; 25,871 more have no rules at all and 57,686 could not be read reliably and are not described anywhere. This reports what the file asks. It does not show whether a crawler can actually reach the site: many sites also block crawlers at their CDN or firewall, and some crawlers ignore robots.txt.
Look up a site
Named and disallowed, by popularity band
Percent of the sites with rules in each band (not cumulative). The last column is the share of the GPTBot disallows in the band that come only from a catch-all rule.
| Popularity band (Majestic rank) | Sites with rules | GPTBot | ClaudeBot | Google-Extended | CCBot | OAI-SearchBot | PerplexityBot | Googlebot | GPTBot in effect | GPTBot: only a catch-all rule |
|---|---|---|---|---|---|---|---|---|---|---|
| Top 1,000 | 798 | 19.7% | 18.8% | 16.4% | 21.4% | 7.9% | 14.8% | 0.3% | 23.3% | 15.6% |
| 1,001 to 10,000 | 6,771 | 16.7% | 15.2% | 13.6% | 17.4% | 5.0% | 8.7% | 0.1% | 18.4% | 9.5% |
| 10,001 to 100,000 | 63,089 | 13.3% | 11.3% | 10.5% | 13.5% | 2.6% | 4.1% | 0.3% | 18.2% | 26.6% |
| 100,001 to 300,000 | 145,784 | 7.9% | 6.6% | 6.3% | 7.8% | 0.9% | 1.5% | 0.1% | 44.5% | 82.1% |
One file layout is the largest single source of those catch-all disallows: 53,477 sites serve a robots.txt with the same crawler lines (it names Googlebot, Bingbot, PetalBot, CCBot and Bytespider) whose catch-all rule disallows every crawler it does not name. 51,598 of them are past the top 100,000, 35.4% of the sites with rules there. Without them GPTBot is named and disallowed by 12.3% and disallowed in effect by 14.1% of the other 94,186 sites there.
Disallow rates by popularity (in effect, cumulative)
| Popularity | Readable sites with rules | GPTBot | ClaudeBot | Google-Extended | CCBot | OAI-SearchBot | PerplexityBot | Googlebot |
|---|---|---|---|---|---|---|---|---|
| Top 1,000 | 798 | 23.3% | 23.2% | 20.1% | 26.8% | 12.2% | 18.8% | 2.0% |
| Top 10,000 | 7,569 | 19.0% | 17.8% | 16.0% | 20.3% | 7.4% | 11.5% | 1.2% |
| Top 50,000 | 35,397 | 17.2% | 15.3% | 14.1% | 17.4% | 5.8% | 8.4% | 1.3% |
| Top 100,000 | 70,658 | 18.2% | 16.3% | 15.4% | 16.0% | 7.4% | 9.3% | 1.4% |
| Top 250,000 | 179,335 | 33.8% | 32.2% | 31.6% | 12.0% | 25.2% | 26.2% | 1.1% |
| Top 300,000 | 216,442 | 35.9% | 34.4% | 33.9% | 11.3% | 27.7% | 28.6% | 1.0% |
Most disallowed crawlers
- CCBot (Common Crawl, training): named and disallowed by 9.8% of sites with rules, disallowed in effect by 11.3%
- GPTBot (OpenAI, training): named and disallowed by 9.8% of sites with rules, disallowed in effect by 35.9%
- Bytespider (ByteDance, training): named and disallowed by 8.5% of sites with rules, disallowed in effect by 10.0%
- ClaudeBot (Anthropic, training): named and disallowed by 8.3% of sites with rules, disallowed in effect by 34.4%
- Google-Extended (Google, training): named and disallowed by 7.8% of sites with rules, disallowed in effect by 33.9%
- meta-externalagent (Meta, training): named and disallowed by 7.7% of sites with rules, disallowed in effect by 33.9%
- Amazonbot (Amazon, training): named and disallowed by 7.4% of sites with rules, disallowed in effect by 33.6%
- Applebot-Extended (Apple, training): named and disallowed by 7.3% of sites with rules, disallowed in effect by 33.5%
What the jobs mean
- Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and others): disallowing keeps content out of future model training.
- Search and citation crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot): disallowing can remove a site from AI answers and the traffic that follows.
- User-action fetchers (ChatGPT-User, Perplexity-User, Claude-User): disallowing stops an assistant reading the page when a person asks.
Browse
GPTBot ClaudeBot Google-Extended CCBot meta-externalagent Bytespider Applebot-Extended Amazonbot cohere-ai Diffbot omgili Timpibot OAI-SearchBot PerplexityBot Claude-SearchBot DuckAssistBot ChatGPT-User Perplexity-User Claude-User meta-externalfetcher Googlebot Bingbot