AI Crawler Atlas / domains
exxpress.at: AI crawler policy in robots.txt
robots.txt as fetched on 2026-10-10: exxpress.at disallows 5 of 22 AI and search crawlers from the site root: GPTBot, ClaudeBot, CCBot, Bytespider, cohere-ai. It leaves the search and citation crawlers and the user-action fetchers open.
A protective stance that stays visible to AI answers: the file refuses training crawlers and leaves open the crawlers that let answer engines cite and link the site.
This reports what the file asks. It does not show whether a crawler can actually reach the site: many sites also block crawlers at their CDN or firewall, and some crawlers ignore robots.txt.
Training crawlers
Disallowing these keeps a site's content out of future model training. It does not remove what was already collected.
| Crawler | Operator | Status | How the file treats it |
|---|---|---|---|
| GPTBot | OpenAI | disallowed | named in the file |
| ClaudeBot | Anthropic | disallowed | named in the file |
| Google-Extended | not disallowed | named in the file | |
| CCBot | Common Crawl | disallowed | named in the file |
| meta-externalagent | Meta | not disallowed | named in the file |
| Bytespider | ByteDance | disallowed | named in the file |
| Applebot-Extended | Apple | not disallowed | named in the file |
| Amazonbot | Amazon | not disallowed | covered by the default rule |
| cohere-ai | Cohere | disallowed | named in the file |
| Diffbot | Diffbot | not disallowed | covered by the default rule |
| omgili | Webz.io | not disallowed | covered by the default rule |
| Timpibot | Timpi | not disallowed | covered by the default rule |
Search and citation crawlers
These feed AI answer engines that cite and link their sources. Disallowing them can take a site out of AI answers and out of the referral traffic that comes with them.
| Crawler | Operator | Status | How the file treats it |
|---|---|---|---|
| OAI-SearchBot | OpenAI | not disallowed | named in the file |
| PerplexityBot | Perplexity | not disallowed | named in the file |
| Claude-SearchBot | Anthropic | not disallowed | named in the file |
| DuckAssistBot | DuckDuckGo | not disallowed | covered by the default rule |
User-action fetchers
These fetch a page live when a person asks an assistant about it. Disallowing them stops the assistant reading the page for that person.
| Crawler | Operator | Status | How the file treats it |
|---|---|---|---|
| ChatGPT-User | OpenAI | not disallowed | named in the file |
| Perplexity-User | Perplexity | not disallowed | named in the file |
| Claude-User | Anthropic | not disallowed | named in the file |
| meta-externalfetcher | Meta | not disallowed | covered by the default rule |
Classic search crawlers
Classic search crawlers, shown as the baseline.
| Crawler | Operator | Status | How the file treats it |
|---|---|---|---|
| Googlebot | not disallowed | covered by the default rule | |
| Bingbot | Microsoft | not disallowed | covered by the default rule |
How exxpress.at compares
Among 65,831 readable sites in the Top 100,000, 12.9% disallow GPTBot, 10.8% ClaudeBot, 6.0% OAI-SearchBot and 7.9% PerplexityBot. Among 236 readable .at sites: 11.0% disallow GPTBot, 8.9% ClaudeBot.
History
First read 2026-10-10; the file has not changed since. Read the file itself.
The AI-related lines in the file
# Googlebot's renderer saw empty skeletons (GSC live test 2026-06-11: 8/137
User-agent: Googlebot-News
User-agent: ChatGPT-User
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Perplexity-User
# Cloudflare beantwortet Anfragen von GPTBot, ClaudeBot und cohere-ai bereits
# durchgelassen: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User,
# PerplexityBot und Perplexity-User antworten alle mit 200. Zitate mit Quelle
# Entscheidung waere: Google-Extended, Meta-ExternalAgent, Applebot-Extended.
# Nachpruefen: curl -s -o /dev/null -w '%{http_code}' -A 'GPTBot/1.0' https://exxpress.at/
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Claude-SearCheck it live. This page shows the file as read on 2026-10-10. Run a live check of exxpress.at and get told when it changes, or get the dataset for a segment.