jotform.us: AI crawler policy in robots.txt

Q: Does jotform.us block GPTBot?

Yes. robots.txt as fetched on 2026-10-10: jotform.us disallows 22 of 22 AI and search crawlers from the site root: GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent, Bytespider, Applebot-Extended, Amazonbot, cohere-ai, Diffbot, omgili, Timpibot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher, Googlebot, Bingbot. That includes crawlers behind AI answers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher). It also disallows Googlebot, Bingbot, classic search crawlers.

robots.txt as fetched on 2026-10-10: jotform.us disallows 22 of 22 AI and search crawlers from the site root: GPTBot, ClaudeBot, Google-Extended, CCBot, meta-externalagent, Bytespider, Applebot-Extended, Amazonbot, cohere-ai, Diffbot, omgili, Timpibot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher, Googlebot, Bingbot. That includes crawlers behind AI answers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, DuckAssistBot, ChatGPT-User, Perplexity-User, Claude-User, meta-externalfetcher). It also disallows Googlebot, Bingbot, classic search crawlers.

The file refuses training crawlers and also some crawlers that power AI answers, so the site is likely to be missing from AI citations and the referral traffic that follows.

This reports what the file asks. It does not show whether a crawler can actually reach the site: many sites also block crawlers at their CDN or firewall, and some crawlers ignore robots.txt.

Training crawlers

Disallowing these keeps a site's content out of future model training. It does not remove what was already collected.

Crawler	Operator	Status	How the file treats it
GPTBot	OpenAI	disallowed	named in the file
ClaudeBot	Anthropic	disallowed	named in the file
Google-Extended	Google	disallowed	covered by the default rule
CCBot	Common Crawl	disallowed	named in the file
meta-externalagent	Meta	disallowed	named in the file
Bytespider	ByteDance	disallowed	named in the file
Applebot-Extended	Apple	disallowed	named in the file
Amazonbot	Amazon	disallowed	covered by the default rule
cohere-ai	Cohere	disallowed	named in the file
Diffbot	Diffbot	disallowed	named in the file
omgili	Webz.io	disallowed	covered by the default rule
Timpibot	Timpi	disallowed	covered by the default rule

Search and citation crawlers

These feed AI answer engines that cite and link their sources. Disallowing them can take a site out of AI answers and out of the referral traffic that comes with them.

Crawler	Operator	Status	How the file treats it
OAI-SearchBot	OpenAI	disallowed	named in the file
PerplexityBot	Perplexity	disallowed	named in the file
Claude-SearchBot	Anthropic	disallowed	covered by the default rule
DuckAssistBot	DuckDuckGo	disallowed	covered by the default rule

User-action fetchers

These fetch a page live when a person asks an assistant about it. Disallowing them stops the assistant reading the page for that person.

Crawler	Operator	Status	How the file treats it
ChatGPT-User	OpenAI	disallowed	covered by the default rule
Perplexity-User	Perplexity	disallowed	covered by the default rule
Claude-User	Anthropic	disallowed	covered by the default rule
meta-externalfetcher	Meta	disallowed	named in the file

Classic search crawlers

Classic search crawlers, shown as the baseline.

Crawler	Operator	Status	How the file treats it
Googlebot	Google	disallowed	covered by the default rule
Bingbot	Microsoft	disallowed	named in the file

How jotform.us compares

Among 65,831 readable sites in the Top 100,000, 12.9% disallow GPTBot, 10.8% ClaudeBot, 6.0% OAI-SearchBot and 7.9% PerplexityBot. Among 95 readable .us sites: 8.4% disallow GPTBot, 8.4% ClaudeBot.

History

First read 2026-10-10; the file has not changed since. Read the file itself.

The AI-related lines in the file

User-agent: Bingbot
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Applebot-Extended
User-agent: PerplexityBot
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: anthropic-ai
User-agent: cohere-ai
User-agent: Diffbot
User-agent: Bytespider
User-agent: OAI-SearchBot

Check it live. This page shows the file as read on 2026-10-10. Run a live check of jotform.us and get told when it changes, or get the dataset for a segment.