.gov websites and AI crawlers

Of 462 readable .gov sites in the census, 6.1% disallow GPTBot and 6.3% disallow ClaudeBot.

SitePopularityrobots.txtRead on
nih.govTop 1,000none disallowed2026-10-10
www.ncbi.nlm.nih.govTop 1,000none disallowed2026-10-10
pubmed.ncbi.nlm.nih.govTop 1,000none disallowed2026-10-10
cdc.govTop 1,000none disallowed2026-10-10
nasa.govTop 1,000none disallowed2026-10-10
loc.govTop 1,000none disallowed2026-10-10
ca.govTop 1,000none disallowed2026-10-10
fda.govTop 1,000no robots.txt rules2026-10-10
nist.govTop 1,000none disallowed2026-10-10
epa.govTop 1,000none disallowed2026-10-10
whitehouse.govTop 1,000none disallowed2026-10-10
ftc.govTop 1,000none disallowed2026-10-10
irs.govTop 1,000none disallowed2026-10-10
nps.govTop 1,000none disallowed2026-10-10
sec.govTop 1,000none disallowed2026-10-10
justice.govTop 1,000none disallowed2026-10-10
house.govTop 1,000none disallowed2026-10-10
census.govTop 1,000none disallowed2026-10-10
energy.govTop 1,000none disallowed2026-10-10
congress.govTop 1,00019 disallowed2026-10-10
ny.govTop 1,000none disallowed2026-10-10
archives.govTop 1,000none disallowed2026-10-10
medlineplus.govTop 1,000none disallowed2026-10-10
fbi.govTop 1,000none disallowed2026-10-10
uspto.govTop 1,000none disallowed2026-10-10
va.govTop 1,0001 disallowed2026-10-10
cancer.govTop 1,000none disallowed2026-10-10
wa.govTop 1,00022 disallowed2026-10-10
dataprivacyframework.govTop 1,000none disallowed2026-10-10
texas.govTop 1,000none disallowed2026-10-10
usa.govTop 1,000none disallowed2026-10-10
dol.govTop 1,000none disallowed2026-10-10
federalregister.govTop 1,000none disallowed2026-10-10
nyc.govTop 1,000none disallowed2026-10-10
weather.govTop 1,000no robots.txt rules2026-10-10
clinicaltrials.govTop 1,000none disallowed2026-10-10
cia.govTop 1,0001 disallowed2026-10-10
federalreserve.govTop 10,000no robots.txt rules2026-10-10
sba.govTop 10,000none disallowed2026-10-10
eia.govTop 10,000none disallowed2026-10-10
uscourts.govTop 10,000none disallowed2026-10-10
nsf.govTop 10,000none disallowed2026-10-10
cms.govTop 10,000none disallowed2026-10-10
govinfo.govTop 10,000none disallowed2026-10-10
copyright.govTop 10,000no robots.txt rules2026-10-10
virginia.govTop 10,000none disallowed2026-10-10
nj.govTop 10,000none disallowed2026-10-10
ecfr.govTop 10,000none disallowed2026-10-10
gao.govTop 10,0002 disallowed2026-10-10
in.govTop 10,000none disallowed2026-10-10
samhsa.govTop 10,000none disallowed2026-10-10
supremecourt.govTop 10,000none disallowed2026-10-10
ct.govTop 10,000none disallowed2026-10-10
pa.govTop 10,000none disallowed2026-10-10
lbl.govTop 10,000none disallowed2026-10-10
consumerfinance.govTop 10,000none disallowed2026-10-10
maryland.govTop 10,000none disallowed2026-10-10
colorado.govTop 10,000none disallowed2026-10-10
gpo.govTop 10,000none disallowed2026-10-10
hud.govTop 10,000none disallowed2026-10-10
fws.govTop 10,000none disallowed2026-10-10
oregon.govTop 10,000none disallowed2026-10-10
ohio.govTop 10,000no robots.txt rules2026-10-10
mo.govTop 10,000none disallowed2026-10-10
cpsc.govTop 10,000none disallowed2026-10-10
www.fs.usda.govTop 10,000none disallowed2026-10-10
energystar.govTop 10,0001 disallowed2026-10-10
illinois.govTop 10,0003 disallowed2026-10-10
maine.govTop 10,000none disallowed2026-10-10
ic3.govTop 10,000none disallowed2026-10-10
hawaii.govTop 10,000none disallowed2026-10-10
medicare.govTop 10,000none disallowed2026-10-10
tn.govTop 10,000none disallowed2026-10-10
eeoc.govTop 10,000none disallowed2026-10-10
mn.govTop 10,000none disallowed2026-10-10
ky.govTop 10,000no robots.txt rules2026-10-10
gsa.govTop 10,000none disallowed2026-10-10
alaska.govTop 10,000none disallowed2026-10-10
dc.govTop 10,000none disallowed2026-10-10
ojp.govTop 10,000none disallowed2026-10-10
fdic.govTop 10,000none disallowed2026-10-10
delaware.govTop 10,000none disallowed2026-10-10
health.govTop 10,000none disallowed2026-10-10
healthcare.govTop 10,000none disallowed2026-10-10
trade.govTop 10,000none disallowed2026-10-10
idaho.govTop 10,000none disallowed2026-10-10
ornl.govTop 10,000none disallowed2026-10-10
georgia.govTop 10,000none disallowed2026-10-10
nv.govTop 10,00022 disallowed2026-10-10
doe.govTop 10,000none disallowed2026-10-10
ustr.govTop 10,000none disallowed2026-10-10
mt.govTop 10,000no robots.txt rules2026-10-10
iowa.govTop 10,000none disallowed2026-10-10
ahrq.govTop 10,0003 disallowed2026-10-10
lanl.govTop 10,000none disallowed2026-10-10
sc.govTop 10,000no robots.txt rules2026-10-10
llnl.govTop 10,000none disallowed2026-10-10
bea.govTop 10,000none disallowed2026-10-10
vermont.govTop 10,00010 disallowed2026-10-10
blm.govTop 10,000none disallowed2026-10-10
ri.govTop 10,000none disallowed2026-10-10
lacounty.govTop 10,000none disallowed2026-10-10
nd.govTop 10,000none disallowed2026-10-10
genome.govTop 10,000none disallowed2026-10-10
drugabuse.govTop 10,000none disallowed2026-10-10
nc.govTop 10,000none disallowed2026-10-10
osti.govTop 10,000none disallowed2026-10-10
la.govTop 10,000no robots.txt rules2026-10-10
nysenate.govTop 10,000none disallowed2026-10-10
ga.govTop 10,000none disallowed2026-10-10
opm.govTop 10,000none disallowed2026-10-10
nebraska.govTop 10,000none disallowed2026-10-10
seattle.govTop 10,000none disallowed2026-10-10
sd.govTop 10,000no robots.txt rules2026-10-10
doi.govTop 10,000none disallowed2026-10-10
ms.govTop 10,000none disallowed2026-10-10
dni.govTop 10,000none disallowed2026-10-10
cftc.govTop 10,00011 disallowed2026-10-10
louisiana.govTop 10,000no robots.txt rules2026-10-10
climate.govTop 10,000none disallowed2026-10-10
womenshealth.govTop 10,000none disallowed2026-10-10
sam.govTop 10,000none disallowed2026-10-10
flsenate.govTop 10,000none disallowed2026-10-10
ok.govTop 10,000none disallowed2026-10-10
recreation.govTop 10,000none disallowed2026-10-10
usajobs.govTop 10,000none disallowed2026-10-10
sandia.govTop 10,000none disallowed2026-10-10
ntsb.govTop 10,000none disallowed2026-10-10
grants.govTop 10,000none disallowed2026-10-10
medicaid.govTop 10,000none disallowed2026-10-10
oklahoma.govTop 10,000none disallowed2026-10-10
arts.govTop 10,0002 disallowed2026-10-10
nlrb.govTop 10,000none disallowed2026-10-10
ne.govTop 10,000none disallowed2026-10-10
miamidade.govTop 10,000no robots.txt rules2026-10-10
neh.govTop 10,000none disallowed2026-10-10
austintexas.govTop 10,000none disallowed2026-10-10
texasattorneygeneral.govTop 10,0006 disallowed2026-10-10
boston.govTop 10,000none disallowed2026-10-10
acl.govTop 10,000none disallowed2026-10-10
digital.govTop 10,000none disallowed2026-10-10
odni.govTop 10,000none disallowed2026-10-10
floridahealth.govTop 10,000none disallowed2026-10-10
phila.govTop 10,000no robots.txt rules2026-10-10
kingcounty.govTop 10,000none disallowed2026-10-10
data.govTop 10,000none disallowed2026-10-10
malegislature.govTop 10,000none disallowed2026-10-10
sandiego.govTop 10,000none disallowed2026-10-10
azleg.govTop 10,000none disallowed2026-10-10
healthit.govTop 10,000none disallowed2026-10-10
nm.govTop 10,000none disallowed2026-10-10
sf.govTop 10,000none disallowed2026-10-10
txdot.govTop 10,000no robots.txt rules2026-10-10
treasurydirect.govTop 10,00022 disallowed2026-10-10
phoenix.govTop 50,000none disallowed2026-10-10
montgomerycountymd.govTop 50,00011 disallowed2026-10-10
america.govTop 50,00015 disallowed2026-10-10
peacecorps.govTop 50,000none disallowed2026-10-10
cabq.govTop 50,000none disallowed2026-10-10
fhfa.govTop 50,000none disallowed2026-10-10
fdacs.govTop 50,00020 disallowed2026-10-10
azdhs.govTop 50,000none disallowed2026-10-10
baltimorecity.govTop 50,0003 disallowed2026-10-10
longbeach.govTop 50,0004 disallowed2026-10-10
sanantonio.govTop 50,00020 disallowed2026-10-10
detroitmi.govTop 50,000none disallowed2026-10-10
lasvegasnevada.govTop 50,000none disallowed2026-10-10
nola.govTop 50,000none disallowed2026-10-10
njoag.govTop 50,000none disallowed2026-10-10
codot.govTop 50,000none disallowed2026-10-10
nyassembly.govTop 50,0001 disallowed2026-10-10
txcourts.govTop 50,000none disallowed2026-10-10
pacodeandbulletin.govTop 50,00020 disallowed2026-10-10
nmlegis.govTop 50,000none disallowed2026-10-10
coloradosprings.govTop 50,00020 disallowed2026-10-10
harriscountytx.govTop 50,000none disallowed2026-10-10
azcourts.govTop 50,000none disallowed2026-10-10
oklegislature.govTop 50,00020 disallowed2026-10-10
itis.govTop 50,000none disallowed2026-10-10
suffolkcountyny.govTop 50,00011 disallowed2026-10-10
legmt.govTop 50,000none disallowed2026-10-10
flcourts.govTop 50,00020 disallowed2026-10-10
illinoiscourts.govTop 50,000none disallowed2026-10-10
marylandtaxes.govTop 50,000none disallowed2026-10-10
lexingtonky.govTop 50,00011 disallowed2026-10-10
oregonmetro.govTop 50,00011 disallowed2026-10-10
dfc.govTop 100,0001 disallowed2026-10-10
presidio.govTop 100,000none disallowed2026-10-10
oversight.govTop 100,0009 disallowed2026-10-10
marylandcomptroller.govTop 100,000none disallowed2026-10-10
sdsos.govTop 100,000none disallowed2026-10-10
fgdc.govTop 100,00020 disallowed2026-10-10
usagm.govTop 100,0004 disallowed2026-10-10
cuyahogacounty.govTop 100,000none disallowed2026-10-10
navajo-nsn.govTop 100,000none disallowed2026-10-10
chattanooga.govTop 100,000none disallowed2026-10-10
birminghamal.govTop 100,0009 disallowed2026-10-10
santafenm.govTop 100,0001 disallowed2026-10-10