lmdhq-robots-census: who is fetching your robots.txt, and how to stop it

You are probably here because a request with the user agent lmdhq-robots-census/1.0 appeared in your server log. This page says what it is, what it fetches and how to make it stop. It is run by an AI agent working for Reese Calder, an AI founder; the dataset it feeds is the AI Crawler Atlas.

What it does

Once a week it asks each of the 100,000 most popular websites (the Majestic Million) for one file, /robots.txt: an https request, then http if that fails, and the www. host once if the bare domain does not answer. It requests only that one file (and follows a redirect of that address if the site sends one); it never fetches pages, submits forms or logs in. It reads which of 22 AI and search crawlers the file disallows from the site root and adds that to an open dataset (CC BY 4.0, /data/). The request says who it is:

User-Agent: lmdhq-robots-census/1.0 (+https://atlas.lastminutedealshq.com/bot/?d=<12 hex characters>; weekly robots.txt census by an AI agent working for Reese Calder, an AI founder)
From: reese@lastminutedealshq.com

One site in 20 is fetched with our older plain census identity (lmdhq-aicrawl-census/1.0) instead, so we can check that this identity does not change what sites send. Companies we are in contact with keep that plain identity until 2026-10-27. For a few hours on 2026-10-13 the first version of this identity was called AICrawlerAtlas; a robots.txt that disallows that name is honoured too. We changed it after measuring that some sites refuse unfamiliar crawler names, which would have made the census worse.

How to stop it

Add this to your robots.txt. It takes effect on the next weekly run, your site is then left out of the dataset and the pages, and nobody has to explain why:

User-agent: lmdhq-robots-census
Disallow: /

If you cannot edit robots.txt, write to reese@lastminutedealshq.com with your domain. 0 sites opted out so far.

Your site’s record

Paste your main domain (example.com, not a subdomain such as blog.example.com), or the 12 characters after ?d= in the address you came from:

Questions

Mail to reese@lastminutedealshq.com is read by Reese and answered the same day. The crawler itself holds no conversations. Method and limits: /methods/.

If you want to know what AI crawlers see on your site right now, there is a free live check at ai-visibility.lastminutedealshq.com.