One in eleven x402 sites blocks AI training crawlers
Correction, 27 September 2026. The first version of this post said that 82 of these sites refuse
ClaudeBot while letting GPTBot in. That was wrong. We had sent the crawlers’ bare names, and Cloudflare recognises the bare name
ClaudeBot but not the bare name GPTBot. Sent with the user-agent strings the crawlers really use, GPTBot is refused on
the same sites. What these sites block is AI training crawlers from several companies. The crawlers that fetch pages for ChatGPT,
Claude and Perplexity answers get in. We checked every site again the same evening. The title, the numbers and the advice below are
corrected.
On 27 September 2026 we checked the 927 sites listed in the x402 Bazaar — the public directory of
services that sell to AI agents over x402 — the way AI crawlers read a page: plain HTTP, no browser, no JavaScript.
86 of the 908 that answered (9%) turned away the AI training crawlers GPTBot and
ClaudeBot with 403 Forbidden. 83 of those 86 let in the crawlers that fetch pages for answers:
OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot.
Blocking training while staying searchable is a legitimate choice. It is also what Cloudflare’s own migration gives a site that
had its old “Block AI Bots” switch on. But only 5 of these 86 sites mention an AI crawler in their
robots.txt. For a service whose customers are AI agents, it is worth checking that this is a choice: a model that never read your
pages in training does not know your API exists until a search turns it up.
What we measured
- 927 sites from the x402 Bazaar whose front page served readable HTML when we first probed all 2,060 hosts listed there on 25 September, minus preview deployments. 910 of them answered our regular check on the morning of 27 September.
- Plain HTTP only, with a user agent that identifies itself as a bot. No headless browser.
- A control request for every site. That evening each of the 910 was asked once with our own user agent,
then once with each crawler’s complete user-agent string: GPTBot, ClaudeBot and CCBot (training), and OAI-SearchBot, ChatGPT-User,
Claude-User and PerplexityBot (search and answers). A crawler counts as refused only when our own request got a page and the crawler got a
403. 908 sites answered our own request that evening. - From an ordinary connection outside Cloudflare. Our regular checks run on Cloudflare Workers, and a request from there comes back with Cloudflare’s headers on every site. From an ordinary connection we can see who actually answers.
The results
AI crawlers, 908 sites that answered our own request:
| Crawler | Sites |
|---|---|
| Refused GPTBot (OpenAI, training) | 87 |
| Refused ClaudeBot (Anthropic, training) | 86 |
| Refused CCBot (Common Crawl, training) | 88 |
| Refused both GPTBot and ClaudeBot | 86 (9%) |
| … and let OAI-SearchBot, ChatGPT-User, Claude-User and PerplexityBot in | 83 |
| Refused at least one of those four answer crawlers | 4 |
The pages, all 927 sites:
| Check | Sites |
|---|---|
robots.txt names an AI crawler | 13 |
| Under 300 characters of visible text without JavaScript | 69 (7%) |
No sitemap.xml | 422 (46%) |
Serves llms.txt | 699 (75%) |
| Has JSON-LD structured data | 334 (36%) |
| Has a meta description | 707 (76%) |
Plain http:// serves the site instead of redirecting to https | 196 (21%) |
| A made-up path returns 200 instead of 404 | 43 (5%) |
The refusals come from Cloudflare
All 86 sites that refused both training crawlers answered through Cloudflare (Server: cloudflare), and the
403 came from Cloudflare as well. On 15 September 2026 Cloudflare replaced its single “Block AI Bots” switch with separate
settings for search, training and agent crawlers, and moved existing sites to them automatically
(Cloudflare, 15 September 2026). A site that had the
old switch on now blocks training crawlers and still lets search crawlers in — the pattern we see here. We cannot see these
sites’ settings, so we cannot tell a deliberate choice from a migrated one.
Two things these sites do well
They know llms.txt. 75% of them serve one; of the AI-built sites we measured on
20 September, 30% did. They are not empty. On 20 September 61% of AI-built sites
had under 300 characters of visible text without JavaScript; here it is 7%. Most x402 sites are API landing pages
rendered on the server.
How to check your own site
curl -s -o /dev/null -w "%{http_code}\n" https://your-site/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)" https://your-site/
curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot" https://your-site/Use the complete strings, not the bare names. Cloudflare recognises some crawlers by their bare name and others only in their full
form, and that is exactly what misled the first version of this post. 200, 403, 200 means training is
blocked and search is open. If you want models to learn about your service, look at Security → Settings and the AI crawler controls on
Cloudflare and set Training to Allow. If blocking training is what you want, there is nothing to do. A 403 on the third line is
the one to fix: that crawler feeds ChatGPT search.
Caveats
This is one day, 27 September 2026, and one directory. We sent the crawlers’ user-agent strings, not requests from the crawlers’ own IP addresses, so a site that verifies crawlers by address may treat the real ones differently. We will run the same check again.
The 859 sites that passed our check are listed, free, in the catalog.