Three weeks on, 80 of 82 x402 sites still block AI training crawlers

On 18 October 2026, three weeks after our first look, we sent the same eight requests to the same x402 Bazaar sites. Of the 86 sites that had refused GPTBot and ClaudeBot, the crawlers that collect pages for AI training, on 27 September, 82 were still online, and 80 of those still refused both. Of the 770 sites that had let the training crawlers in, not one had started to block them.

Our September report caught these sites twelve days after Cloudflare split its single switch for AI bots into separate settings for training, search and agents. A pattern seen so soon after a platform change could have been a passing state: sites moved onto a new default their owners had not yet noticed, about to be switched back. Three weeks later it has hardly moved. Whatever put the block in place, a decision or a migration, it has stayed.

What we measured

• The same sites. All 910 x402 Bazaar sites that our service had reached in its regular pass early on 27 September. 908 of them had also answered the evening run that the September report rests on.
• The same eight requests. The front page, fetched once under our own bot name and once under each of seven crawlers’ full user-agent strings, copied character for character from the September run. Three of them collect pages for training: GPTBot, ClaudeBot and CCBot. Four fetch pages for search and answers: PerplexityBot, OAI-SearchBot, ChatGPT-User and Claude-User. Redirects were not followed and no JavaScript ran; each request had fifteen seconds, and ten sites were checked at a time.
• The same vantage point. An ordinary connection outside Cloudflare, as in September, so the response headers show who really answers.
• Site by site. Every site’s eight answers were set against its own eight from 27 September. We call a crawler refused when the same site served our own request and gave that crawler a 403.
• The noise. Straight after the main run, the whole set went out a second time. 848 of the 851 sites that answered in both runs returned identical codes for all eight requests, so the differences between the two dates are real changes rather than a flaky connection.

The results

Sites that served our own request, on each date:

The counts fell mainly because fewer sites answered at all; the share that blocks both training crawlers held at 9%. Totals cannot tell you whether the same sites stand behind them. Following each site can.

Site by site

Of the 86 that blocked both in September, 80 still do. 2 now let every one of the eight requests through; both had refused GPTBot, ClaudeBot and CCBot, and one had refused PerplexityBot as well. The remaining 4 no longer serve a page to anyone.

2 of those 4 deserve a second look. Our own request and the four answer crawlers now get a server error from them, while GPTBot, ClaudeBot and CCBot get the same 403 as in September. Whatever has failed behind those sites, the refusal sits in front of it and carries on working.

Nothing moved the other way. Of the 770 sites that answered on both dates and let the training crawlers in during September, not one refuses them now.

The answer crawlers, the ones that fetch a page when someone asks ChatGPT, Claude or Perplexity a question, were left alone almost everywhere. In September 4 sites refused at least one of them; today 1 does, and it turns away all seven crawler strings, exactly as it did three weeks ago.

Across the whole list, 842 of the 852 sites that answered on both dates gave every one of the eight requests the same status code as before. Of the other 10, 2 are the sites that opened up and 3 now redirect every request to another address. The last 5 differ in one or two requests: a rate limit, a timeout, a single 403 that did not come back. That is the kind of difference our repeat run produced within minutes.

A side finding: sites that went dark

The check also counted something we did not set out to measure. Of the 908 sites that answered on 27 September, 56 (6%) did not serve our own request on 18 October, and 55 of those still failed in the repeat run.

• 30 answer 404 at the front page. The service may live on at its API addresses; we asked only for the page a crawler starts from.
• 12 return a server error.
• 11 no longer take a connection: the name does not resolve, or the connection is cut or times out.
• 3 refuse our own request outright.

Every one of these sites had a readable front page when we first went through the Bazaar. Our count of x402 front pages on 25 September found that about half of all listed hosts had none; three weeks later, part of the other half has stopped answering too. For a directory that agents use to choose where to send a payment, that is a lot of quiet loss in a short time.

If your service is on the list

Want AI models to learn what your service does? Then check what GPTBot gets from your front page. The three commands at the end of our September post still do it, and they use each crawler’s complete string, which matters: Cloudflare recognises some crawlers only in that form. A 200 for the first command and a 403 for the second means training is blocked. On Cloudflare, the switch is under Training in the AI crawler controls.

Three weeks of data add one point. A setting like this does not drift back by itself: 80 of 82 sites kept it. If the block arrived with the migration rather than from you, it will stay until you change it.

What we did not measure

The real crawlers. We sent their user-agent strings from our own connection, not requests from their published addresses, and a site that checks a crawler by its address may treat the real GPTBot differently, in either direction. That was true in September and is true now.

robots.txt. Our September figure for it came from a different check, run that morning through our own service; putting it next to anything measured here would compare two different tools.

The reasons. We see status codes, not settings. The 2 sites that opened may have chosen to allow training or may have moved to a host that blocks no one, and the steady 80 may be a decision or a default nobody has opened since.

New services. The list is the one we took from the Bazaar on 25 September. A service listed after that is not in it, so a newcomer that blocks training crawlers from its first day could not appear here as a new blocker.