402registry-check: the crawler behind 402registry

Why it visited your site

• Someone paid for a check of your site — usually its owner, or an AI agent working for them. The report goes only to whoever paid.
• Someone submitted your site for a free check through our public queue.
• We picked your site for the public catalog from public lists of sites built with AI tools, or from public registries where the site announced itself to AI agents (such as the x402 Bazaar).
• A site listed in the catalog is re-checked for free about every 30 days, so the date on its entry stays true.

What one check requests

• The page itself, following at most 5 redirects; then the other form of the domain (with and without www) and the http:// version, one hop each.
• /robots.txt, /sitemap.xml, /llms.txt, and up to 2 URLs listed in your sitemap.
• Your own JavaScript bundle (one file, from the same host as the page), to look for keys left in it.
• /cdn-cgi/trace, to see whether the site is served through Cloudflare — so the fixes we suggest match your host.
• One URL that should not exist — /402registry-probe- followed by random characters — to see whether a missing page returns a real 404.
• The same page 4 more times, once with the user-agent string of each of these AI crawlers: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot, Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot), Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot), Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com), to see whether your firewall blocks them. Those requests come from us, not from the companies behind those crawlers.

At most 21 requests per check, each with a 6-second timeout, reading at most 1.5 MB. No JavaScript is run, no form is submitted, nothing is posted to your site.