We fetched 104 AI-built sites the way GPTBot does. 61% were empty.
On 20 September 2026 we fetched 120 websites built with AI app builders the way GPTBot, ClaudeBot and PerplexityBot fetch a page: one plain HTTP request, no browser, no JavaScript. 104 of them answered. For 63 of those 104 (61%), the HTML held fewer than 300 characters of visible text.
In a browser they look finished. To a crawler that does not run JavaScript, most of them are an
empty <div id="root"></div> and a script tag.
Why a crawler sees less than you do
Sites generated by AI app builders are usually single-page apps: the server sends an almost empty HTML file and the browser builds the page with JavaScript. Google renders JavaScript, so such a site can be indexed and ranked. The crawlers behind AI answers do not. Vercel and MERJ analysed crawler traffic across Vercel's network and found that “none of the major AI crawlers currently render JavaScript”, naming OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot and PerplexityBot (The rise of the AI crawler, December 2024). Google's Gemini is the exception, because it uses Googlebot's rendering.
So a site can rank on Google and still be missing from what ChatGPT, Claude and Perplexity know about it.
What we measured
- 120 sites: 72 on their own domains (61 from the public
Made with Lovable gallery, 11 from public lists of sites built with Cursor, Claude Code or v0),
42 on
*.lovable.appand 6 v0 and Vercel templates. - 104 answered with HTML. 16 (13%) were already dead while still listed in a gallery.
- Plain HTTP only. Every check is a request from a server with a user agent that identifies itself as a bot. No headless browser anywhere, because the crawlers we care about do not use one either.
- A control group of 5 established, server-rendered sites, two of them large technology news sites, went through exactly the same checks. A test that marks everything red tells you nothing; the control shows that these checks can come back clean.
The results
| Check | AI-built (n = 104) | Control (n = 5) |
|---|---|---|
| Under 300 characters of visible text without JavaScript | 63 (61%) | 0 |
Page body is an empty #root or #app container | 60 (58%) | 0 |
| A made-up path returns 200 instead of 404 (soft 404) | 70 (67%) | 0 |
| www and the bare domain both serve the site, no redirect (own domains, n = 60) | 24 (40%) | 0 |
| www or the bare domain is dead (own domains, n = 60) | 7 (12%) | 0 |
No sitemap.xml (404) | 42 (40%) | 0 |
| No canonical link | 66 (63%) | 0 |
| No JSON-LD structured data | 65 (63%) | 1 |
No llms.txt | 73 (70%) | 3 |
noindex | 0 | 0 |
robots.txt blocks everything | 0 | 0 |
Redirect without a Location header | 0 of 120 | 0 |
On average a live AI-built site had 5.2 of the 16 problems we counted; the control group had 1.2. Only 6 of the 104 sites (6%) passed every core visibility check; 3 of the 5 control sites did.
What is not the problem
We expected the classic technical SEO failures: a stray noindex, a robots.txt
that blocks everything, redirects that go nowhere. Not one of the 104 sites had any of them. AI-built
sites are not blocked. They are empty.
A side finding: user agents
We also sent the same request with eight different User-Agent strings. Our own
self-identifying bot agent got a 200 from all 104 sites, and so did the strings of PerplexityBot,
ChatGPT-User and Bingbot. Python's default urllib agent got a 403 from 78 of them:
78 of the 79 sites behind Cloudflare. The GPTBot string got a 403 from
5 sites and a 429 from 2; the ClaudeBot and CCBot strings got a 403 from
5 each.
One caveat. We sent these strings from our own server. The real crawlers come from their own published IP ranges and may be treated differently, so this measures how sites react to the user-agent string, not what GPTBot itself receives.
How to fix it
- Put the content in the HTML. Pre-render the pages that matter at build time, or
render them on the server. The test takes one line:
curl -s https://your-site/ | grep "your headline". If the headline is not in the output, AI crawlers do not see it either. - Return a real 404. Request
/this-page-does-not-exist. If the answer is 200, every broken link looks like a real page to a crawler. For a single-page app this is usually a hosting setting: most hosts serveindex.htmlfor every path unless told otherwise. - Pick one host. Redirect www to the bare domain, or the other way round, with a 301, and make sure the other one is not dead.
- Publish a sitemap at
/sitemap.xmland name it inrobots.txtwith aSitemap:line. - Describe the site in structured data. A small JSON-LD block (Organization or WebSite, plus Product or SoftwareApplication if you sell something) gives a crawler facts it does not have to guess from the page.
Caveats
This is a snapshot of one day, 20 September 2026, and most of the sites come from one builder's gallery. If AI app builders start rendering on the server by default, the biggest finding here disappears overnight, and we will measure again.