What Do AI Crawlers Actually See on Your Site? The 2026 JavaScript Gap
Forty-two percent of JavaScript-rendered content is invisible to AI crawlers. And the llms.txt file the industry told you to create for them? Ninety-seven percent get zero crawler visits.
If you pulled up a network trace of GPTBot hitting your website right now, you'd see something slightly unsettling: it makes the request, reads the HTML it gets back, and moves on. No JavaScript execution. No waiting for React to mount. No fetching the client-side chunks that actually contain your product descriptions, navigation, and the bulk of your text.
Onely's February 2026 analysis of six thousand websites found that 42% of JavaScript-rendered content is effectively invisible to AI crawlers. That's not a vague estimate — that's measured at scale, against the crawlers actively deciding what AI systems know about your site.
So the question is: how bad is this, and what can you realistically do about it?
Where does this data come from?
The findings here draw on three sources. Onely's February 2026 study crawled 6,000 websites with both a JavaScript-capable browser and a plain HTTP client, then measured the delta in visible content. Lantern's large-scale analysis tracked 569 million GPTBot requests across a major hosting network and found zero evidence of JavaScript execution in the entire sample. For llms.txt adoption figures, we're using Rankability's June 2026 survey of the top 1,000 domains and an Ahrefs study of 137,000 sites.
Which AI crawlers actually render JavaScript?
The short answer: Googlebot does. AI search crawlers don't.
All of the major AI-specific crawlers — GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot — fetch your HTML and stop there. They don't run a headless browser. They don't execute your JavaScript bundles. Lantern's analysis of 569 million GPTBot requests found not a single instance of JavaScript execution across that entire dataset. The sample is large enough that this isn't a rounding error.
Googlebot is the exception — it has rendered JavaScript reliably since 2019. AppleBot has partial support, though it isn't consistent. Every major AI-specific crawler you'd actually care about for AI search visibility doesn't render JavaScript at all.
This is where a lot of sites are getting caught out. If your SEO is solid — Googlebot can see your content — you might assume everything can. For traditional search rankings, that's true. For AI visibility, there's a real gap.
How much content does that actually hide?
Onely's 2026 study found the gap is 42%. On a client-side rendered site, when GPTBot requests your homepage, it's reading something like this:
<!DOCTYPE html>
<html>
<body>
<div id="root"></div>
<script src="/bundle.js"></script>
</body>
</html>
Your product descriptions, value proposition, navigation links, and blog content all live in /bundle.js. The crawler fetches the page, doesn't execute the script, and moves on with essentially nothing.
The problem is worst for certain site types. E-commerce platforms that load product listings via client-side API calls. SaaS marketing pages where the copy mounts after a component hydrates. Documentation sites built entirely as single-page apps. Static sites and server-rendered apps are mostly fine — what the server sends is what the crawler sees.
The frustrating part is this problem was invisible for years. Googlebot renders JavaScript, so if Google could see your content, you assumed everything could. AI crawlers broke that assumption without making much noise about it.
What about llms.txt — isn't that the fix?
The llms.txt spec was proposed in late 2024 as a way to give AI systems a curated map of your site. Put a Markdown file at /llms.txt listing your most important pages with short descriptions. Optionally add /llms-full.txt with the full page content embedded. The idea: tell AI crawlers what to read instead of hoping they figure it out.
Adoption has grown fast. Rankability's June 2026 survey found 8.7% of the top 1,000 websites now publish one — up from roughly 1% eighteen months ago. That's nearly a 9x increase.
But here's the data that puts it in perspective: Ahrefs studied 137,000 sites and found that 97% of llms.txt files received zero crawler visits in May 2026.
AI search crawlers aren't reading the file. GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot overwhelmingly skip /llms.txt and crawl your HTML directly. Google has explicitly said it doesn't use llms.txt as a ranking signal.
The one context where llms.txt actually appears to work: developer tools and AI-assisted coding environments. If your site serves technical documentation, a well-structured llms.txt measurably improves how AI coding assistants reference it. That's a real use case. It's just not the AI search visibility fix most people are deploying it for.
What should you actually do with this?
The most reliable fix for the JavaScript visibility gap is also the most direct: put your content in the HTML the server sends, before any JavaScript runs. For public-facing pages — product descriptions, landing pages, blog posts, documentation — the initial server response should contain the full text. Framework-specific terms vary (SSR, ISR, static generation), but the outcome is the same from the crawler's perspective.
If server-rendering isn't on the near-term roadmap, pre-rendering is the practical middle ground. A pre-rendering layer fetches your pages with a JavaScript-executing browser, stores the rendered HTML, and serves that static version to known crawlers based on user-agent. The tradeoff is cache freshness — you need to regenerate when content changes — but for pages that don't update constantly, it works well.
On llms.txt specifically: the 97% zero-traffic figure is fairly clear guidance. If you have developer documentation and setting it up takes less than a day, ship it. But don't build a quarter of SEO effort around it while your product pages are invisible to AI crawlers because they rely on client-side rendering.
The underlying reality is straightforward. AI crawlers are capable at reading well-structured HTML. They are not going to execute JavaScript to find your content. The sites that end up with strong AI visibility are the ones making their content legible in the first HTTP response — not the ones optimising for spec files the crawlers aren't actually reading yet.