What Do AI Crawlers Actually See When They Visit Your Site?

42% of JavaScript-rendered content is invisible to AI crawlers — that figure comes from Onely's February 2026 analysis of content indexation across major AI systems. If your site relies on client-side rendering, the product descriptions, article copy, and navigation that users see in a browser are often completely absent from the HTML that AI crawlers receive. The question is no longer whether this affects you. It's which parts, and by how much.

Where does this data come from?

The figures here draw on Vercel and MERJ's analysis of more than 500 million GPTBot fetches, HybridRanking's technical audit of 23 major AI crawlers conducted mid-2026, Rankability's ongoing llms.txt adoption tracker for the Tranco top-1,000, and MarketerFirst's comparison of readable content on major retail product pages in 2026.

Can any AI crawlers actually execute JavaScript?

Of the 23 major AI crawlers audited in mid-2026, 69% cannot execute JavaScript at all. GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Meta-ExternalAgent, ByteSpider — none of them run scripts. They fetch the initial HTML response, extract whatever text is in that payload, and stop. JavaScript files do get downloaded sometimes — GPTBot requests them about 11.5% of the time and ClaudeBot roughly 23.8% — but neither executes them. A React, Vue, or Angular SPA returns a bare <div id="root"></div> to these crawlers with nothing else inside.

The exceptions are narrow. Googlebot runs a full headless browser as part of its indexation pipeline, and its AI-derived products inherit that capability. Bingbot has limited JavaScript execution for a subset of scenarios. Everyone else works from static HTML only.

That split matters more than it looks. Bingbot powers a significant share of AI assistant answers — estimates put it at around 92% of ChatGPT's agent query index. A SPA that's fully transparent to Googlebot can still be completely invisible to the AI systems that increasingly influence referral traffic.

What does a crawler actually receive from your pages?

The Vercel/MERJ analysis makes the gap concrete: GPTBot downloads JavaScript 11% of the time and executes none of it. ClaudeBot downloads scripts 23% of the time with the same result. Both crawlers request the files and then do nothing with them — it looks like JS execution might be coming eventually, but as of mid-2026 it hasn't arrived for either.

This isn't only a single-page app problem. Hydration-heavy frameworks can render a page server-side but then overwrite key content during the client hydration step. Any content marked for deferred loading — behind an intersection observer, a scroll listener, or a setTimeout — won't appear in the initial response either. The three failure modes that catch most sites out are: entirely empty body (pure SPA with no static fallback), content cleared during hydration, and content gated behind user interaction such as lazy loading or tabbed interfaces.

The retail product page testing is the sharpest illustration of the practical impact. Testing four major retailers found that between 49% and 89% of visible text blocks were absent from the HTML an AI crawler would receive. Product names sometimes made it through — usually because they appear in the page title or a static heading. But prices, specifications, descriptions, and reviews — the content an AI assistant actually needs to evaluate a product — were frequently missing entirely.

Is llms.txt closing the gap — or is it mostly noise?

As the rendering gap has become more widely documented, many site owners have moved toward a different approach: telling AI systems explicitly what to read, rather than hoping they can work it out. llms.txt is a plain-text file served at your domain root that lists your key content with short descriptions — a curated navigation guide aimed at AI rather than human readers.

The file format is simple. An H1 heading with your site name, a short blockquote summary, and H2 sections listing key pages or documentation with brief descriptions. The whole thing can be generated from your existing sitemap in a few minutes. The harder part is deciding what to include: a flat dump of every page is less useful than a curated subset of your most important content, organised by section.

The adoption numbers are growing fast. In July 2025, just 1.04% of the Tranco top-10,000 sites published a valid llms.txt file. By June 2026, that had risen to 5.61% of the top 10k and 8.7% of the top 1,000. Higher adoption at the top end makes sense — larger organisations have both the technical capacity to respond to emerging standards and more to gain from being accurately understood by AI systems.

But the usage data has a catch. One analysis of over 60,000 AI bot visits across a 90-day period found that roughly 0.1% of those visits actually requested the llms.txt file. The signal is being sent. Whether AI crawlers are reliably reading it remains unclear — none of the major AI labs have publicly committed to acting on llms.txt in their production crawlers.

What should you actually do with this?

Server-side rendering or static generation is the only reliable fix for the JavaScript rendering problem. This doesn't require a full architecture rewrite — most modern frameworks support incremental SSR, and a pre-rendering proxy at the edge can serve static HTML to bots while keeping your client-side setup intact for users. The practical question to start with: what does your site's initial HTML response contain, before any JavaScript executes? If the answer is not much, AI crawlers are seeing an empty shell.

Even on SSR sites, watch out for lazy-loaded content. Anything that only appears after a scroll event, user interaction, or timer is typically absent from the initial payload and invisible to crawlers. Third-party widgets — pricing tools, recommendation engines, dynamically loaded reviews — are similarly invisible regardless of what your core framework does. Treat the raw HTML source as the complete picture of what any AI system sees when it visits your page.

llms.txt is worth adding if your content justifies it. Given the current 0.1% request rate from AI bots, it's not a high-ROI optimisation right now — but the file is straightforward to build and maintain. If AI crawlers eventually treat it as a primary discovery mechanism, having it in place early will matter. The real current value is consistency: a well-structured llms.txt makes your content inventory legible to any system that does choose to read it, without requiring any changes to your site architecture.

The wider point is that "visible to search engines" and "visible to AI crawlers" are no longer the same thing. You can rank well in Google and still be invisible to every AI system that doesn't inherit Googlebot's rendering pipeline. Auditing what AI crawlers actually receive — not just what your analytics dashboard shows — is worth making part of your regular technical review.