Three Ways Your Content Disappears Before AI Crawlers Can Read It
69% of AI crawlers can't execute JavaScript — but the real problem is more subtle. Three failure modes hide content even after you've moved to SSR. Here's how to find them.
69% of AI crawlers can't execute JavaScript — and yet most teams debug AI visibility the same way they'd debug a traditional SEO problem. They check meta tags, audit structured data, run Lighthouse. None of that reveals whether the HTML an AI crawler actually receives contains any meaningful content. A study of 23 major AI agents found that not one of them renders client-side JavaScript. But the real problem is more nuanced than "just use server-side rendering." There are three specific failure modes that hide your content from AI crawlers, and two of them persist even after you've migrated to SSR.
Where does this data come from?
The JavaScript execution figures come from a joint analysis of over 500 million GPTBot fetches conducted across a major web infrastructure provider's network. Lazy loading adoption numbers are from the 2025 HTTP Archive Web Almanac, which crawls over 12 million pages monthly. The JS download rates for individual bots come from instrumentation across multiple high-traffic domains tracked over a 30-day period.
What is the blank body problem — and do you have it?
If your site uses a pure client-side rendered framework without SSR configured, an AI crawler fetches your page and receives something like this:
<!DOCTYPE html>
<html>
<head><title>My Product</title></head>
<body><div id="app"></div></body>
</html>
No product names. No descriptions. No body text at all. Just a mount point waiting for JavaScript to fill it — JavaScript that will never run.
In the 500 million GPTBot fetches analysed, GPTBot downloaded JavaScript files in 11.5% of requests. ClaudeBot downloaded them in 23.84% of requests. Neither executed a single one. Googlebot, by contrast, uses a headless browser and renders your JavaScript — typically with a crawl delay, but it does render. AI crawlers don't. They fetch the raw HTML, parse what's there, and move on.
The quick diagnostic is a single curl command: curl -A "GPTBot" https://yoursite.com/your-page | grep -c "<p>". If that returns zero or close to it, you have blank body pages. The fix is server-side rendering for the pages that contain text you want AI systems to ingest — product pages, blog posts, category descriptions, landing pages.
Does lazy loading hide your best content from AI crawlers?
You've implemented SSR. Your headings, first paragraphs, and key metadata are in the initial HTML. But how much of your page is below the fold?
The 2025 HTTP Archive Web Almanac shows that 68.1% of desktop images and 68.2% of mobile images carry the loading="lazy" attribute. For 16 to 17% of pages, even the LCP element — the single largest, most prominent content block — is lazy-loaded. AI crawlers don't simulate scroll events. They don't trigger viewport intersection callbacks. Content that loads on scroll or via intersection observers isn't in the HTML they analyse.
For text content, this matters a lot. If your product specifications, FAQ sections, customer reviews, or pricing tables load via JavaScript after the user scrolls into view, they're invisible to AI crawlers even though your page passes a basic SSR check. For images, the impact is more contextual: the image itself is missing, but if you've written descriptive alt text, that alt text is in the initial HTML and is visible.
The practical question to ask for each important section of your page: "Is this text in the HTML the server sends, or does it appear after a scroll event?" If it's the latter, it won't be in the AI crawl.
What is hydration-only state, and why is it so easy to overlook?
This failure mode is the trickiest because your site passes a basic curl test. The server sends real HTML. But the HTML contains placeholder values, not actual data:
<span class="price">$--</span>
<div class="stock">Checking availability...</div>
<section class="reviews">Loading reviews...</section>
JavaScript then hydrates these components and replaces the placeholders with live data fetched from your API. In a browser, this happens in milliseconds and users never notice. For an AI crawler, the page content is whatever was in the server-rendered HTML — and that's the skeleton, not the real content.
This pattern is extremely common. E-commerce sites do it for dynamic pricing and stock status. SaaS products do it for personalised dashboard content. Content sites do it for recommendation sections, read counts, and comment totals. The architecture is perfectly reasonable — it's just invisible to any bot that doesn't run JavaScript.
The fix is to move the data fetch server-side for content that isn't user-specific. In practice, this usually means replacing a client-side useEffect data fetch with a server-side data fetch at render time. The price of a product, the headline features of a plan, the most recent review — none of these need to be fetched after the page loads. If they're fetched server-side, they land in the initial HTML and AI crawlers can read them.
What should you actually do about this?
The fastest way to see all three failure modes is to compare what your server sends with what your browser renders. Load your most important pages in a browser, then fetch them with curl using a bot user agent and read the raw output. The gap between the two is your AI crawl problem.
Most teams find that one of the three failure modes is dominant. Pure CSR apps with a blank body need a framework migration — that's the biggest lift. Lazy loading and hydration failures are often a matter of auditing a handful of templates and moving a few data fetches. Neither requires rebuilding your application.
Start with your highest-traffic pages or the pages you most want AI systems to reference. Run the curl test. Check whether the content you want to be discovered is actually in the HTML the server sends. If it isn't, one of these three failure modes is the reason.