Is Your Site Invisible to AI Crawlers? The Data Might Surprise You
97% of llms.txt files get zero AI crawler traffic, and AI crawlers can't render JavaScript either. Here's what your site actually looks like to the bots deciding whether to cite you.
Ahrefs ran a large-scale analysis of 137,000 sites publishing llms.txt files in May 2026 and found that 97% of them received zero requests from AI crawlers. Not low traffic — zero. The file that half the SEO industry has been recommending you add is being almost entirely ignored by the systems it was built for.
That's the smaller problem. The bigger one is that a large fraction of websites look completely blank to AI crawlers whether or not they publish an llms.txt file — because the crawlers don't execute JavaScript, and modern websites deliver most of their content through JavaScript.
Where does this data come from?
The llms.txt adoption figures come from Rankability's June 2026 survey of the Tranco top 1,000 domains and Ahrefs' crawler log analysis across 137,000 sites. JavaScript rendering data comes from HybridRanking's detailed analysis of how major AI crawlers handle client-side rendering, corroborated by Vercel's "Rise of the AI Crawler" study — which analysed 500 million GPTBot requests. Content visibility impact figures are from Onely's February 2026 technical audit.
So how widely has llms.txt actually spread?
Growth looks impressive on paper. Adoption in the top 10,000 sites went from 1% in July 2025 to 5.6% by June 2026 — roughly 5x in twelve months. Among the top 1,000 reachable websites, 15.8% now publish one. Shopify is an outlier worth noting: 78% of Shopify merchants have a file, because Shopify started auto-generating it.
Here's the catch: AI crawlers aren't fetching these files. No major AI search provider has publicly committed to reading or acting on llms.txt in its production crawling pipeline. Server log analysis across large sample sets consistently shows the major AI crawlers skipping /llms.txt entirely and crawling your HTML pages directly — when they can read those pages at all.
This might change. The spec is still evolving and some providers may eventually commit to reading it. But as of right now, llms.txt has real value for developer documentation, AI coding assistants, and internal RAG pipelines where you control what the model reads. As a signal to public AI search bots deciding whether to cite you? The evidence isn't there.
Can AI crawlers actually read your pages?
This is the question that matters more than llms.txt, and the answer depends entirely on how your site is built.
None of the major AI crawlers execute JavaScript. GPTBot fetches JavaScript files in 11.5% of its crawl requests — but it never runs them. The dominant AI assistant bots behave similarly. A technical analysis covering 23 major AI agents found that 69% are completely incapable of running client-side code.
Googlebot is the significant exception — it's been rendering JavaScript since 2015 and handles it reasonably well. But Googlebot crawls for traditional search rankings, not to train or power AI assistant responses.
Onely's February 2026 audit put a number on the gap: 42% of JavaScript-rendered content is never indexed by AI systems. Content that exists on your site, that users see every day, that AI search products treating your site as a source would never encounter.
The practical upshot: if your homepage, product pages, or core marketing copy are delivered via client-side JavaScript without server-side rendering, the AI crawlers see an empty shell. A <div id="app"></div> with nothing in it.
What does your site actually look like to these crawlers?
There's a quick test that takes about ten seconds. Fetch your own homepage with curl — no JavaScript, no browser:
curl -s https://yoursite.com/ | head -200
What you get back is close to what AI crawlers see. If the <body> is mostly empty divs and <script> tags, your site is invisible to AI systems regardless of how much effort you've put into structured data or llms.txt.
A few patterns to recognise:
React/Vue/Angular SPAs without SSR present crawlers with an empty body. The content only materialises after JavaScript runs, which AI crawlers never do. Your product descriptions, about page, pricing — none of it is there.
Next.js or Nuxt with Server Components (the default in recent versions of both frameworks) ship server-rendered HTML. Sites using these frameworks at their defaults are generally fine.
Hybrid sites with lazy-loaded sections are trickier. The page frame might be readable, but specific sections — pricing tables, testimonials, product specs — may only appear after JavaScript execution. Those sections are invisible even when the rest of the page isn't.
CMS-powered sites (WordPress, traditional CMS platforms) typically render HTML server-side before delivery. These tend to be the most consistently readable by AI crawlers.
What should you actually do about this?
If you have a JavaScript rendering problem, that's the fix — nothing else moves the needle until your content is in the initial HTML response.
For sites on modern JavaScript frameworks, check whether you're using SSR (server-side rendering) or SSG (static site generation). Most Next.js sites built in the last two years use one of these by default. If you're running a pure SPA or aren't sure what rendering mode you're in, that's the first thing worth investigating.
Once your content is genuinely available in the initial HTML, a few additional changes help:
Schema.org JSON-LD in your <head> gives crawlers structured, machine-readable context about what type of content you're serving. This data is present in the initial server response — it doesn't need JavaScript to execute.
Open Graph and meta tags in static HTML ensure that even crawlers focused primarily on metadata get something useful about your page's purpose and content.
An llms.txt file is worth the hour it takes to create, once your rendering is sorted. You're positioning yourself for whenever AI providers decide to act on it consistently, and you're already giving developer-facing tools and AI coding assistants a useful map of your most important content.
What llms.txt is not — right now, based on the evidence — is a ranking or citation tactic for AI search. The data on that is pretty clear. Crawlers aren't reading the file, and even if they were, no provider has explained how (or whether) it influences what their system cites.
The uncomfortable reality is that AI discoverability in 2026 is mostly a rendering and crawl-access problem. Get your content into server-rendered HTML first. Everything else is secondary.