Why Is 97% of Your llms.txt Being Ignored?

Here's a number worth sitting with before you add another file to your site root: in May 2026, 97% of llms.txt files received zero requests from AI systems. Not low engagement — zero. This is despite llms.txt adoption growing 8.8x over the previous 12 months, from around 4,000 instances to over 36,000. So what exactly is everyone implementing, and is any of it actually working?

Method

The adoption and request-rate figures come from Originality.ai tracking across tens of thousands of llms.txt instances, published in mid-2026. Breakdown by site tier comes from Rankability's June 2026 crawl of the top 1,000 websites and Casey Burridge's HTTP Archive BigQuery analysis across millions of sites. SE Ranking published a separate 300,000-domain study with citation-correlation analysis. Crawler rendering behavior data comes from Vercel and MERJ's analysis of over 500 million GPTBot fetches and SearchViu's audit of 23 major AI agents.

Who's actually publishing llms.txt?

Adoption rates look wildly different depending on what slice of the web you measure. Among the top 1,000 websites, 8.7% have a llms.txt as of June 2026. Zoom out to the top 10,000, and it's 5.61% — up from just 1.04% in July 2025. SE Ranking's broader 300,000-domain sample puts the number at 10.13%.

What's striking is how flat adoption is across traffic tiers. Low-traffic sites with under 100 monthly visits show 9.88% adoption. High-traffic sites with over 100k monthly visits show 8.27%. There's almost no correlation between how much traffic a site gets and whether someone bothered to add the file. That flatness points to platform defaults rather than individual technical decisions.

Shopify stores show 78.1% adoption. WordPress sites show 8.7%. Contentful-powered sites show 22.9%. Adobe Experience Manager properties come in at 19%. The Shopify number isn't because Shopify merchants are more technically engaged — Shopify started generating the file automatically at the platform level. Strip out platform-driven adoption and the deliberate-choice numbers look considerably thinner.

Are AI systems actually reading these files?

The 97% zero-requests figure answers this directly. Most of the systems that do read llms.txt are AI agents — tool-using assistants doing structured retrieval, like a chatbot navigating your documentation or an AI trying to understand your API surface. Bulk training crawlers and the retrieval layer of AI search engines generally don't read it.

This makes sense when you think about what llms.txt is trying to do. It's a Markdown file designed to give a language model a context-window-friendly summary of your site — useful when an AI assistant is actively browsing and wants to understand what you offer. It was never designed as a signal to training crawlers. The spec shares DNA with robots.txt, which crawlers respect selectively at best.

SE Ranking's citation analysis found no measurable correlation between having a llms.txt file and how often a domain gets cited by AI search engines, once site authority, schema density, and content recency are controlled for.

Can AI crawlers even see your content?

Before worrying about what your llms.txt says, there's a more fundamental question to ask: can AI crawlers see anything on your site at all?

For a large portion of the web, the answer is no. A study of 23 major AI agents found that 69% cannot execute JavaScript. Vercel and MERJ's analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution across any of them. GPTBot downloads JavaScript files on roughly 11.5% of requests — likely to detect frameworks or parse import maps — but never runs them. ClaudeBot downloads JavaScript at a higher rate (about 23.84% of requests) and also never executes any of it.

What does this mean for a client-side-rendered app? When GPTBot visits a React SPA or Vue app that populates content after page load, it receives an HTML shell: a <div id="root"> with nothing inside. Product descriptions, blog posts, schema markup — all of it exists only in the DOM after JavaScript runs, which never happens. The crawler logs a request but gets essentially nothing usable.

Googlebot is the clear exception. It renders pages using a headless browser, and Google dropped its JavaScript SEO health warning in March 2026 because its rendering pipeline is now considered mature. But that applies only to Googlebot. The AI crawlers that determine your training-data coverage and AI search visibility are still reading flat HTML.

Platform defaults versus actual coverage

The CMS adoption data makes the underlying gap concrete. Shopify at 78.1% looks impressive, but Shopify natively serves server-rendered HTML — the platform handles rendering, so AI crawlers can actually access the content. Those merchants have a llms.txt pointing to content that's already visible. That's coherent, even if the file goes largely unread.

WordPress at 8.7% is the messier story. Many WordPress sites use JavaScript-heavy page builders, headless setups with client-side rendering, or interactive components that put content behind JS execution. Some portion of that 8.7% likely has a llms.txt pointing to pages that are partially or completely invisible to AI crawlers.

What should site owners actually do?

The most useful diagnostic is simpler than most llms.txt implementation guides suggest: fetch your own site as a plain HTTP client would. Running curl -s -H "User-Agent: GPTBot/1.0" https://yoursite.com and checking the response body tells you what AI crawlers are actually receiving. If the HTML is mostly empty scaffolding with no readable text, that's the real problem — and a llms.txt file pointing to invisible content helps nothing.

If your content is readable in raw HTML, the path forward splits by goal. For AI search visibility, the signals that actually correlate are structured schema markup, factual content that can be quoted without surrounding context, and clean heading hierarchy. For AI agents specifically — the tools-using systems that will actually read a llms.txt — the file has genuine value, because it gives those systems a shortcut to understanding your site structure. But that's a narrower use case than the surrounding narrative implies.

Sequence matters: fix the rendering foundation before adding the signpost.