Your llms.txt Is Published. Has Any AI Crawler Actually Read It?

One in twelve of the world's top 1,000 websites now publishes an llms.txt file. Rankability's June 2026 study put the number at 8.7% — representing 8.8× growth in twelve months. So here's the obvious follow-up question: is any AI crawler actually reading those files?

Almost certainly not. Server log analysis covering thousands of published llms.txt files found that 97% received zero requests from AI systems. The files that did see traffic were mostly hit by SEO audit tools checking whether the file exists — not by AI crawlers doing anything useful with the guidance inside.

Where does this data come from?

The adoption figures come from three independent studies. Rankability scanned the top 1,000 websites in June 2026. ProGEO.ai audited all 500 Fortune 500 companies in March 2026. A broader SE Ranking study covered approximately 300,000 domains. The zero-request finding comes from server log analysis published by PPC.Land tracking AI crawler behaviour against published llms.txt files. The JavaScript rendering data draws from a Vercel and MERJ analysis of over 500 million GPTBot fetch events, plus independent testing of every major AI crawler user agent.

How widespread is llms.txt really?

The adoption numbers look solid until you compare them with robots.txt. ProGEO.ai found that 92.8% of Fortune 500 companies have a robots.txt file. The equivalent figure for llms.txt? 7.4% — that's 37 out of 500 companies. A 12× gap between a convention that's been around since 1994 and one proposed in late 2024.

What do the files actually contain when companies do implement them? The format is plain Markdown: an H1 title, a blockquote description, then H2-labelled sections with bullet-pointed links to key pages. Each link gets a short description explaining what the page covers. An "Optional" H2 section, by convention, flags secondary content — changelogs, legal pages, archive material — that an AI can skip when it's running short on context.

The highest adoption is concentrated in developer tooling and documentation platforms. Mintlify and GitBook both auto-generate llms.txt files for their customers' sites, which explains part of the growth curve — a platform decision by a single documentation provider adds thousands of entries to the count overnight.

Is the problem that nobody is reading them?

The 97% zero-request figure is the number that rewrites the story. Of the files that do see AI traffic, most comes from a narrow set of developer-oriented coding tools — AI assistants that are explicitly built to follow documentation links during coding sessions. General-purpose AI assistants and the search indexers behind major AI platforms are, per available server log data, almost never requesting these files.

The reason is structural, not a temporary gap that's about to close. As of mid-2026, no major AI provider has publicly committed to incorporating llms.txt into their production answer or search pipelines. Google's search team confirmed in 2025 that they do not support it and have no plans to. The file format remains a community proposal, not an adopted standard. AI crawlers are engineered to crawl the open web — they weren't built to prioritise a voluntary guidance file that no production system treats as authoritative.

This doesn't make llms.txt categorically worthless. For documentation sites whose content will be queried directly by coding agents, it's a real signal — Mintlify reports meaningful agent traffic on their customers' documentation using it. But that's a specific, narrow use case. Extrapolating it to general AI search visibility isn't supported by the request data, and the data goes back far enough now to say something meaningful.

What's the bigger accessibility gap that's actually costing sites visibility?

If llms.txt has almost no uptake from production AI crawlers, the more structurally urgent problem is JavaScript rendering. Analysis of over 500 million GPTBot fetch events found zero evidence of JavaScript execution — not a low rate, zero. The search crawlers behind the major AI assistants and PerplexityBot behave identically. Every significant AI crawler in production today fetches raw HTML and reads what it finds in the initial response. Content that only appears after JavaScript executes is invisible to all of them.

This matters because client-side rendering is the dominant architecture for sites built in the last five years. A React app, a Vue SPA, anything using hydration-heavy frameworks — these deliver an essentially empty HTML shell on the initial HTTP response. The actual content is injected by JavaScript running in the user's browser. AI crawlers don't run that JavaScript. From their perspective, the page looks approximately blank.

Googlebot handles this differently. It runs a headless rendering pipeline that executes JavaScript, waits for the DOM to settle, and indexes the rendered result. Because most SEO tooling is calibrated against what Googlebot sees, developers often don't realise their content is invisible to every AI crawler. The site ranks well on Google — Google renders it correctly — and nobody checked what GPTBot or PerplexityBot actually receives.

What should you actually change?

On llms.txt: the honest advice is to wait until major AI platforms publicly commit to supporting it before treating implementation as urgent. That calculus could flip quickly — if a dominant AI search product adopts the format and documents it, implementation becomes meaningful almost overnight. Check the official documentation from platforms that drive traffic to your site before adding this to any list of priorities.

On JavaScript rendering: this is a real, active problem worth fixing regardless of what happens with llms.txt. Pre-rendering HTML for identified bot user agents — either via server-side rendering at the framework level, or via a reverse proxy that intercepts known bot user-agent strings and serves statically rendered snapshots — is the most direct fix. You don't have to abandon your React or Vue app. Serving pre-rendered HTML specifically to crawlers, while keeping the interactive application intact for human visitors, is a pattern with a well-documented implementation path.

The practical test is simple: fetch one of your most important pages with a bare HTTP request and no JavaScript execution, then look at what the HTML response actually contains. Strip the tags. Is the substantive content on that page present in the response body? If not, every AI crawler on the web is receiving an empty shell where your content should be.

That gap matters more than whether you have an llms.txt file. Fix the rendering problem first.

This post was generated autonomously by Wrenda's content pipeline.