Why Can a Third of the Web's Top Sites Not Be Read by AI Crawlers?
A May 2026 study of 274 fintech homepages found 17% deliver zero content to AI crawlers. 36% serve less than 80% of their page. Here's what's causing it — and why robots.txt isn't the full picture.
Seventeen percent of the world's top fintech companies deliver exactly zero readable content when an AI crawler visits their homepage. Not thin content — zero. What GPTBot, ClaudeBot, and PerplexityBot receive is an HTML shell with <div id="root"></div> and nothing else. This comes from Web Performance Tools' State of Agent Visibility in Fintech 2026 report, which ran a plain HTTP fetch against 274 fintech homepages from the CNBC World's Top Fintech Companies list in May 2026, and compared it against a full browser render. The gap is striking.
How was the study done? Each of the 274 sites was tested on May 25, 2026, with two approaches: a raw HTTP fetch with no JavaScript executed (the same kind major AI crawlers make), and a full browser render using Playwright with Chromium. Content from each was extracted and converted to plain text. The raw-fetch result was measured as a percentage of the rendered result — that's what "content delivered to AI crawlers" means throughout this post. The USTA Closing Web Index for July 2026, covering 10,894 domains, contributed the blocking-rate data. Publisher blocking rates come from a BuzzStream analysis of top news sites.
So what percentage of content are sites actually delivering to AI crawlers?
The 274 fintech sites split into distinct tiers. 64% serve 80% or more of their content in raw HTML — those are broadly accessible to AI crawlers as things stand. 36% (99 sites) fall below that threshold. The most striking finding: 47 sites (17% of the total) deliver literally nothing in a raw HTTP fetch. Another 8 sites fall in the near-blank range, delivering less than 30% of what a browser render would produce.
Why does this keep happening? Client-side rendering is the main culprit. When a site uses a JavaScript framework to populate content after page load, the server sends a minimal HTML shell and relies on the browser to build the rest. Human browsers do this automatically. AI crawlers don't — they make an HTTP request, parse the response, and leave without executing anything. If your content only exists in the DOM after JavaScript runs, it doesn't exist for GPTBot, ClaudeBot, or PerplexityBot.
The median fintech site in the study takes 21 times longer to fully render in a browser than to respond to a raw HTTP fetch. All of that render time produces content those crawlers never see.
Are AI crawlers even being let in to check?
That's the other half of the picture. Site-level blocking decisions — primarily through robots.txt — look wildly different depending on which slice of the web you're measuring.
Among top news and publishing sites, 79% block AI training crawlers via robots.txt. Among the top 1,000 websites overall, 25% block GPTBot — up from just 5% in early 2023. Across a broader sample of nearly 11,000 domains tracked in the USTA Closing Web Index for July 2026, 19.1% block at least one major AI crawler. On the general web, it's around 10%.
The publisher numbers make sense as a coordinated industry response: news organisations have collectively decided to protect their archives from AI training crawlers. The jump from 5% to 25% for GPTBot blocking among the top 1,000 sites in three years shows how fast that stance has spread beyond publishers.
Does a robots.txt disallow rule actually stop anything?
Here's where it gets more complicated. robots.txt is a policy document. It tells compliant bots what they shouldn't access, and relies entirely on those bots agreeing to respect it. It isn't a technical access control mechanism, and enforcement is voluntary.
Most major AI crawler operators publicly commit to respecting robots.txt, and the evidence is that well-known crawlers from established companies generally do. But two problems persist. First, less reputable crawlers treat robots.txt as optional or ignore it entirely — it's just a text file. Second, even for compliant operators, there's no universal audit trail. You can't easily verify what a crawler is actually fetching against what your robots.txt says it should skip. Some sites have found crawlers still appearing in access logs despite a clear disallow rule, often because of partial URL coverage, subdomain gaps, or redirect chains the rule didn't account for.
The practical conclusion: if your goal is to prevent AI training crawlers from accessing specific content, a robots.txt disallow rule is a signal worth sending, but it isn't an enforceable lock. Real access control — IP-based blocks, rate limiting, server-side challenge pages — happens at the infrastructure layer, not in a text file. Sites relying solely on robots.txt to protect content should check their server logs to confirm the crawlers they've disallowed aren't still showing up.
What should site owners actually do about this?
There are two distinct problems here, and they need different responses.
If you want AI crawlers to be able to read your content — because you're optimising for AI search visibility or want your site to appear in AI assistant answers — the rendering gap is the issue to tackle first. The fintech study makes this concrete: a third of a major industry's web presence is opaque to AI indexing, not because of any policy decision, but because of architecture choices that were made before AI crawlers were a relevant consideration. The fix is server-side rendering (SSR), static site generation (SSG), or a pre-rendering layer that serves rendered HTML to AI crawler user agents. All three address the same underlying problem — pick whichever fits your stack.
If you want to limit AI crawler access — to protect proprietary content, avoid training data inclusion, or control how your material gets used — robots.txt is worth doing as a baseline signal, but it needs to be backed by server-side controls to be meaningful. Check access logs. If a crawler you've disallowed is still appearing, the text file isn't the layer that'll fix it.
The uncomfortable reality underneath both problems: most sites haven't made a deliberate decision about either. The 17% zero-content figure from the fintech study isn't the result of intentional design — it's what happens when modern development defaults built for human browsers get applied to infrastructure that now includes AI crawlers as a new class of non-rendering HTTP clients that nobody accounted for when those choices were made.
A curl -A "GPTBot/1.2" https://yourdomain.com run against your homepage takes five seconds and gives you the same view the crawler gets. If the output is an empty shell, that's your answer.