Numbers from the AI crawler edge.
Twice-weekly research on AI crawler accessibility, bot traffic, measurement, and the MCP protocol. Every post is grounded in fresh data.
What can an AI agent actually do once it connects to your MCP server?
PolicyLayer audited 2,031 MCP servers in June 2026 and found 31,000 tools — 7.8% of which can permanently delete data or move money. And 40% of internet-facing MCP services have no authentication at all.
Why Is Your Website Invisible to AI Crawlers? Most Teams Are Getting This Completely Wrong
97% of llms.txt files across 137,000 domains got zero AI requests in May 2026. Most teams are solving the wrong problem while the real visibility gap grows.
Which AI bots are actually crawling your site in 2026 — and are any of them sending traffic back?
AI crawler traffic grew 757% in 2024, but the mix keeps shifting. Bytespider fell 83% while Meta-ExternalAgent jumped 733% — and training bots still account for most of that volume, returning almost no referral traffic.
Is the AI Bot Crawling Your Site Actually Going to Send You Any Traffic?
Eight in ten AI crawler requests to your site are for model training, not for answering someone's question and sending them your way. Understanding the three-bot model changes everything about how you read your traffic data.
43% of teams say AI search is a priority. Why do only 14% actually measure it?
43% of marketing teams say AI search is a core priority in 2026. Only 14% track whether their brand appears in AI-generated answers. Here's the measurement stack that closes that gap.
Is the robots.txt you wrote in 2023 still working for you in 2026?
One in four of the top 1,000 websites blocks GPTBot — but GPTBot is a training crawler, not a search indexer. The bot that drives AI search citations uses a different user-agent, and most sites have no rule for it at all.
Which AI bots are actually crawling your site — and why does the answer keep changing?
Fifty billion AI crawler requests hit the web every day, but 97% of them aren't sending anyone to your site. Here's what the mid-2026 data shows about which bots are crawling, what they want, and why the mix keeps shifting.
Which AI bots are actually crawling your site — and how many of them ever send you a visitor?
Training crawlers now account for over half of all AI bot requests, and they send back almost nothing. Here is what your server logs reveal that your analytics tools won't.
Who's Really Crawling Your Site — and Are Any of Them Sending You Visitors?
AI crawlers now top 20% of verified bot traffic — but most of them return fewer than one referral per thousand crawls. Here's what the June 2026 bot mix actually looks like and which bots are worth your attention.
Is your MCP server actually safe to put in front of real AI agents?
97 million monthly SDK downloads, 10,000 registered servers, 28% of Fortune 500 companies using it. But 86% of those servers live on someone's laptop, and only 8.5% implement OAuth. So what's actually happening with MCP in production?
Is your website invisible to AI crawlers without realising it?
Over 500 million GPTBot requests on one network, zero JavaScript executions. If your site renders content client-side, most AI crawlers see a blank shell — and the fastest-growing ones are exactly those that cannot render.
Are the AI Bots Crawling Your Site Actually Worth Letting In?
Bots crossed 57.5% of HTML web traffic in June 2026. But when ClaudeBot crawls nearly 24,000 pages per referral while GPTBot sits at 1,276:1, the real question is which bots are worth the bandwidth.
Which AI bots are actually hitting your site — and should you care about any of them?
Over half of AI crawler requests in 2026 go to training datasets, not search. So which bots are actually hitting your site — and does any of it ever come back as referral traffic?
Do any AI assistants actually pass referrers — or are they all appearing as direct traffic?
Testing shows Copilot passes zero referrers. Paid ChatGPT in-content links carry noreferrer. And Google AI Mode had a weeks-long bug making all its traffic look organic. Here’s the per-platform breakdown.
AI Crawlers Are Hitting Your Site Thousands of Times a Day — So Why Aren't You Getting the Traffic?
Bots now generate 57.5% of web traffic — machines outnumber humans online for the first time. GPTBot visits your site 857 times for every reader it sends back. Here's what the actual data says about which bots are worth caring about.
Why do 97% of MCP tool descriptions fail — and is fixing them actually worth it?
97% of MCP tool descriptions contain at least one quality problem. Two recent papers measured the impact on AI agent performance — and the results are more complicated than you would expect.
Why Are 97% of llms.txt Files Getting Zero Bot Requests?
97% of llms.txt files on the web got zero bot requests in May 2026 — yet adoption has grown 5x in a year. So what's actually going on, and should you bother shipping one?
Does any AI crawler actually read your llms.txt? The data says almost certainly not.
Ahrefs studied 137,000 domains in May 2026 and found 97% of published llms.txt files were never requested by any bot. Here's why adoption figures mislead, and what actually matters.
Why Does Only 2.6% of AI Bot Traffic Actually Matter to Your Business?
AI bots hit your site in record volumes — but just 2.6% of requests come from real-time fetches that could ever drive a referral click. Here's what the other 97.4% is actually doing.
Why would the same AI bot crawl your site once a month — then spike to 39,000 hits a minute?
Most site owners treat all AI crawlers the same way. But training bots and real-time fetchers behave nothing alike — and confusing them is costing you visibility in AI search.
Why Is Your AI Bot Visitor Crawling 24,000 Pages and Sending Back Exactly One Click?
Most AI crawlers are taking without giving. One leading training bot crawled nearly 24,000 pages per single referral in Q1 2026. Here's the breakdown and what it means for you.
Is GA4's New AI Channel Actually Showing You Your Most Valuable Traffic?
GA4 launched a native AI Assistant channel in May 2026. But with 35-70% of AI sessions hiding in Direct and Google's own AI invisible inside Organic Search, how much of your AI traffic can you actually see?
Does Any AI Agent Actually Know Your MCP Server Exists?
73,799 MCP servers are listed across the major registries as of July 2026. But the spec that lets AI agents find them automatically is still being finalized. So who actually knows your server exists?
Which AI bots are actually sending you traffic — and which are just hoovering up your content?
GPTBot crawls 1,700 pages for every referral it sends back. The leading training bot's ratio is 73,000 to one. Here's what the real data says about which AI crawlers are worth optimising for.
Why Is ClaudeBot Crawling Your Site 71,000 Times for Every Single Visitor It Sends?
ClaudeBot crawled 70,900 pages per referral in June 2025. GPTBot: 1,700:1. Here's the bot-by-bot breakdown of what AI crawlers are actually doing — and which ones are worth letting in.
Training Bots vs. Retrieval Bots: Has the AI Crawler Mix Shifted More Than You Think?
Bots now account for 57.5% of HTML web traffic — the first bot majority. But the real shift is inside: training crawlers are losing share to real-time retrieval bots that want fresh content, not archives.
Are AI Crawlers Actually Reading Your robots.txt?
60% of reputable websites now block at least one AI crawler — up from 23% in 2023. But documented cases of undeclared browser-impersonating bots raise a harder question: how much of that control is real?
Which AI Crawler Actually Earns Its Keep? The Crawl-to-Referral Gap, Explained
ClaudeBot reads 11,122 pages to send back one visitor. GPTBot needs 857 crawls per referral. Here's what that asymmetry means for your content strategy.
What Happens When You Block AI Crawlers? News Publishers Just Found Out the Hard Way
82% of news publishers now block at least one AI crawler — and a peer-reviewed Wharton study found those that did lost 7% of weekly traffic within six weeks. Here is what the data actually shows.
How Much AI Traffic Is Your Analytics Actually Catching?
ClaudeBot crawls 13,528 pages for every click it sends back. When those clicks arrive, only 9% show up in GA4. Here's what's hiding in your Direct channel — and how to measure all three layers.
Why Are AI Agents Calling Your MCP Tools Wrong?
97% of MCP tool descriptions have quality defects that cause AI agents to pick the wrong tool or pass wrong parameters. Here is what is actually going wrong and how to fix it.
Why Is Your Site Invisible to AI Search? The JavaScript Rendering Problem Nobody's Talking About
A Vercel analysis of over 500 million AI crawler requests found not a single instance of JavaScript execution. If your site uses client-side rendering, most AI search engines see just an empty shell where your content should be.
What are AI bots actually doing on your site — and is any of it helping you?
57.5% of web traffic is now bots, but most of that AI crawling won't put a single visitor in your analytics. Here's what the 2026 data shows about training crawlers, browsing agents, and the gap between.
1,800+ MCP Servers Accept Requests Without Any Auth — What's Actually Exposed?
A February 2026 analysis of 1,412 production MCP servers found 38.7% had zero authentication. When researchers looked inside, they found confirm_transfer, get_kyc_status, and get_resumes sitting open to anyone.
GA4 Added an AI Traffic Channel Last Month. But Is It Actually Showing You Everything?
Perplexity drove 15.1% of measurable AI referral traffic in H1 2026 — yet GA4's new AI Assistant channel doesn't include it. Here's what each analytics approach actually captures, and what to do about the gaps.
What's Actually Crawling Your Site? The AI Bot Mix in Mid-2026
More than half of all web traffic is automated, and the AI crawler slice grew 187% in 2025 alone. Here's who's showing up, what they want, and why the training-vs-agent split changes everything.
GPTBot Downloads JavaScript on 11.5% of Crawls. It Executes Zero of Them.
GPTBot fetches JavaScript files on 11.5% of its 500M+ logged crawls; execution events across that dataset: zero. Here is the complete 2026 rendering tier map for the major AI crawlers now driving indexing decisions.
70% of AI-Driven Visits Have No Referrer. The Server-Side Detection Hierarchy That Recovers Them.
GA4’s new AI Assistant channel captures roughly 29% of AI-driven sessions. A 446,405-session nginx log study found 70.6% arrived with no referrer header — here’s the server-side detection stack that recovers the rest.
76% of MCP Tool Schemas Score an F. Loading the Other 24% Still Eats Your Context Budget.
ToolBench graded 218,422 MCP tool definitions in March 2026: 76.6% scored F. Meanwhile three production MCP servers consumed 143K of 200K available context tokens before a single query ran.
OAI-SearchBot's 3.5× Surge and the Crawl-to-Refer Gap
OAI-SearchBot crawl events grew 3.5× after the August 2025 AI model release. Yet AI crawlers still take thousands of crawls to generate a single referral. Here is what the data shows.
llms.txt Has 10% Adoption and No Proven Effect on AI Citations
Among the 50 domains most cited by AI assistants, only one publishes an llms.txt file. A 300,000-domain study found no correlation between the file and AI citation frequency. Here is what the data shows.
Bytespider Tripled Its Share in 90 Days. Your robots.txt Hasn't Caught Up.
An unknown crawler hit 10.5% of AI bot traffic in three months, yet appears in only 4.23% of robots.txt DISALLOW rules — the widest blocking gap of any top-five AI bot in 2026.
Nine Out of Ten AI Crawler Requests Will Never Return a Citation
Only 9.3% of AI crawler traffic in May 2026 is search-purpose — the kind that can generate a referral. The other 90.7% is extraction. Here is what that split looks like across your log files.
1 in 18 AI Crawler Requests Is Fake: What Your Server Logs Can and Cannot Tell You
5.7% of requests presenting an AI crawler user-agent string are spoofed — ChatGPT-User runs at 1:5. Here is what each server-side signal reliably detects, where each fails, and how to combine them.
10,000 MCP Servers, One Fragmented Registry: The Discovery Gap Holding Back AI Tool Integration
The MCP ecosystem counts 9,600+ public servers. Independent crawlers found 17,400. The gap reveals a structural problem: no standard mechanism lets AI agents discover MCP endpoints without first establishing a live connection.
AI Crawler Failure Modes: 57% of Top Sites Invisible, Lazy Load Fails for All
A May 2026 audit of the top 1,000 most-visited sites found 57% serve content only after JavaScript runs. The AI crawlers now generating referral traffic execute none of it. Here is the failure-mode breakdown.
Analytics Platforms and AI Referral Traffic: What Each One Misses in 2026
GA4's AI Assistant channel launched May 13, 2026 with no historical backfill. Matomo 5.8 ships dedicated chatbot reports. Neither captures mobile app dark traffic. Here's what each platform sees—and what it doesn't.
41% of Production MCP Servers Have No Auth — and Only 8.5% Meet the Spec
Forty-one percent of official MCP registry servers have no authentication. Only 8.5% implement OAuth 2.1, the standard the spec mandates. Censys counted 21,000+ internet-accessible services by May 6. Here is the data and what to fix first.
AI Bot Traffic in 2026: Training Crawlers, Scrapers, and the Referral Engine That Actually Pays
New cross-dataset research reveals a striking gap: the bot sending 13,500 pages of crawl requests returns just one referral visit. Here is what the 2026 data says about who is really hitting your site.
41% of Pages Have JSON-LD — AI Crawlers Only Read the Server-Rendered Half
71% of audited sites use at least one schema type, but only 22% pass validation cleanly. AI crawlers can only read the server-rendered fraction. Here is the adoption gap and what to do about it.