Is Your AI Bot Traffic Actually Worth Anything?
Most AI crawlers scrape your site hundreds of times for every visitor they send back. Here's the data on which bots are actually worth allowing — and which are pure server overhead.
Here's a number worth sitting with: one of the most active AI training crawlers in mid-2026 is generating 1,917 requests to websites for every single visitor it sends back. Google's crawler does a similar job at roughly 4.6 requests per referral. So maybe the question isn't just "are AI crawlers hitting my site?" — it's "what are they actually giving me in return?"
How was this data collected?
The figures here come from two sources. Crawl volumes come from CDN-level HTTP request log analysis published by major infrastructure providers through mid-2026, tracking user-agent traffic across millions of sites. Referral traffic figures come from SEO research firms aggregating analytics data across large publisher networks, tracking sessions where the referrer header matched a known AI assistant origin.
Both have limits. The referral data misses a large share of AI-driven traffic that arrives with no referrer at all — we'll get to that. Crawl-volume data only captures bots that identify themselves honestly. Together though, they're considerably more useful than anything a standard analytics dashboard will show you.
Who's actually in your logs right now?
The AI crawler list has gotten longer than most people's robots.txt has kept up with. GPTBot remains one of the highest-volume crawlers. ClaudeBot — which surged over 800% in crawl volume between November and December 2025 — is now a substantial presence across the web. Bytespider, the crawler tied to ByteDance, grew its crawl volume 138% between November 2025 and February 2026, and now accounts for roughly a third of all AI training crawl activity, according to BrightEdge research. Most site owners have never explicitly allowed or blocked it.
Worth flagging: the AI crawler ecosystem has split into two meaningfully different categories. Training crawlers — GPTBot, ClaudeBot, Google-Extended, Bytespider — pull content to improve future model versions. They work in batch cycles and aren't fetching your page because someone just asked a question. Real-time search agents — OAI-SearchBot, PerplexityBot, and their equivalents — retrieve pages live when a user query needs a source. Those are the bots that drive citation traffic.
If you've written a blanket block on all AI user-agents, you've almost certainly blocked both categories. That means staying out of future training datasets (which might be fine) and also declining to appear in AI-generated answers (which is probably not what you intended).
What's your actual return on that crawl traffic?
The crawl-to-referral ratio tells you how many requests a bot makes before generating one human visit. It's the clearest way to measure how much server overhead each crawler is actually earning.
PerplexityBot sits at 225:1. GPTBot — associated with the ChatGPT search product — improved sharply from 1,104:1 in July 2025 to 647:1 in June 2026, and then down further to 251:1 in July 2026. That trajectory suggests a crawler cleaning up its act as its parent product matures and depends more on referral trust. ClaudeBot is the main outlier at 1,917:1 — nearly 2,000 requests for every single visitor returned. Whether you rate-limit it or absorb the overhead is a call you should be making deliberately rather than by default.
Across the board, AI crawler traffic is up substantially year-over-year. GPTBot alone grew 305% between May 2024 and May 2025. AI crawlers now represent roughly 22% of all bot HTTP requests, up from around 10% two years ago. That growth is exactly why the crawl-to-referral ratio matters more now than it did when AI bots were a rounding error in your access logs.
Which platforms actually send traffic?
Crawl efficiency matters, but so does total referral volume. If you're going to prioritise which AI systems to optimise for, the platform breakdown is what you need.
ChatGPT accounts for roughly 75% of trackable AI referral sessions — an outsized share given how much attention every other AI product receives in the press. Perplexity, despite positioning itself as a search engine with high citation frequency, generates only about 7% of actual referral visits. Google's AI search features account for around 12%, despite being a newer entrant to the citation game.
The practical read: if driving AI referral traffic is a goal, optimising for ChatGPT citations is the single highest-leverage move you can make. That means clean server-side HTML, structured content that loads and parses quickly, and pages that respond fast enough for a real-time live-fetch to complete without timing out.
What about the traffic you can't see?
Here's the gap that makes all the above numbers conservative: somewhere between 35% and 70% of AI-driven visits arrive without a referrer header. They land in "Direct" in your analytics. You have no idea whether they came from an AI recommendation, a Slack message, or someone who typed your URL from memory.
The mechanism is straightforward. Most AI assistant interfaces are chat windows, not browser pages. A user sees a URL in a ChatGPT response, copies it, and opens a new tab. No referrer gets passed. Your analytics logs a Direct session. A January 2026 study from SparkToro found that only 12–18% of Perplexity citations actually translate into click-through traffic — meaning more than 80% of the times your site gets cited in an AI answer, nobody visits.
Those invisible citations aren't worthless — they influence how AI systems represent your brand in future answers — but standard analytics won't surface them. What you can improve: set up a custom channel group in your analytics platform with regex patterns covering known AI assistant domains, and keep it updated as new platforms emerge. That at least prevents the attributable sessions from washing into "Direct" and giving you a false read on organic performance.
So what's actually worth changing?
Three things are actionable right now without any specialist tooling.
First, audit your robots.txt against the training/search split. Allowing OAI-SearchBot and PerplexityBot while blocking or rate-limiting GPTBot and ClaudeBot is a completely valid strategy — they serve different purposes, and there's no rule that says you have to treat them the same.
Second, set up a crawl-volume alert in your log monitoring. The late 2025 ClaudeBot surge caught a lot of site operators off guard. An alert that fires when any single user-agent exceeds a threshold of requests per hour gives you early warning before unexpected bot traffic becomes a load problem.
Third, don't assume your analytics platform's built-in AI channel group captures everything — it doesn't. Maintaining your own custom regex list for AI assistant domains, updated as new bots arrive and old ones rebrand, is the only way to keep your attribution picture close to reality.
The AI crawler landscape is moving fast enough that the specific ratios here will look different in six months. The underlying pattern — more crawlers, improving referral efficiency from the mature ones, persistent attribution gaps — is likely to hold even as the numbers shift. What matters now is having the monitoring in place to see when something changes before it shows up as a surprise in your traffic report.