Which AI bots are crawling your site right now — and do any of them actually send you traffic?

One AI crawler harvested roughly 24,000 pages of publisher content for every single referral visit it sent back in early 2026. Googlebot, by comparison, runs at about 5:1. That 4,800× gap is the number that should probably be front-of-mind when someone suggests optimising for AI discovery — because not all AI crawlers represent the same kind of opportunity, and some represent almost none at all.

AI bot traffic grew 187% during 2025. Human web traffic grew about 3%. The raw scale of the crawling shift is real, but the averages hide who's actually showing up and what they're doing when they get there.

Data sources: Fastly's Q2 2025 Threat Insights Report (covering 6.5 trillion monthly requests across their CDN), network-level year-in-review data from a major CDN operator tracking verified bot traffic globally, and Akamai's “Securing the Agentic Storefront” analysis of AI bot patterns across e-commerce (July–December 2025). These three networks see enough global traffic to make the figures meaningful, though each reflects a different slice of the web.

Which bots are actually hitting your site?

The mix shifted dramatically in 2025 and kept moving into 2026. In May 2024, Bytespider — ByteDance's training crawler — accounted for 42% of AI crawler traffic. By May 2025, that had collapsed to 7%. GPTBot ran the opposite direction: from 5% to 30% of AI crawler traffic over the same period, a 305% year-over-year jump. ClaudeBot grew steadily from about 6% to 10%. The dominant AI crawler of a year prior is now a minor player; the one everyone talks about is the biggest one.

By July 2026, the rankings had shifted again. ClaudeBot led on all 31 days of that month at 16.28% of AI-bot traffic, with GPTBot at 9.74%. The landscape is still in motion — any strategy built on today's bot mix will need revisiting in six months.

In raw verified bot volume, Googlebot still accounts for more than 28% of all verified bot requests. GPTBot sits second at roughly 7.5%. AI crawlers as a category make up about 20% of verified bot traffic, while traditional search crawlers hold around 40%. So AI crawlers are significant and growing, but they haven't displaced search crawling — they've added to it.

Are training crawlers and real-time fetchers actually the same thing?

No, and this is probably the most practically useful distinction to understand.

Fastly's Q2 2025 analysis found that training crawlers — bots scraping content to build or update AI model training datasets — account for roughly 80% of all AI bot traffic by volume. Real-time fetchers, which fire when a user asks an AI assistant a question and the assistant needs to pull current information, account for the other 20%.

Among training crawlers, three provider families dominate: Meta (via Bytespider and Meta-ExternalAgent) leads at 52% of training crawler requests, followed by Google's crawlers at 23%, and GPTBot at around 20%. Together those three families account for 95% of training crawler volume. Among real-time fetchers, OAI-SearchBot and ChatGPT-User combined account for about 98% of real-time fetcher requests. PerplexityBot is in this category too, at a smaller share.

The growth picture is lopsided. Training crawlers grew 187% in 2025 — fast, but predictable in character. User-triggered AI crawling grew more than 15× over the same period, starting from a smaller base but accelerating much harder. Fastly recorded peaks above 39,000 real-time fetcher requests per minute.

The behavioural difference is worth noting too. Training crawlers run in sustained, periodic bursts — hundreds of pages in minutes, then quiet for hours, then restart. Real-time fetchers appear in tight spikes correlated with user query patterns. If you've seen mysterious traffic spikes in your server logs followed by long silences, a training crawler is the most likely explanation.

So do any of these crawlers actually send you traffic?

Here's where the crawl-to-refer ratio matters, and it's the number that should recalibrate expectations. For every page a bot crawls, how often does it generate a referral visit back to that site?

Googlebot: roughly 5:1. Makes intuitive sense — Google indexes your pages and routes search traffic back.

GPTBot: roughly 1,276:1 across the measured period. At scale, that means one referral visit for every 1,276 pages crawled.

ClaudeBot: between 10,300:1 and 24,000:1 depending on the measurement window. This crawler operates at very large scale and routes almost no traffic back to the sites it crawls. There's no consumer product that sends referral visits the way search does — the content goes in; visitors don't come back out.

The implication is fairly direct. Training crawlers are not a meaningful source of referral traffic. They take content; they don't send visitors. Real-time fetchers — OAI-SearchBot, ChatGPT-User, PerplexityBot — behave more like search engines because they're responding to live user queries and citing sources. Their crawl-to-refer ratios are far closer to Googlebot's, which is why they're worth treating differently.

What are they actually looking at?

E-commerce is currently getting the most AI bot attention. Shopping and e-commerce accounted for 31.58% of AI crawler requests in July 2026, with internet and telecom at 20.61% and consumer electronics at 19.19%. Those three sectors combined take about two-thirds of AI crawler traffic.

On format: 72.5% of what AI crawlers fetch is HTML — not PDFs, not JSON, not structured data feeds. This creates a concrete problem. Most AI crawlers don't execute JavaScript. GPTBot, ClaudeBot, Bytespider, and Meta-ExternalAgent all read raw HTML only. If your content loads dynamically after a JavaScript framework renders it, these bots are seeing a near-empty page or a loading skeleton. Googlebot renders JavaScript; most AI crawlers don't.

Training crawlers don't show strong preferences for specific page types — they're optimising for volume, not curation. They start at your homepage and fan out broadly. What they do prefer is content density: longer, richer pages contribute more to whatever model is being trained. Thin pages with minimal text get crawled at similar rates but don't add much.

Akamai's commerce analysis found that more than 70% of AI bot hits in e-commerce are training crawlers pulling product data, pricing, and inventory. If you're running a shop, there's a good chance a significant fraction of your bot traffic is training runs you're not being compensated for.

What should you actually do about this?

First, separate what you're optimising for. If you want to appear in AI assistant answers — the real-time fetcher category — you need pages that are readable by bots that don't execute JavaScript, semantically well-structured, and updated often enough that real-time fetchers re-index them. The referral dynamic here resembles search: the bot needs to understand your page to recommend it.

If you're concerned about bulk training scrapes, the crawl-to-refer data makes a reasonable case for blocking or rate-limiting them. The referral return on training crawlers is close to zero, and the crawl volume can be substantial enough to show up in your server load.

Second, check your JavaScript dependency before anything else. If your key content — product pages, articles, documentation — is rendered client-side, most AI crawlers aren't reading it at all. Pre-rendering to static HTML, whether server-side or via a reverse proxy that serves pre-rendered content to identified bot user-agents, is the most direct fix. It also benefits traditional search crawlers that can't or don't fully render JS.

Third, monitor bot user-agents separately rather than treating all AI traffic as one category. Training crawlers and real-time fetchers have different user-agents, different crawl patterns, and a very different commercial relationship with your site. Lumping them together means you might block OAI-SearchBot (which can refer traffic) because you're annoyed at GPTBot (which almost certainly won't). The 80/20 split exists, but the 20% real-time fetcher category is where any referral value concentrates.