What Are AI Crawlers Actually Doing on Your Site — and Are Any of Them Worth Welcoming?

A single training pipeline can send half a million requests to your server for every user it eventually sends back. Most site owners assume AI bot traffic is mostly harmless curiosity — the reality is a 4,580:1 average crawl-to-refer ratio across the dominant AI assistants. That number hides the story though: you have three fundamentally different categories of AI bot hitting your site, and only one of them has any plausible path to sending you actual readers.

The data here is pulled from Fastly's August 2025 threat research covering mid-April through mid-July 2025 across their global CDN, Imperva's 2026 Bad Bot Report, and a 30-day production log study published in April 2026 tracking twelve live sites. Where sources disagree, the more conservative figure is used.

Which bots are actually in your logs?

Before you can answer "is AI crawler traffic good for my site?", you need to distinguish three things that get lumped together under that label.

Training crawlers operate in batch mode to harvest content for model training datasets. They hit your site on a schedule you don't control, index as many pages as they can reach, and generate most of the raw request volume in your AI-bot logs. Fastly's research found training crawlers made up roughly 79% of all AI bot traffic on their network in Q2 2025. Within that training-crawler category, Meta's AI bots dominated with 52% of requests — more than Google's crawlers (23%) and GPTBot (roughly 20%) combined.

Real-time scrapers fire in response to actual user queries. When someone asks an AI assistant to summarise a news article and the assistant fetches it on the spot, that's a real-time scraper. These bots are connected to genuine user behaviour — which gives them the only meaningful path to referral traffic. They're a smaller fraction of total AI bot volume but are growing fast.

Agentic browsers are the newest category: headless browser sessions acting on behalf of users for multi-step tasks like navigating checkout flows or extracting data from a knowledge base. Under 2% of AI bot traffic as of December 2025, but up 7,851% year over year. They interact with your site rather than just fetching it, which creates a distinct set of infrastructure requirements.

How has the mix shifted over 2025?

Training crawlers sat at roughly 90% of all AI-driven requests at the start of 2025. By December, Imperva's data puts them at 74%, with real-time scrapers climbing to 24% and agentic bots crossing 1.7%.

The composition shift matters more than the total growth number when it comes to infrastructure planning. Training crawlers tolerate high latency — they're not waiting on a human. Real-time scrapers are more sensitive to response time, since a slow page extends the wait for the actual user who triggered the query. Agentic bots need predictable page structures and don't play well with loading spinners or dynamic content that assumes a JavaScript-capable client. Each category needs a different response from your stack.

How often are they coming back?

More often than most site owners realise. A 30-day log study across twelve production sites measured average daily hit counts per site: GPTBot at 4,200 requests, ClaudeBot at 1,800, and PerplexityBot at 980. These are per-site figures, not distributed across a crawl frontier.

GPTBot revisits high-traffic pages roughly every 2.4 days. PerplexityBot returns every 1–3 days for active content. ClaudeBot's schedule is looser — 3 to 14 days — but its crawl rate approximately doubled between Q3 2025 and Q1 2026.

One sharp shift worth noting: after a major AI assistant product launch in mid-2025, OAI-SearchBot activity grew roughly 3.5x and now generates more log events than batch training crawlers on some sites. A major product release from an AI assistant company can reshape your bot traffic composition within weeks.

Which paths are they going after?

AI training crawlers are not distributed evenly across your URL space. GPTBot shows a consistent preference for /blog/, /docs/, and /about/ paths — text-dense pages with clear semantic structure. Product listing pages with dynamic content and thin static HTML attract fewer training-crawler visits than editorial content, even when those product pages represent your highest commercial value.

The most common failure mode is JavaScript-rendered content without a static HTML fallback. If your blog posts exist as React-hydrated shells where the article body appears after a useEffect, AI training crawlers likely see an empty page. Whatever remains visible when JavaScript is disabled is, roughly, what AI training systems see when they index your content. That gap can mean entire sections of a site are completely invisible to AI systems despite being heavily trafficked by humans.

What does this mean for how you manage the traffic?

Separate your optimisation by category, not by bot name. The setup that serves training crawlers well — rich static HTML, clear semantic structure, a permissive robots.txt for the bots you want — is not the same setup that serves real-time scrapers well (fast responses, clear canonical structure, schema markup that makes you citable). Treating all AI bots identically means you miss the goal for both.

Cache the bot-facing response. GPTBot sending 4,200 requests per day to your origin is a real infrastructure cost for near-zero referral return if you're not caching. Serving the pre-rendered static version to training crawlers cuts origin load dramatically with no accuracy tradeoff — they don't need the live state of your site, just the content. Real-time scrapers benefit from cached responses too: a faster response improves the end-user experience for whoever triggered the query.

Track the composition, not just the volume. AI bot traffic grew 187% across 2025, which sounds like a single trend. But the shift from 90% training crawlers to 74% — with real-time scrapers doubling their share — means the distribution across categories is changing faster than the total count. If your current monitoring buckets "AI bots" as one category, you're missing the signal that actually predicts whether any of that traffic translates into referrals.