What's actually in your AI bot traffic? The training/retrieval split your logs are missing

Your server logs are full of AI crawler hits — but fewer than 3% of all AI bot requests actually originate from a real user asking a real question. The rest? Training data collection: crawlers systematically working through your site to feed model pipelines, with no intent of surfacing your pages to a human. If you've been optimising for "AI visibility" without first understanding where your crawler traffic actually comes from, there's a good chance you've been spending effort on the wrong 97%.

Where does this data come from?

The numbers here draw from three places: a 30-day log study across 12 production sites published in April 2026 by Digital Applied; network-level bot traffic reports tracking per-user-agent volumes across Q1 and Q2 2026; and the user-agent documentation published directly by each major AI platform. Where figures differed between sources, we used the more conservative estimate.

So what kind of AI bot is actually hitting your site?

Through the first half of 2026, training crawlers consistently accounted for the single largest bucket of AI bot traffic — somewhere between 47% and 52% of all AI bot requests depending on the measurement window. Mixed-purpose crawlers, which share a pipeline for both indexing and retrieval, added another 35%. Search-mode bots — the ones actually responding to live user queries in real time — represented less than 10% of total AI bot requests. Agentic bots, those taking actions on behalf of a real user rather than passively harvesting content, made up under 3%.

The practical implication is uncomfortable: the bulk of the AI crawler traffic in your logs is training jobs. They are not a sign that users are being directed to you.

How aggressive are the big crawlers — and does raw hit count even matter?

A 30-day log study across 12 production sites found GPTBot averaging 4,200 daily hits per site — more than twice ClaudeBot's 1,800, and more than four times PerplexityBot's 980. By raw request count, GPTBot is the dominant AI crawler. But this is where the metric starts to mislead. GPTBot is a training crawler. High crawl frequency from a training bot tells you your content is being harvested for model training — it says nothing about whether a user will ever be directed to your site.

The number that actually maps to visibility is the crawl-to-referral ratio. In Q1 2026, ClaudeBot's training pipeline generated one referral for every 23,951 pages it crawled. PerplexityBot came in at approximately 111:1. Googlebot, for comparison, sits near 4.9:1 for traditional search.

PerplexityBot is more than 200 times more efficient at turning page crawls into visible traffic than ClaudeBot's training crawler. That isn't because Perplexity crawls more cleverly — it's because PerplexityBot's primary job is retrieval for live user queries, while ClaudeBot is batch-collecting content for future model training. The crawl volumes are enormous; the referral return on those crawls is not.

Why does each major platform run three crawlers — and why does your robots.txt probably only know about one?

This is where the majority of robots.txt configurations get it wrong. Both the ChatGPT platform and the AI lab behind ClaudeBot run at least three distinct user-agents, each with a different job — and the vast majority of sites treat all of them as a single thing.

ChatGPT platform bots:

• GPTBot — training data collection. Blocking this prevents your content from future training rounds, but has no effect on whether you appear in ChatGPT's live search answers.
• OAI-SearchBot — powers ChatGPT's web search feature. This is the crawler that indexes your content for retrieval in live user queries.
• ChatGPT-User — triggered by an actual user asking ChatGPT to read a specific page. The clearest signal that a real user is actively being sent to your content right now.

ClaudeBot family:

• ClaudeBot — training crawler. Blocking it removes your content from future model training, not from live answers.
• Search-index bot — indexes content for the AI assistant's search feature. The exact user-agent token is published in the platform's developer documentation.
• Live-fetch bot — on-demand retrieval triggered by a real user query. This is the log entry that confirms a person is actively being directed to your content.

What does this mean practically? If you've added Disallow: / User-agent: GPTBot to your robots.txt, you have not removed yourself from ChatGPT's search answers. You've opted out of model training. These are genuinely different decisions, and conflating them is the single most common configuration mistake in AI crawler management right now.

As of mid-2026, roughly 44.9% of prominent public websites block at least one major AI crawler — but a meaningful share of those are blocking training bots while assuming they've blocked everything that matters for answers.

What content are these crawlers actually retrieving?

Content type distribution matters too. HTML makes up 72.3% of what AI bots retrieve; JSON accounts for 6.9%, images 5.2%. By site type, shopping and e-commerce content attracts a disproportionate share of verified bot traffic — roughly 25.7% of all verified AI bot requests land on retail pages, most likely because structured product data is high-value for model training. Geographically, the US accounts for 53.5% of global bot traffic by volume, though the distribution of bot types varies by region: markets like Japan and Australia see a higher share of live-retrieval bots relative to training crawlers than European markets do, which changes how much of your AI bot traffic has any direct referral potential.

What should site owners actually do with this?

Start by separating your log analysis. AI bot traffic is not one thing — break it into training crawlers (GPTBot, ClaudeBot, Meta-ExternalAgent, Bytespider) and retrieval/search bots (OAI-SearchBot, ChatGPT-User, ClaudeBot's retrieval bots, PerplexityBot). The first group tells you about model training exposure. The second tells you about citation potential and visible traffic. If you're not segmenting these, you're averaging away the only number that matters.

Then revisit your robots.txt with that distinction in mind. If your goal is appearing in AI-generated answers, the bots you need to allow are the retrieval ones. Blocking OAI-SearchBot removes you from ChatGPT's live search results. Blocking GPTBot does not. Opting out of training while remaining visible in answers is achievable — it just requires knowing which user-agent does which job, and each major platform publishes that documentation.

Finally, stop using crawler hit volume as a proxy for AI visibility. 4,200 GPTBot requests per day is a training harvest rate, not a visibility score. The signal worth pulling from your logs is how often ChatGPT-User or ClaudeBot's live-fetch bot appear, broken down by URL. Those requests represent real users who received a link from an AI assistant and followed it. That's the number that actually connects your crawler traffic to revenue.