Which AI bots are actually crawling your site right now?
ClaudeBot just overtook GPTBot for second place in AI crawler traffic, leading on all 31 days in July 2026. But 47% of that traffic is for training — not sending you visitors. Here is what the data shows.
Something shifted in July 2026. For the first time, ClaudeBot — not GPTBot — held the number-two spot in AI crawler traffic every single day of the month. Meanwhile, the share of crawl requests that will ever route a human to your site sat at 2.66%. So the question isn't just "which bots are hitting you?" — it's "why, and does any of it matter for your traffic?"
Where did this data come from?
The figures below draw on three sources: websearchapi.ai's monthly AI crawler reports covering May through July 2026, a 30-day server-log study across 12 production sites published by Digital Applied, and the US Tech Automations Closing Web Index released July 2026 tracking 10,000+ sites. Where sources disagree — usually because of sample composition — we note it.
So who's actually winning the crawl race?
The May-to-July trajectory is clean. GPTBot dropped from 11.48% of AI bot HTTP requests to 9.74%. ClaudeBot went from 9.73% to 16.28%, leading GPTBot on all 31 days of July.
Googlebot is still the largest single crawler at roughly 27% of AI-adjacent bot traffic — that's not changing soon. But the gap between the assistant-mode crawlers is wider than most site owners realise. GPTBot still crawls at higher raw volume per site (more on that below), but the market-share shift shows where the growth is.
What does it mean? Probably that real-time web retrieval in AI assistants is accelerating. When a user asks a question and the assistant fetches a live page, that shows up as an assistant-mode crawl, not a training crawl. More of those means more crawler activity from assistants that actually serve your content to users.
Does more crawling mean more traffic?
Not necessarily — and this is the part that trips up most interpretations of AI crawler data.
In June 2026, 47.4% of all AI bot traffic was for model training. Only 10.5% was for search indexing. Just 2.66% was for live user-request fetches — the kind that actually sends someone your way after an AI assistant answers a question.
Training crawls ingest your content as a data point in a model pipeline. That may matter for your brand's representation in AI-generated content months later, but it doesn't convert to traffic. A spike in GPTBot hits today does not mean a spike in AI assistant referrals.
And this split has been drifting. In January 2026, mixed-purpose crawlers (which include traditional search bots) led at 48.3%. By March, dedicated training crawlers had taken the plurality for the first time. The direction is consistent: training is the dominant intent behind AI crawler activity.
How often do they actually show up?
The 30-day log study found meaningful variation in crawl aggression:
GPTBot hits a typical production site about 4,200 times per day. ClaudeBot runs at about 1,800 daily hits. PerplexityBot sits around 980. These numbers vary significantly by domain authority and how frequently you publish — a news site updated hourly sees much higher rates than a static product page.
The relative ordering matters here: GPTBot crawls more than twice as often as ClaudeBot by raw hit volume, even as ClaudeBot leads on market share. That means GPTBot is probably doing more batch training work per site, while ClaudeBot's share growth is coming from assistant-mode fetches — which tend to be more targeted and less voluminous.
What's actually blocking them?
As of July 2026, 19.1% of top sites block at least one AI crawler in their robots.txt. GPTBot is the most-blocked: 8,432 sites have it listed. Yet 84.2% of all sites have no AI crawler policy at all.
The 84.2% number is the important one. Most site owners look at their analytics dashboards and see referral traffic from AI sources, but that view misses 97.34% of actual AI bot visits — the training crawls that never convert to traffic. A server-log-level view tells a much more complete story.
The nuance: blocking one crawler doesn't block the others, and it doesn't retroactively stop training data ingestion from before the block. The more useful frame is: which crawlers do I want to actively serve optimized content to, versus which ones get the raw page? That's a different question from "should I block AI crawlers?" and it has a more productive answer.
What does this mean for your site?
Don't conflate crawler volume with referral traffic. Server logs showing GPTBot hits every few minutes don't mean GPTBot is sending you users. Check your referral data separately — and know that most AI assistant traffic strips referrer headers anyway, so direct traffic is likely underreported.
The assistant-mode crawlers matter most for near-term referrals. The bots fetching pages in response to live user queries — rather than batch training crawls — are the ones that actually influence what AI assistants tell users about your site. ClaudeBot's growth trajectory suggests that cohort is expanding fast.
Your robots.txt posture is a business decision, not a default. The 84.2% of sites with no AI crawler policy are passively exposing their content to training pipelines while also missing the opportunity to serve optimized content to the assistant-mode crawlers that drive referrals. That's a gap worth closing deliberately, not by accident.