Which AI bots are actually reading your site right now?

Pull last month's server logs from any site with meaningful crawler traffic and there's a decent chance you're seeing ClaudeBot in the top two — possibly ahead of GPTBot. Twelve months ago that order was reversed. So what's actually changed, and does it matter which AI crawler is winning your log file this month?

The short answer is: the shift is real, the causes are interesting, and the question you're probably asking is probably the wrong one.

Where does the data come from?

The market-share numbers in this post come from network-wide telemetry covering verified bot traffic across tens of millions of websites — the kind of infrastructure-layer dataset that captures crawler behaviour at scale rather than from a single site's perspective. Per-site crawl volumes come from a 30-day production server log study published in mid-2026 across roughly 800 production sites. I'm focusing on July 2026 as the most recent complete reporting period.

So who's winning right now?

In July 2026, ClaudeBot holds 16.28% of AI crawler traffic. GPTBot sits at 9.74%. A year earlier the order was inverted — GPTBot at 13.15%, ClaudeBot at 11.23%.

The month-by-month progression is even more revealing. April 2026 had ClaudeBot marginally ahead (11.69% vs 9.84%). May saw GPTBot briefly reclaim the lead (11.48% vs 9.73%). Then in June, ClaudeBot nearly doubled overnight to 19.77% while GPTBot barely moved, landing at 9.40%. July pulled ClaudeBot back to 16.28%, with GPTBot recovering slightly to 9.74%.

What does the volatility tell you? These aren't steady, ambient crawlers like Googlebot. AI training crawlers run in batches — when a lab kicks off a major training run, crawler traffic spikes hard for a few weeks, then subsides. The month-to-month swings in market share are essentially each company's internal engineering calendar made visible. ClaudeBot's June spike almost certainly corresponds to a training data collection run. Its July pull-back means that run ended.

Watching which bot is "ahead" month-to-month is a bit like watching two delivery lorries from different companies racing each other on the motorway. The race itself tells you nothing about where either is going.

Are ClaudeBot and GPTBot even doing the same thing?

This is where most discussions about AI crawler share go wrong — and where the answer actually matters for site owners.

By 2026, every major AI company runs at least two separate types of crawler with distinct user agents and fundamentally different purposes:

Training crawlers (GPTBot, ClaudeBot) collect content to update model weights. Your page becomes training data. You receive no referral traffic, no citation, and no way to know how your content was used.

Search and index crawlers (OAI-SearchBot, PerplexityBot) build the retrieval indexes that power AI search products. When they surface your content in an answer, they cite the source and can send a visitor back.

Real-time user agents (ChatGPT-User) fetch your page live when an actual user mid-conversation asks a question that requires it. These are high-intent, one-off visits.

The market-share figures above only measure training bots, because training accounts for most crawler volume. As of June 2026, 52% of all AI crawler requests across the network were training-related — up from just 22% in spring 2025. Training bots are growing faster than search bots, which means a larger proportion of AI crawler hits on your site are currently extracting value without returning any.

Most robots.txt setups don't distinguish between these categories. A site that blocks GPTBot to opt out of model training often still appears in AI search results — the search-specific crawler uses a different user agent that the disallow rule doesn't cover. Whether that's intentional or accidental varies by site.

Which bot is actually hitting you hardest by volume?

Here's where market share diverges sharply from practical impact. Despite its declining share, GPTBot still hits individual sites much harder than any other AI crawler: a median 4,200 requests per day, compared to 1,800 for ClaudeBot and 980 for PerplexityBot. GPTBot also revisits high-traffic pages every 2.4 days — the most aggressive recrawl schedule of any named AI bot.

In other words: "market share" measures how broadly a crawler visits sites across the web. Volume per site measures how aggressively it crawls when it does visit. GPTBot's drop in share hasn't translated into a lighter footprint on the sites it does crawl.

For engineering and infrastructure purposes, GPTBot is still the one most likely to show up as a spike in your request logs. If you're not caching bot responses separately, GPTBot's volume is the number most likely to appear in a capacity-planning conversation.

What's actually in your logs that your analytics can't see?

The bots you can identify by user agent — GPTBot, ClaudeBot, PerplexityBot — are the ones that announce themselves honestly. Well-known crawlers respect robots.txt, but there's a fast-growing category your clean log analysis won't capture: agentic traffic.

AI-driven agentic traffic grew 7,851% year over year according to the HUMAN Security 2026 benchmark. These are AI systems running automated workflows, completing research tasks, or browsing on behalf of users — and they typically arrive with browser-style user agents, not bot-declaring ones. They look like humans in your analytics because the AI is operating a browser rather than a dedicated crawler.

Your GPTBot and ClaudeBot numbers are real, but they're almost certainly a floor, not a ceiling. The harder-to-identify fraction is growing faster than the labelled fraction.

What should you actually do with this?

Separate training from search in your robots.txt. If you want to appear in AI search results without contributing to model training data, you can block the training-specific user agents while leaving the search-specific ones unblocked. Check that each disallow rule targets the correct user agent string — one rule won't cover both categories from the same company, because training and search crawlers use different user agents even when they're from the same operator.

Don't lose sleep over the monthly market share charts. The ClaudeBot/GPTBot gap will swing again when the next training run starts. What actually influences whether AI systems recommend your content is the quality of what they indexed when they did crawl — not which bot topped the chart this month.

Consider publishing llms.txt if you haven't already. Adoption currently sits at just 7.4% of sites, which means early publishers still stand out. Where robots.txt is binary — block or allow — llms.txt lets you specify which parts of your site you actively want AI systems to use, and with what context. With AI crawlers now accounting for 20.3% of all verified bot traffic and still growing, explicit crawl-intent signalling is becoming more practical, not less.

The bot landscape has fragmented significantly. There are now at least three functionally distinct categories of AI crawler hitting your site, from multiple operators, on different schedules, for different purposes. The monthly market share chart gives you a rough map of which companies are running training campaigns right now. What it can't tell you is whether those crawlers are actually seeing what you want them to see — and that's the more important question.