Three types of AI bot are hitting your site — and only one of them follows your rules

Somewhere between mid-2025 and now, the idea that you could manage "AI crawler traffic" as a single category fell apart. By July 2026, AI bots account for 26.8% of all verified bot traffic — but that number bundles together three very different populations that hit your servers at different rates, for different reasons, and with entirely different relationships to your access rules.

Which one is actually responsible for that spike in your server logs? And more importantly, which one are you even able to control?

Where this data comes from

The numbers in this post draw on three separate published analyses: a 24-million-request server log study covering January through March 2026 that compared AI crawler volumes to traditional search bots; a 14-day first-party log breakdown examining per-bot request frequency and return rates; and aggregate data from crawler analytics platforms tracking AI bot behaviour across millions of sites through July 2026. Where figures overlap across sources, the estimates are consistent.

What are the three categories?

The clearest structural change in 2026 is that the major AI assistant providers have deliberately separated their crawlers into distinct user-agents for different purposes:

Training crawlers (GPTBot, ClaudeBot, CCBot) batch-fetch content to update model weights. They run on irregular schedules — weeks between visits to the same URL is typical. They're the ones most likely to vacuum up copyrighted content, which is why 79% of top news publishers specifically target them in robots.txt DISALLOW rules.

Search indexers fetch pages on a regular basis to power AI-generated answers. Several providers now run a separate bot specifically for this purpose — distinct from their training crawler. That separation matters because it means you can potentially block one without blocking the other.

Agent crawlers are a different animal entirely. When a user asks an AI assistant to "look at this page" or follow a link in conversation, the resulting request comes from a real-time browsing user-agent. These hit your site on behalf of live human users asking questions right now, not batch jobs running in the background.

Here's the breakdown of AI crawler traffic by purpose category for H1 2026:

Training and mixed-purpose crawls still dominate raw volume, but the search and agent categories are growing fastest — search crawling was up 48% year-over-year by mid-2026, and the agent category surged 2,825% year-over-year across 2025.

Who's actually showing up the most?

The crawler leaderboard shifted significantly over the past year. By July 2026, ClaudeBot overtook GPTBot as the top AI training crawler by traffic share, finishing the month at 16.28% of AI-bot traffic to GPTBot's 9.74%. A year earlier those numbers were essentially reversed.

GPTBot's share contraction doesn't necessarily mean less activity — the underlying volume of AI crawler traffic is growing across the board. What it means is that the ecosystem has diversified significantly. PerplexityBot, for example, remains tiny in absolute terms but recorded the highest growth rate of any major AI crawler in 2025 by a wide margin. The field is more fragmented than it looks from the top-line percentages, and that fragmentation is accelerating.

The practical problem here: if you're making crawler decisions by blocking the single biggest bot, you're optimising for a snapshot that shifts every quarter.

Does blocking actually work?

The compliance picture for robots.txt is messier than most site owners realise. Studies analysing 2025 server logs found that up to 72% of AI crawlers violated robots.txt rules at some point. More specifically: 39.5% of sites that explicitly blocked GPTBot were still serving it anyway — whether through crawler non-compliance, CDN misconfigurations, or other gaps in the enforcement chain.

Looking at which user-agents appear most frequently in DISALLOW rules as of Q1 2026:

GPTBot leads the blocking charts, followed closely by CCBot and ClaudeBot. But notice what's absent from that list: agent crawlers. The real-time browsing bots used by AI assistants don't typically appear in DISALLOW rules, because blocking them via robots.txt would be semantically equivalent to blocking an ordinary user's browser session.

That's not a loophole — it's by design. Documentation from the providers behind the major agent crawlers explicitly states that user-triggered fetches, where a person asks an AI assistant to read a specific URL, aren't covered by training-data restrictions. TollBit's State of the Bots report covering H1 2026 confirmed that agent crawlers are reaching pages that have explicit DISALLOW rules, and that's largely working as intended.

What should you actually do about this?

The three-category split changes what "managing AI crawler traffic" means in practice. Here are the decisions worth separating out:

Training vs. search is now a legitimate distinction you can make. Because several providers publish separate user-agents for training and search-indexing purposes, you can block training crawls while still appearing in AI search results. If you care about visibility in AI-generated answers but not about contributing content to model training, check whether the crawlers you care about publish distinct user-agent strings for each purpose — many now do.

Assume robots.txt is a policy statement, not a guarantee. The 39.5% non-compliance rate for disavowed crawlers means monitoring needs to sit alongside your rules. Watch your server logs for access from user-agents you've blocked. If a crawler keeps hitting you after a DISALLOW entry, the next step is infrastructure-level blocking — rate limits or IP range blocks against verified crawler ASNs. That's a very different workflow to adding a robots.txt line.

Agent crawlers need a content strategy, not an access-control strategy. You can't practically block real-time browsing user-agents without impacting genuine users, and doing so reduces your citation rate in AI assistant responses. The more effective lever is ensuring that what an agent crawler sees is actually useful — pages that render cleanly without JavaScript execution, key facts in semantic HTML rather than buried in client-side state, and the structured data that helps AI systems understand context and source credibility. If you're only thinking about crawler access rules and not crawler legibility, you're solving half the problem.

The shift from treating "AI crawlers" as a monolithic category to recognising three distinct populations — each with different rules, incentives, and crawl frequencies — is one of the more significant infrastructure changes to hit web operations in the past year. Your monitoring and your controls both need to match that granularity.