Are You Blocking the Wrong AI Bots?

Nearly 6 in 10 AI bot requests hitting your website aren't from training crawlers scraping data for model datasets — they're from live retrieval and browsing bots that fire when someone asks an AI assistant a question right now. If you've been treating all AI crawlers as a single threat and blocking them wholesale, you may have cut yourself off from the exact traffic channel that was sending real readers your way.

That's not a prediction. It's what comes out when you run 48 days of server logs through a bot categorisation lens covering 16 distinct AI user agents.

Where does this data come from?

The numbers below are drawn from two main sources: a 48-day real-world server log study across multiple sites, and industry-level verified bot traffic data aggregated across tens of millions of websites. Bot counts use verified identification only — reverse-DNS confirmation or signed request headers — so spoofed user-agents don't inflate the AI totals.

What's actually hitting your server right now?

Break AI bot traffic into functional buckets and the picture shifts fast. Live in-session agent bots — user-agent strings that fire when someone explicitly prompts an AI assistant to visit a URL — accounted for 44.3% of AI bot requests in the log study. These are real-time, on-demand fetches triggered by actual human intent. Retrieval bots that ground AI answers in fresh content added another 15.1%. Training crawlers — the ones everyone's writing DISALLOW rules for — came to 29.6%.

So the majority of AI traffic is citation-relevant, not training-relevant. And that distinction matters enormously for how you manage it.

Here's the awkward bit: robots.txt training-crawler rules often don't apply to in-session user-directed requests. When a user asks an AI assistant to summarise a specific URL, providers frequently treat that fetch as user-directed access rather than crawling, putting it outside the scope of standard training-bot restrictions. That means your carefully crafted DISALLOW rules might be missing nearly half the AI traffic you care about.

Which bots are winning the market share race?

The two dominant AI crawlers have been swapping positions all year. In April 2026, ClaudeBot led the AI bot share race at 11.69% versus 9.84% for GPTBot. May flipped it: GPTBot took 11.48% to ClaudeBot's 9.73%. By July, ClaudeBot pulled decisively ahead at 16.28% to 9.74% — and held that lead across every single day of the month.

For reference, Googlebot sits at roughly 48% of all verified bot traffic, still the dominant crawler overall. But AI crawlers as a combined category now account for around 26.7% of verified bot traffic (20.3% from AI crawlers, 6.5% from AI search bots), and that gap is narrowing.

The volatility is what's genuinely surprising here. A swing of several percentage points in market share between two crawlers over three months means your bot traffic mix isn't stable. Which AI assistant is shipping new features, which is in a heavy indexing phase, which company is pre-training a new model version — all of these affect your server logs in ways that aren't visible unless you're watching user-agent telemetry on a weekly cadence.

How hard are site owners pushing back — and does it work?

AI crawlers have grown from roughly 10% of all bot traffic to 22% in about two years, with GPTBot alone growing 305% year-over-year. Search-purpose AI crawling is up 48% year-over-year. Bots now account for around 30.6% of all web traffic as of Q1 2026.

Site owners are responding by blocking more aggressively, and the numbers show it. AI bot requests hitting 403 Forbidden responses nearly doubled in a single year: from 3.63% in Q2 2025 to 8.56% in Q2 2026. GPTBot is the most-blocked AI crawler, appearing in 5.52% of all DISALLOW rules as of Q1 2026, with ClaudeBot at 4.88%.

But blocking is messier than it looks. Among websites explicitly blocking GPTBot in their robots.txt, 39.5% were still actively serving pages to GPTBot in practice. You can't write a DISALLOW rule and assume the problem is solved. You have to verify against your own server logs that the traffic actually stopped.

The consequences of getting this wrong cut both ways. 79% of top news sites block at least one AI training bot. But research published late 2025 found that publishers blocking AI crawlers via robots.txt saw a 23.1% decline in monthly visits and a 13.9% decline in human-only browsing — suggesting AI-driven referral traffic is significant enough that cutting it off has real collateral damage, even when that's not the intent.

What does this mean for how you manage your site?

The framing of "AI crawlers vs. your content" has stopped being useful. The more productive question is: which category of bot are you dealing with, and what does it actually want from your site?

Training crawlers fetch large volumes of pages and care about breadth, not recency. Retrieval bots fetch pages to answer specific queries — they care intensely about freshness, structure, and whether your content directly answers the question being asked. In-session agent bots behave most like a real user with a task — they visit single pages on demand, often care about whether content is fully rendered, and can drive direct referrals.

If your goal is to appear in AI assistant answers, blocking retrieval bots or agent bots defeats the purpose entirely. If your goal is controlling training data licensing, training crawlers are your actual target — but you need to verify your robots.txt is working, not just assume it.

The market share volatility also changes what a useful monitoring setup looks like. Static robots.txt audits and occasional crawl checks aren't enough when the bot mix can shift meaningfully in a single month. What you actually need is server log data broken down by user-agent, refreshed weekly, so you can see what's reaching your site and in what volumes — and catch changes before they compound.

The 39.5% compliance gap on GPTBot blocks is a useful gut-check for any assumption in this space. If you've blocked a bot but haven't confirmed via logs that it stopped, your block is a hypothesis, not a fact. Treat it like any other unverified assumption in your stack: test it first, then trust it.