Which AI Bots Are Actually Hitting Your Site — and Why They're Not All the Same?

The first time machines outnumbered humans on the web happened quietly in mid-2026. Automated requests now account for 57.5% of HTML traffic — the first time in internet history that non-human requests have held the majority. But lumping all those bots together misses the point entirely. Among AI crawlers specifically, the traffic has fragmented into at least three distinct job types that behave so differently from each other that a single blanket robots.txt rule handles none of them particularly well.

Where does this data come from?

Two main sources: a 30-day server log study by DigitalApplied covering request patterns across thousands of sites through June 2026, and Pebblous's crawler composition report comparing July 2026 against the same period in 2025. A third data point — blocking adoption — comes from the US Tech Automations Closing Web Index, which sampled top-ranked domains in July 2026. Where sources agree, the numbers are used directly. Where methodology differs, the discrepancy is flagged.

So which bots are actually showing up — and how hard are they hitting?

Four AI crawlers dominate AI-origin traffic at the site level: GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Their daily request volumes, measured across a 30-day median window, vary by nearly 8× between the most and least aggressive.

GPTBot runs at roughly 4,200 requests per day (median, per site), revisiting high-traffic pages approximately every 2.4 days. ClaudeBot trails at 1,800. PerplexityBot comes in around 980, and Google-Extended is the lightest at 540. The implication for anyone serving dynamically enriched content to AI bots: you're doing it at 2.3× the frequency for GPTBot compared to ClaudeBot. That asymmetry compounds quickly in compute and caching costs if you're rendering on the fly per request.

Path preferences are also consistent across reports. GPTBot skews heavily toward /blog/, /docs/, and /about/ paths — the structured prose sections of a site — rather than product listing pages, checkout flows, or dynamically assembled category pages. If your analytics only watch homepage and product traffic, you're missing where the real indexing work is concentrated.

Are all these bots actually doing the same job?

No — and this is where most blanket-blocking strategies go wrong. Each major AI company runs what amounts to a split fleet: a training bot that collects content for future model updates, a search-index bot that builds the retrieval corpus for AI answers, and a real-time agent that fetches pages on demand when a live user's query requires current information. The user-agents may look similar, but the jobs are fundamentally different, and so is the right policy for each.

The relative size of these categories is shifting fast.

Training-purpose crawling grew from 35.7% of all AI crawler traffic (July 2025) to 44.5% a year later. Real-time agent requests — still the smallest category — jumped from 1.1% to 2.7% over the same period, a 145% increase off a small base. Search-purpose crawling is up 48% year over year separately. The practical point: the category that's growing fastest is the one you have the least direct control over. Real-time agents fire when a user asks a question, which puts them in a definitional grey area where standard robots.txt semantics apply inconsistently, if at all.

Does blocking even work?

About 19.1% of top-ranked sites now block at least one major AI crawler via robots.txt, according to July 2026 data. Among news publishers specifically, that figure jumps to 82.4% — almost the entire sector has locked the door on training bots, at least declaratively.

But blocking has limits that are worth understanding clearly. Around 13% of AI bots observed in the wild ignore robots.txt entirely. The definitional uncertainty around real-time agents makes policy enforcement even murkier: is a fetch triggered by a live user query subject to the same robots.txt interpretation as a background batch crawl? Vendors answer this differently. The spec doesn't settle it.

Enforcement is ratcheting up in parallel. The rate of 403 responses to AI bots more than doubled year over year — from 3.63% to 8.56% of AI-bot requests. Sites are no longer just declaring policy; they're actively enforcing it at the edge. Whether enforcement is working is harder to measure, partly because the bots that ignore the declaration also tend to rotate user-agents and IPs.

What should you actually do about this?

The starting point is treating AI bots as distinct categories rather than a homogeneous threat. Training crawlers and real-time search agents have genuinely different economic relationships with your content. Blocking training crawlers is a defensible choice for paywalled, IP-sensitive, or proprietary content — your content won't appear in future model training data. Blocking search-index bots, by contrast, makes you invisible in AI-assisted search results and assistant citations, which is an increasingly significant source of new visitor discovery. Knowing which user-agent maps to which function — and updating that mapping as vendors add new strings — is now a basic operations task, not an edge case.

Second: the paths that AI crawlers hit hardest are exactly the ones that build inbound brand authority. Blog posts, guides, documentation, about pages — if these return a blank HTML shell because they're client-side rendered, no amount of robots.txt management solves the visibility problem. Pre-rendering these paths and serving static HTML to bot requests is an increasingly standard infrastructure pattern, not a niche optimisation.

Third: get visibility into your 403 rate. If your WAF or CDN is returning errors to AI bots in ways you didn't deliberately configure — catch-all bot rules, rate limits set for scraper defence — you may be blocking search-index bots alongside training crawlers without meaning to. Logging by user-agent, categorising by bot intent, and managing access explicitly is what distinguishes teams with a real AI visibility strategy from those flying blind.