Which AI Bots Are Actually Reading Your Site — And What Do They Want from It?
GPTBot hits some sites 4,200 times a day, but training crawlers now account for less than half of all AI bot traffic. Here’s who’s actually in your logs in 2026 and what each bot is after.
What if the AI bot hitting your site the hardest isn’t the one you’ve been worrying about?
Check your access logs for the past 24 hours. If GPTBot is there — and it probably is — you’re likely looking at somewhere between 1,000 and 4,200 requests just from that single crawler. A 30-day study across twelve production sites (DigitalApplied, April 2026) put GPTBot’s average at 4,200 daily hits per site. ClaudeBot averaged 1,800. PerplexityBot came in at 980. That’s three distinct bots, three distinct behaviours, and a combined footprint your server is handling whether or not you’ve thought about it.
Where does this data come from?
The per-bot figures come from DigitalApplied’s 30-day site log analysis published in April 2026, covering twelve production sites across B2B SaaS, e-commerce, agency, and publisher categories. The broader traffic-mix percentages come from HUMAN Security’s 2026 State of AI Traffic & Cyberthreat Benchmark Report, which analysed over one quadrillion interactions observed across 2025. CDN-level traffic monitoring from mid-2026 fills in the overall automated-vs-human picture.
Is training traffic even the majority anymore?
Ask most developers what “AI bot traffic” means and they say something about training crawlers — large batch jobs collecting content to build or update AI models. That was the dominant reality in 2025. It’s no longer the whole story.
By the first half of 2026, dedicated training crawlers accounted for 48.5% of AI bot traffic — down from a position above 67% twelve months earlier. Mixed-purpose bots (those doing training alongside real-time retrieval) had grown to 40.2%. Dedicated search and retrieval crawlers took 8.5%, and agentic bots — software acting on a specific user’s real-time task — held 2.3%.
The small percentages for search and agent bots are misleading without context. HUMAN Security’s report found that agentic AI traffic grew 7,851% year-over-year through 2025. AI scraper traffic — the inference layer feeding live products like answer engines — grew 597% over the same period. The 8.5% and 2.3% slices of today’s pie are the fastest-growing parts of it, by a significant margin.
How different are GPTBot, ClaudeBot, and PerplexityBot, really?
In terms of raw volume, quite different. But the more useful difference is in cadence and intent.
GPTBot revisits high-traffic pages approximately every 2.4 days, consistent with a training pipeline that prioritises content freshness. ClaudeBot revisits on roughly a 6.8-day cycle, reflecting a different training cadence. PerplexityBot concentrates heavily on recently-updated pages — ones updated in the last 30 days are crawled 2.3× more often than static content — which makes sense for a product that serves live search answers and needs current information, not archival bulk.
A single WAF rule or robots.txt Disallow treats all three identically. But blocking GPTBot means opting out of training corpus coverage; blocking PerplexityBot means opting out of the referrals that follow when your content answers a user’s query. They’re different tradeoffs, and the infrastructure to distinguish them is largely already there — it’s just unused.
What does each bot actually see when it arrives?
This is where most sites have a gap they haven’t measured. All three bots send HTTP GET requests without executing JavaScript. If your pages are client-rendered shells — an <html> tag, a <div id="root">, and 150 bytes of visible text — those bots see nothing useful, regardless of how many times they crawl you.
The log study found that pages with full server-side or static rendering received between 9× and 46× more AI crawler attention than equivalent URLs serving near-empty client-rendered HTML. For e-commerce sites specifically, server-rendered product pages received 3.4× more visits than hydration-dependent equivalents at the same URLs.
There’s a related complication: PerplexityBot has been documented running behind undeclared user-agent strings — a generic browser UA on IP ranges traceable to its parent company — when its declared bot is blocked by robots.txt. A user-agent-based block may not stop all traffic from the same source, and access-log analysis that relies only on user-agent matching will undercount it.
So what should you actually do about this?
Three questions are worth answering in roughly this order.
First: what does your server actually return when an AI bot requests your most important pages? The fastest check is curl -A "GPTBot/1.2" https://yourdomain.com/your-key-page and counting the readable words. Under 500 words on a page your users see as content-rich means your content isn’t feeding any AI system in a useful form, regardless of how many crawl hits you’re getting.
Second: which bots are actually in your logs, and at what volume? Standard analytics platforms don’t capture this — the JavaScript tag never fires. A 48-hour slice of raw server access logs, filtered by known AI user-agent strings, tells you more than months of analytics data about actual bot behaviour on your site.
Third: are you treating all AI bots the same when they have fundamentally different purposes? High-frequency training crawlers benefit from comprehensive sitemap coverage and consistent, crawlable HTML. Search and retrieval crawlers benefit from recently-updated, well-structured content that gives clear answers. Getting your pages into shape for one category generally helps the other — but the edge cases around caching strategy, rendering approach, and robots.txt granularity are worth thinking through separately once you know who’s visiting.