Is Your robots.txt Blocking the Wrong AI Bots?

Sixty percent of reputable websites now block at least one AI crawler in their robots.txt. That number has more than doubled since September 2023, when it stood at 23%. It's one of the fastest adoption curves in web crawler governance — and a lot of those blocks are pointed squarely at the wrong bots.

Data for this post comes from three places: an academic study published in the ACM Web Conference 2026 proceedings (arXiv: 2510.10315) that tracked robots.txt configurations across thousands of domains from September 2023 through May 2025; a BuzzStream analysis of 200+ publisher robots.txt files from mid-2025; and compliance data collected from 47 UK business websites monitored across October and November 2025.

How big has the AI-blocking wave actually gotten?

News publishers have led the charge. Of the top news sites running a robots.txt file, 79% now block at least one AI training crawler — the highest blocking rate of any content category measured. Zoom out to the broader reputable web, and you hit 60%. Compare that to 9.1% for misinformation sites. The gap matters, and we'll come back to it.

The growth rate is as striking as the current level. The shift from 23% to 60% happened over roughly 20 months. The catalyst was not one event but a cascade: high-profile AI content disputes, class action lawsuits, heavy trade press coverage, and the dawning realisation that model-training crawlers were scraping at scale with no compensation. Site owners reached for the most readily available control mechanism — a text file.

Are they actually blocking the right bots?

Here's the uncomfortable part. Among news publishers, PerplexityBot is blocked by 67% of sites — higher than GPTBot's 49.4% rate. PerplexityBot is a search and citation crawler, not a model-training crawler. Block it and your site does not appear in AI-powered search answers that route traffic back to you. Blocking GPTBot prevents your content going into model training — a reasonable call for many publishers. Those are not the same thing, and confusing them costs you visibility.

The confusion has a clear origin. In early 2024, Perplexity's crawling practices attracted public criticism after documented cases of ignoring robots.txt via rotating IPs. Publishers added PerplexityBot to their disallow lists, the news cycle moved on, and most of those rules stayed in place. Meanwhile the bot landscape kept splitting: every major AI assistant operator now runs at least two distinct crawlers. One handles model training. A different user-agent handles real-time search indexing. GPTBot and OAI-SearchBot are different tools doing different jobs for the same operator. Block one and you have done nothing to the other. Most robots.txt files written in 2023 and 2024 do not account for this — and a lot of site owners don't know the distinction exists.

Does the blocking actually work?

A compliance study run across 47 UK websites between October and November 2025 found that 72% of those sites had AI crawlers violating their robots.txt directives despite having explicit disallow rules. The average site recorded 156 violation requests from blocked bots during the three-week monitoring window. A broader Q4 2024 dataset found that 30% of AI scrapes in that period ignored robots.txt permissions outright, with at least one major AI assistant's user-initiated crawler hitting a 42% non-compliance rate.

The enforcement gap is not a bug — it is how the protocol works. robots.txt carries no technical weight; it is a voluntary signal. Well-resourced crawlers from the largest AI operators generally comply. But lower-tier scrapers, crawlers using rotating IPs, and bots running under generic browser user-agent strings see no reason to honour a polite text file. The crawlers you are most worried about are exactly the ones least constrained by a disallow rule.

What is the misinformation angle doing here?

The arXiv study's most pointed finding: reputable sites block AI crawlers at 60%; misinformation sites do so at 9.1%. If you are building a model or a search index and you crawl what is available, you read from a pool that is systematically lighter on credible sources. Reputable publishers have been removing themselves from the accessible web for AI at roughly seven times the rate of low-quality sources. The structural asymmetry is real and it is growing with every new blanket disallow rule added by a site that does not want to be scraped.

So what should you actually change?

The single most actionable step is separating your blocking rules by crawler type, not by operator name. AI training crawlers and AI search-indexing crawlers are distinct user-agent strings — even when operated by the same company. If you want to prevent your content being used in model training without surrendering your AI search presence, block the training user agent specifically and leave the search indexer off the disallow list, or explicitly allow it.

If you are relying on robots.txt as your primary enforcement mechanism, find out what is actually happening. Server log analysis showing which declared user agents are complying and which are not will tell you where the robots.txt is effective and where you need a firewall rule instead. A robots.txt block that is being violated 72% of the time is not a content protection strategy — it is a false sense of security.

And if your robots.txt was last updated more than a year ago, it almost certainly does not reflect the current crawler landscape. The list of active AI user agents and their purpose classifications has changed substantially since 2023. A rule written to stop one thing may now also be blocking several things it was never intended to touch.