Wrenda research · 100 sites · audited 28 September 2026

Britain's biggest websites treat AI crawlers differently from Google.

We checked 100 of the UK's highest-traffic websites for one thing: can the crawlers behind ChatGPT, Claude, Perplexity and Google's AI products reach a homepage, and is there anything readable when they get there? Not one site in the sample turns Googlebot away. Up to one in six turns away the AI bots.

0%
Disallow Googlebot
Zero of the 79 sites with a readable robots.txt
16%
Disallow ClaudeBot
The most-blocked AI crawler in the sample
67%
Readable without JavaScript
The rest need a browser to say anything at all
51
Sites clear both bars
Reachable by an AI bot and readable without JS
01

Google gets in everywhere. The AI crawlers don't.

Every site in this sample that publishes a readable robots.txt allows Googlebot. None block it. The same files block AI crawlers at rates between 9% and 16% — and that gap is a decision somebody made.

Sites disallowing each crawler on the homepagen = 100 · 21 had no readable robots.txt
Googlebot0 — noneChatGPT-UserChatGPT-User: 9 of 100 sites disallow it9Google-ExtendedGoogle-Extended: 11 of 100 sites disallow it11OAI-SearchBotOAI-SearchBot: 12 of 100 sites disallow it12GPTBotGPTBot: 14 of 100 sites disallow it14PerplexityBotPerplexityBot: 14 of 100 sites disallow it14ClaudeBotClaudeBot: 16 of 100 sites disallow it16

Homepage / rules only. A site with no reachable robots.txt counts as unknown, never as permission.

ClaudeBot is blocked most, ChatGPT-User least

That ordering is itself the finding. ChatGPT-User fetches a page because a person asked for it in a conversation; GPTBot and ClaudeBot are associated with bulk collection. Publishers are drawing a line between a reader who arrived via an assistant and a crawler that trains on them.

02

Pick a crawler. See what it actually got.

Robots rules are a request, not a gate. We fetched every homepage with each crawler's real User-Agent. Between 28% and 35% of the sites that allow a given bot still failed to return it a page.

56
reached the homepage
14
blocked in robots.txt
20/65
allowed by robots, still failed to fetch
44
reached it AND could read it
7
reached it and found almost nothing
What came back, per crawlern = 100 sites
ChatGPT-UserChatGPT-User — Reached the page: 61 sites61ChatGPT-User — Blocked by robots.txt: 9 sites9ChatGPT-User — Refused at HTTP level: 20 sites20ChatGPT-User — Challenge or CAPTCHA: 3 sitesChatGPT-User — Error or timeout: 7 sites7Google-ExtendedGoogle-Extended — Reached the page: 60 sites60Google-Extended — Blocked by robots.txt: 11 sites11Google-Extended — Refused at HTTP level: 20 sites20Google-Extended — Challenge or CAPTCHA: 4 sitesGoogle-Extended — Error or timeout: 5 sitesGPTBotGPTBot — Reached the page: 56 sites56GPTBot — Blocked by robots.txt: 14 sites14GPTBot — Refused at HTTP level: 21 sites21GPTBot — Challenge or CAPTCHA: 3 sitesGPTBot — Error or timeout: 6 sites6PerplexityBotPerplexityBot — Reached the page: 55 sites55PerplexityBot — Blocked by robots.txt: 14 sites14PerplexityBot — Refused at HTTP level: 22 sites22PerplexityBot — Challenge or CAPTCHA: 2 sitesPerplexityBot — Error or timeout: 7 sites7ClaudeBotClaudeBot — Reached the page: 51 sites51ClaudeBot — Blocked by robots.txt: 16 sites16ClaudeBot — Refused at HTTP level: 25 sites25ClaudeBot — Challenge or CAPTCHA: 2 sitesClaudeBot — Error or timeout: 6 sites6
Reached the pageBlocked by robots.txtRefused at HTTP levelChallenge or CAPTCHAError or timeout

Every fetch came from one cloud datacenter IP. A block here is a block of this probe — not proof that OpenAI's or Anthropic's own ranges are refused. See caveat 1.

03

Arriving is not the same as reading.

A crawler that fetches your homepage still has to find words in it. We stripped JavaScript and measured what survived. Two thirds of these sites are fine. The rest hand a fetcher an empty shell.

The eight emptiest homepages in Britain
DomainDownloadedOf which scriptWords a crawler reads
live.com14 KB16%1
msn.com46 KB79%1
ryanair.com5 KB18%1
bing.com70 KB86%6
channel4.com14 KB89%10
evri.com1.9 MB0%11
temu.com279 KB98%12
thetimes.com1 KB6%14

Each of these returned a valid page to a crawler. evri.com sent almost two megabytes and eleven words. There is no sentence here for a model to quote.

Text available without JavaScript
Server-rendered: 67 of 100 sites67Partial: 7 of 100 sitesThin shell: 8 of 100 sites8No body returned: 18 of 100 sites1867 of 100 return usable text with no JavaScript at all.33 do not — 18 returned nothing, 8 empty shells, 7 partial.
Server-rendered67Partial7Thin shell8No body returned18
Thinnest homepages
live.comlive.com: 1 words without JavaScript, 14 KB downloaded1msn.commsn.com: 1 words without JavaScript, 46 KB downloaded1ryanair.comryanair.com: 1 words without JavaScript, 5 KB downloaded1bing.combing.com: 6 words without JavaScript, 70 KB downloaded6channel4.comchannel4.com: 10 words without JavaScript, 14 KB downloaded10evri.comevri.com: 11 words without JavaScript, 1.9 MB downloaded11temu.comtemu.com: 12 words without JavaScript, 279 KB downloaded12thetimes.comthetimes.com: 14 words without JavaScript, 1 KB downloaded14

Six to seven sites per crawler are a false positive

They allow the bot in robots.txt, they return HTTP 200 to it, and the page holds almost no text. On any dashboard that measures crawler access these sites look healthy. An assistant asked about them has nothing to quote.

04

Banking is ready. News is blocking. Travel is invisible.

The pattern splits by sector, and not in the direction you would guess. The most regulated industry in the sample is the most machine-readable one.

No-JS extractability by sectorbar width = sites in sector
Retail & EcommerceRetail & Ecommerce — Server-rendered: 18 of 2418Retail & Ecommerce — Partial: 2 of 242Retail & Ecommerce — Thin shell: 1 of 24Retail & Ecommerce — No body: 3 of 243n=24OtherOther — Server-rendered: 14 of 2314Other — Partial: 3 of 233Other — Thin shell: 3 of 233Other — No body: 3 of 233n=23News & MediaNews & Media — Server-rendered: 12 of 1912News & Media — Partial: 1 of 19News & Media — Thin shell: 2 of 192News & Media — No body: 4 of 194n=19Government & PublicGovernment & Public — Server-rendered: 7 of 97Government & Public — No body: 2 of 92n=9Banking & FinanceBanking & Finance — Server-rendered: 9 of 99n=9Streaming & EntertainmentStreaming & Entertainment — Server-rendered: 4 of 84Streaming & Entertainment — Thin shell: 1 of 8Streaming & Entertainment — No body: 3 of 83n=8TravelTravel — Partial: 1 of 4Travel — Thin shell: 1 of 4Travel — No body: 2 of 42n=4TelecomTelecom — Server-rendered: 3 of 43Telecom — No body: 1 of 4n=4
Server-renderedPartialThin shellNo body

Travel (n=4) and Telecom (n=4) are small groups — a signal to check, not a finding.

Banking & finance

All nine banks render server-side and not one disallows an AI crawler. The security-conscious sector builds the most crawlable sites, because it was never able to lean on client-side rendering.

News & media

Seven of 19 publishers disallow GPTBot — a third of the group, and half of every block in the sample. That is a licensing position, not a technical failure, and it is the clearest deliberate signal in the data.

05

All 100 sites

Search, filter and sort the full sample. Bot cells show the outcome of a live fetch with that crawler's User-Agent; the crawler you picked above is highlighted.

100 sites
#▲Domain▲Sector▲Without JS▲Words▲GPT · CGU · Claude · Plx · GglX
1google.comOtherPartial72AAAAA
2bbc.co.ukNews & MediaServer-rendered1,692RRRRR
3amazon.co.ukRetail & EcommerceNo body0RRRRR
4bing.comOtherThin shell6AAAAA
5wikipedia.orgOtherNo body17AAHAA
6ebay.co.ukRetail & EcommerceServer-rendered1,318EEEEA
7dailymail.comNews & MediaServer-rendered11,521HHHHH
8theguardian.comNews & MediaServer-rendered3,506CCHHC
9live.comOtherThin shell1AAAAA
10yahoo.comNews & MediaServer-rendered460HHHHH
11service.gov.ukGovernment & PublicServer-rendered805AAAAA
12gov.ukGovernment & PublicServer-rendered805AAAAA
13netflix.comStreaming & EntertainmentServer-rendered567AAAAA
14rightmove.co.ukOtherServer-rendered542RAAAA
15cloud.microsoftOtherServer-rendered1,126AAAAA
16microsoft.comOtherServer-rendered1,011AAAAA
17bbc.comNews & MediaServer-rendered2,743RRRRR
18metoffice.gov.ukOtherServer-rendered1,142AAAAA
19imdb.comStreaming & EntertainmentNo body0RRRRR
20skysports.comOtherServer-rendered1,694RAAAA
21google.co.ukOtherPartial82AAAAA
22sky.comStreaming & EntertainmentServer-rendered4,202AAAAA
23telegraph.co.ukNews & MediaServer-rendered118RRRRH
24royalmail.comGovernment & PublicServer-rendered939HHHHH
25argos.co.ukRetail & EcommerceServer-rendered3,902HHHHH
26nytimes.comNews & MediaNo body9RRRRR
27booking.comTravelNo body0AEEEE
28thesun.co.ukNews & MediaServer-rendered2,609AARRR
29temu.comRetail & EcommerceThin shell12AAAAA
30fandom.comStreaming & EntertainmentServer-rendered1,169RHRHA
31pornhub.comOtherServer-rendered1,806AAAAA
32vinted.co.ukRetail & EcommerceServer-rendered544AAAAA
33autotrader.co.ukOtherServer-rendered1,568AAAAH
34duckduckgo.comOtherPartial67AAAAA
35etsy.comRetail & EcommerceNo body9HHHHA
36paypal.comBanking & FinanceServer-rendered1,489CCCCC
37next.co.ukRetail & EcommerceServer-rendered538HHHHH
38spotify.comStreaming & EntertainmentNo body4CCCCC
39apple.comOtherServer-rendered931AAAAA
40msn.comNews & MediaThin shell1AAAAA
41disneyplus.comStreaming & EntertainmentServer-rendered3,848AAAAA
42indeed.comOtherNo body4HHHAH
43office.comOtherServer-rendered970AAAAA
44github.comOtherServer-rendered1,098AAAAA
45evri.comOtherThin shell11AAAAA
46shopify.comRetail & EcommerceServer-rendered1,137AAAAA
47itv.comStreaming & EntertainmentNo body0EEREE
48premierleague.comOtherServer-rendered293AAAAA
49microsoftonline.comOtherNo body0EEEEE
50tesco.comRetail & EcommercePartial17HHHHH
51sainsburys.co.ukRetail & EcommerceServer-rendered573HHHHH
52boots.comGovernment & PublicServer-rendered4,864HAAAA
53marksandspencer.comRetail & EcommerceServer-rendered2,350AAAAA
54johnlewis.comRetail & EcommerceServer-rendered746EEEEA
55screwfix.comRetail & EcommerceServer-rendered364AAAAA
56asos.comRetail & EcommerceServer-rendered1,702EEEEE
57currys.co.ukRetail & EcommerceServer-rendered1,067AAAAA
58ao.comRetail & EcommerceNo body3HHHHH
59shein.co.ukRetail & EcommerceServer-rendered190AAAAA
60hotukdeals.comRetail & EcommerceServer-rendered1,414RRRRR
61zoopla.co.ukOtherServer-rendered1,314AAAAA
62gumtree.comOtherServer-rendered1,383AAAAA
63mirror.co.ukNews & MediaNo body0RHRRA
64independent.co.ukNews & MediaServer-rendered1,674AAHHA
65express.co.ukNews & MediaNo body0RARRA
66metro.co.ukNews & MediaServer-rendered2,407HHHHH
67standard.co.ukNews & MediaServer-rendered1,293AAAAA
68skynews.comNews & MediaPartial17HHHHH
69gbnews.comNews & MediaServer-rendered2,300HHHHC
70channel4.comStreaming & EntertainmentThin shell10RRRRR
71nhs.ukGovernment & PublicServer-rendered366AAAAA
72hmrc.gov.ukGovernment & PublicServer-rendered1,438AAAAA
73company-information.service.gov.ukGovernment & PublicNo body0EEEEE
74citizensadvice.org.ukGovernment & PublicServer-rendered448AAAAA
75moneysavingexpert.comBanking & FinanceServer-rendered2,818AAAAA
76which.co.ukOtherServer-rendered839AAAAA
77lloydsbank.comBanking & FinanceServer-rendered170AAAAA
78barclays.co.ukBanking & FinanceServer-rendered1,481AAHAA
79hsbc.co.ukBanking & FinanceServer-rendered1,999AAAAA
80natwest.comBanking & FinanceServer-rendered2,260AAAAA
81nationwide.co.ukBanking & FinanceServer-rendered1,490AAAAA
82bt.comTelecomServer-rendered662AAAAA
83ee.co.ukTelecomServer-rendered1,190AAAAA
84vodafone.co.ukTelecomServer-rendered3,861AAAAA
85virginmedia.comTelecomNo body3HHHHH
86ft.comNews & MediaServer-rendered2,614AARRR
87economist.comNews & MediaNo body9RRRRR
88thetimes.comNews & MediaThin shell14AARRR
89diy.comRetail & EcommerceServer-rendered4,026AAHAH
90asda.comRetail & EcommerceServer-rendered499HAHHH
91ikea.comRetail & EcommerceServer-rendered506AAAAA
92santander.co.ukBanking & FinanceServer-rendered1,521AAAAA
93halifax.co.ukBanking & FinanceServer-rendered124AAAAA
94easyjet.comTravelPartial17HHHHH
95ryanair.comTravelThin shell1AAAAA
96tripadvisor.co.ukTravelNo body9HAHHA
97dvla.gov.ukGovernment & PublicNo body26HHHHH
98jdsports.co.ukRetail & EcommerceServer-rendered1,600AAAAA
99very.co.ukRetail & EcommerceServer-rendered94HHHHH
100sportsdirect.comRetail & EcommercePartial17HHHHH

Key — A allowed · R robots-blocked · H HTTP-blocked · C challenge · E error or timeout.

06

How this was measured — and what it does not show

The sample is the UK top 100 by traffic (Similarweb, August 2026) with 15 social networks, messaging apps, AI chatbot interfaces and YouTube removed, renumbered, and topped up to 100 with high-traffic UK content and commerce domains. For each site we read robots.txt, fetched the homepage with each crawler's real User-Agent, extracted the text with JavaScript disabled, and flagged single-page-app shells.

What would change these numbers
  1. 1Every fetch came from one cloud datacenter IP. CDN bot management may refuse us while admitting the real OpenAI, Anthropic and Google ranges — or the reverse. The 18 sites returning no body are the figure most exposed to this, and should be read as "we could not retrieve it", not "AI crawlers are blocked".
  2. 2Homepages only. An article, product page or listing can extract very differently from the front door. A site marked thin here may have perfectly readable article pages.
  3. 3Ranks 51–100 are composite fills, not vendor-certified positions. They are high-traffic UK domains, but the ordering past 50 is ours.
  4. 4Headless rendering was not re-run for this pass. The no-JS measurement stands alone; the headless comparison columns are empty by design.
  5. 5Soft-block detection is heuristic and can misfire on marketing copy that mentions captchas.

This was a single pass on one day — a snapshot of a moving target. Robots files change, CDN rules change, and sites ship rendering changes weekly. The method is repeatable and the raw per-site JSON was retained, so the same audit can be re-run and compared.

Find out which column you are in

Wrenda shows you exactly what each AI crawler receives from your site, and changes it. 14 days, no card. New domains start in shadow mode, so nothing you serve changes until you say so.

Start 14-day trialHow optimization works