Operate
Bot handling
Crawler rules decide what each visitor gets. A rule is a user-agent match plus an action; the highest-priority matching rule wins. This page covers the actions, the bot categories the registry knows about, the recommended starting set, and how to prove a rule does what you think.
The four actions
| Action | What happens | Cached | Counts as a request |
|---|---|---|---|
| Optimize | Wrenda fetches the page, runs the AI enrichment pipeline (structure, entity context, FAQs, schema) and serves the result. Meant for AI crawlers and agents. | 1 hour by default (per Template Optimization) | Yes, on a cache miss |
| Pre-render | A headless browser loads the page, executes JavaScript and serves the resulting HTML. Meant for search engines on JS-heavy sites. | 24 hours | Yes, on a cache miss |
| Pass-through | The origin response, unchanged, with your CDN cache rules applied. | CDN edge rules | No |
| Block | A 403 with an X-Blocked-By header. Nothing is fetched from the origin. | — | No |
Static assets — images, CSS, JS, fonts, media, PDFs and framework asset paths such as /_next/image — always take pass-through, even for a crawler whose rule says optimize. Only HTML documents are transformed.
Bot categories
Wrenda maintains one registry of crawler user-agent tokens, shared by the proxy, the analytics and the install snippets. Each token has a vendor and one of five categories. In Rules → Bot Handling you pick bots with checkboxes grouped by these categories and Wrenda generates the matching pattern; an “Advanced: edit regex” disclosure exposes the raw pattern if you need it.
| Category | Examples | Typical action |
|---|---|---|
| AI training | GPTBot, ClaudeBot, Claude-Web, Google-Extended, Meta-ExternalAgent | Optimize (or block if you don’t want to be in training sets) |
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Gemini | Optimize — these feed answers that cite you |
| AI agents | ChatGPT-User, Claude-User, Perplexity-User, Copilot, Meta-ExternalFetcher | Optimize — a person is waiting on the other end |
| Search engines | Googlebot, Google-InspectionTool, Bingbot, GoogleOther; also link-preview fetchers such as facebookexternalhit | Pre-render on JS-rendered sites, otherwise pass-through |
| Other bots | Scrapers and generic crawlers (Scrapy, peer39_crawler, …) | Block |
The registry is updated centrally; when a new crawler appears it is added for every tenant and, if your rule selected its category, it is covered without you editing anything.
Recommended rules
When a domain is created Wrenda applies a starting set automatically, and the Bot Handling page shows an Apply recommended rules banner whenever a domain has none. The set is idempotent by rule name, so applying it again never duplicates rules:
| Rule | Matches | Action | When |
|---|---|---|---|
| AI crawlers → optimize | Every AI training, AI search and AI agent token | Optimize | Always |
| Known scrapers → block | Registry “other bot” tokens, excluding generic words that could match legitimate tools | Block | Always |
| Search engines → pre-render | Search-engine tokens | Pre-render | Only when the domain was detected as JS-rendered (you can add it by hand otherwise) |
There is also a Rule Templates library (e-commerce, blog, SaaS, news, documentation, portfolio) that installs crawler rules together with matching Template Optimizations.
Priority and matching
- Rules are evaluated from the highest priority number down; the first active rule whose pattern matches the user agent decides the action. The default priority is 100.
- Give narrow exceptions a higher number than broad rules. For example: “Googlebot → pass-through” at 200 above “Search engines → pre-render” at 100 for a section that renders fine without a browser.
- Patterns are case-insensitive regular expressions tested against the full
User-Agentstring. - No match means pass-through. Humans never match any bot token.
- Rules are per domain. Inactive rules are skipped.
Testing a rule
Rules → Bot Testing fetches a URL twice — once with a browser user agent and once with the crawler you pick (GPTBot, Googlebot, ClaudeBot, PerplexityBot…) — and shows the rendered preview, the HTML source, the response headers and a diff side by side, with size and timing stats and the action taken. Use it before and after changing a rule. Agent Testing does the same for AI-agent user agents.
From a terminal, the response headers tell you everything:
| Header | Meaning |
|---|---|
X-Crawler-Action | optimize · pre-render · pass-through · block |
X-Matched-Rule-ID | Which rule decided the action |
X-AI-Optimized: true | The body is the AI-optimized copy |
X-Pre-Rendered: true | The body is the browser-rendered copy |
X-Cache-Hit | Served from the optimize / pre-render cache |
X-Cache-Rule, X-Edge-TTL | Pass-through: which CDN cache rule matched and its edge TTL in seconds |
X-Wrenda-Mode | Present in Shadow and Paused modes |
X-Wrenda-Status | over-limit: the plan allowance is used up, so this is a cached copy or the original page · over-limit-passthrough: the 30-day warning window has ended and everything passes through · disconnected: the account is deactivated |
X-Correlation-ID | Present on every response handled by Wrenda; quote it to support |
X-Processing-Time | Milliseconds spent in Wrenda |
The six-crawler curl checklist
Run these against an Active domain. Each comment shows the header to look for.
# 1. Pre-render (Googlebot) — needs a pre-render rule for search engines
curl -I -H "User-Agent: Googlebot/2.1" "https://DOMAIN/"
# → X-Pre-Rendered: true
# 2. AI optimization (GPTBot)
curl -I -H "User-Agent: GPTBot/1.0" "https://DOMAIN/"
# → X-AI-Optimized: true
# 3. Pass-through (browser)
curl -I -H "User-Agent: Mozilla/5.0 Chrome/120" "https://DOMAIN/"
# → X-Crawler-Action: pass-through
# 4. CDN cache on a static asset
curl -I -H "User-Agent: Mozilla/5.0 Chrome/120" "https://DOMAIN/image.png"
# → X-Cache-Rule: Images, X-Edge-TTL: 2592000
# 5. Static asset from an AI crawler skips optimization
curl -I -H "User-Agent: GPTBot/1.0" "https://DOMAIN/style.css"
# → X-Crawler-Action: pass-through (and NO X-AI-Optimized)
# 6. Search Console's URL Inspection tool
curl -I -H "User-Agent: Google-InspectionTool/1.0" "https://DOMAIN/"
# → X-Pre-Rendered: trueThe first optimize or pre-render request for a URL takes seconds (it does the work); repeat it and you should see X-Cache-Hit and a sub-100 ms X-Processing-Time. Check 4 assumes the default CDN cache rules (images 30 days = 2,592,000 s); if you changed them the TTL differs.