Install

Web server & CDN snippets

When DNS can't change, your server or CDN keeps serving the site and forwards only bot requests to Wrenda. The snippets below are generated by the same code the dashboard uses, with example.com as the domain — the wizard renders them with yours.

How it works

Every snippet does one thing: test the User-Agent against Wrenda's canonical bot pattern and, on a match, proxy the request to https://api.wrenda.ai/proxy/{your-domain}{path}. Wrenda then classifies the bot against your rules and applies optimize, pre-render, pass-through or block, exactly as it would in DNS proxy mode. Because the classification happens on Wrenda's side, the pasted pattern never needs updating when new crawlers appear — Wrenda's registry is updated centrally.

The snippets forward these headers so Wrenda can log and route correctly:

Host
Your domain — Wrenda uses it to find the domain record.
X-Forwarded-Host
The hostname the visitor requested.
User-Agent
The original crawler user agent, unchanged.
X-Real-IP
Best-effort visitor IP for logs. Wrenda only trusts the IP it sees at its own edge.

Server snippets

In the dashboard these are shown on the final wizard step for CDN Integration and Web Server Proxy, with your domain substituted. Replace example.com and <api key> if you copy from here.

Nginx

nginx.conf
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Nginx configuration.
#
# Routes ALL bot traffic to Wrenda. Wrenda classifies the bot and applies
# the right action (optimize / pre-render / pass-through / block) based
# on routing templates configured in your dashboard.

# Detect bots that should be routed to Wrenda
map $http_user_agent $is_wrenda_crawler {
    default 0;
    "~*GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot" 1;
}

server {
    # ... your existing server configuration ...

    location / {
        # Forward bot traffic to Wrenda
        if ($is_wrenda_crawler) {
            proxy_pass https://api.wrenda.ai/proxy/example.com$request_uri;
            proxy_set_header Host example.com;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Host $host;
            proxy_set_header User-Agent $http_user_agent;
            proxy_set_header Authorization "Bearer <api key>";
            proxy_ssl_server_name on;
        }

        # Your existing configuration for non-bot traffic
        # proxy_pass http://your-origin;
    }
}

Apache

httpd.conf / .htaccess
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Apache VirtualHost or .htaccess.
# Requires: mod_proxy, mod_proxy_http, mod_rewrite, mod_headers
#
# Routes ALL bot traffic to Wrenda. Wrenda decides per-bot what to do
# based on routing templates configured in your dashboard.

RewriteEngine On

# Forward bot traffic to Wrenda
RewriteCond %{HTTP_USER_AGENT} (GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot) [NC]
RewriteRule ^(.*)$ https://api.wrenda.ai/proxy/example.com/$1 [P,L]

# Proxy settings for Wrenda requests
ProxyPreserveHost Off
ProxyPassReverse / https://api.wrenda.ai/proxy/example.com/

# Forward required headers
RequestHeader set X-Forwarded-Host "%{HTTP_HOST}e"
RequestHeader set X-Real-IP "%{REMOTE_ADDR}e"
RequestHeader set Authorization "Bearer <api key>"

# Enable these Apache modules if not already enabled:
# a2enmod proxy proxy_http rewrite headers

Caddy

Caddyfile
# Wrenda AI Crawler Proxy Configuration for example.com
# Add this to your Caddyfile.
#
# Routes ALL bot traffic to Wrenda. Per-bot routing decisions happen
# server-side via routing templates in your dashboard.

example.com {
    @wrenda_crawler {
        header_regexp User-Agent (?i)(GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot)
    }

    # Forward bot traffic to Wrenda
    reverse_proxy @wrenda_crawler https://api.wrenda.ai {
        header_up Host example.com
        header_up X-Forwarded-Host {host}
        header_up X-Real-IP {remote}
        header_up Authorization "Bearer <api key>"
        rewrite /proxy/example.com{uri}
    }

    # Your existing configuration for non-bot traffic
    # reverse_proxy your-origin:port
}

Cloudflare Worker

worker.js
// Wrenda AI Crawler Proxy — Cloudflare Worker
// Deploy this worker and create a route for example.com/*
//
// This forwards ALL bot traffic to Wrenda. Wrenda then classifies the
// request and applies the right action (optimize / pre-render /
// pass-through / block) based on the routing templates you configure
// in your dashboard. You don't need to update this file when new AI
// crawlers emerge — Wrenda's bot list is updated server-side.

const WRENDA_BOT_PATTERN = /GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot/i;
const WRENDA_PROXY_URL = 'https://api.wrenda.ai/proxy/example.com';

export default {
  async fetch(request, env, ctx) {
    const userAgent = request.headers.get('User-Agent') || '';

    // Forward all bot traffic to Wrenda — let Wrenda decide what to do.
    if (WRENDA_BOT_PATTERN.test(userAgent)) {
      const url = new URL(request.url);
      const wrendaUrl = WRENDA_PROXY_URL + url.pathname + url.search;

      const wrendaRequest = new Request(wrendaUrl, {
        method: request.method,
        headers: new Headers({
          'Host': 'example.com',
          'User-Agent': userAgent,
          'Accept': request.headers.get('Accept') || '*/*',
          'X-Forwarded-For': request.headers.get('CF-Connecting-IP') || '',
          'X-Forwarded-Host': url.hostname,
          'X-Real-IP': request.headers.get('CF-Connecting-IP') || '',
          'Authorization': 'Bearer <api key>'
        })
      });

      return fetch(wrendaRequest);
    }

    // Non-bot traffic: pass through to origin untouched.
    return fetch(request);
  }
};

Express.js

server.js
// Wrenda AI Crawler Proxy Middleware for Express.js
// npm install http-proxy-middleware
//
// Forwards ALL bot traffic to Wrenda. Per-bot routing decisions happen
// server-side via routing templates in your dashboard.

const { createProxyMiddleware } = require('http-proxy-middleware');

const WRENDA_BOT_PATTERN = /GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot/i;

const wrendaProxy = createProxyMiddleware({
  target: 'https://api.wrenda.ai',
  changeOrigin: true,
  pathRewrite: (path) => `/proxy/example.com${path}`,
  onProxyReq: (proxyReq, req) => {
    proxyReq.setHeader('Host', 'example.com');
    proxyReq.setHeader('X-Forwarded-Host', req.headers.host || 'example.com');
    proxyReq.setHeader('X-Real-IP', req.ip || req.connection.remoteAddress);
    proxyReq.setHeader('Authorization', 'Bearer <api key>');
  }
});

// Apply middleware BEFORE your routes
app.use((req, res, next) => {
  const userAgent = req.headers['user-agent'] || '';
  if (WRENDA_BOT_PATTERN.test(userAgent)) {
    return wrendaProxy(req, res, next);
  }
  next();
});

// Your existing routes below...

HAProxy

haproxy.cfg
# Wrenda AI Crawler Proxy Configuration for HAProxy
# Add this to your HAProxy configuration.
#
# Forwards ALL bot traffic to Wrenda. Per-bot routing decisions happen
# server-side via routing templates in your dashboard.

frontend http_front
    bind *:80
    bind *:443 ssl crt /path/to/cert.pem

    # Detect bots via the canonical Wrenda pattern
    acl is_wrenda_bot hdr_reg(User-Agent) -i GPTBot|OAI-SearchBot|SearchGPT|ChatGPT-User|ChatGPT|ClaudeBot|Claude-Web|Claude-SearchBot|Claude-User|Anthropic-User-Agent|anthropic-ai|anthropic|Claude|Google-Extended|Gemini-Deep-Research|Gemini-Ai|Gemini-Pro|Bard|Googlebot|Google-InspectionTool|GoogleOther|AdsBot|Bingbot|BingPreview|msnbot|Copilot|PerplexityBot|Perplexity-User|PerplexityProxy|Perplexity|Meta-ExternalAgent|Meta-ExternalFetcher|FacebookBot|facebookexternalhit|Applebot-Extended|Applebot|Amazonbot|Bytespider|CCBot|cohere-ai|cohere|MistralAI-User|YouBot|YouChat|Diffbot|DuckAssistBot|DuckDuckBot|AI2Bot|Phind|img2dataset|omgilibot|omgili|YandexBot|Yandex|Baiduspider|Baidu|Slurp|SeznamBot|Sogou|Exabot|PetalBot|Timpibot|ia_archiver|Twitterbot|LinkedInBot|WhatsApp|Slackbot|TelegramBot|AhrefsBot|SemrushBot|MJ12bot|DotBot|Scrapy|peer39_crawler|crawler|crawl|spider|scrape|fetch|wget|curl|bot

    # Forward bot traffic to Wrenda
    use_backend wrenda_backend if is_wrenda_bot

    # Default backend for non-bot traffic
    default_backend origin_backend

backend wrenda_backend
    option httpchk GET /health
    http-request set-header Host example.com
    http-request set-header X-Forwarded-Host %[req.hdr(Host)]
    http-request set-header Authorization "Bearer <api key>"
    http-request set-path /proxy/example.com%[path]
    server wrenda api.wrenda.ai:443 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt

backend origin_backend
    # Your origin server configuration
    server origin your-origin:80

CDN providers

For a CDN, Wrenda becomes an additional origin used only for bot traffic.

Cloudflare (zone stays proxied)

  1. Workers & Pages → Create → Hello World; replace the code with the Cloudflare Worker snippet above and deploy.
  2. In the zone: Workers Routes → Add route with the pattern your-domain.com/* and select the worker. Routes bind a pattern to a script; there is no separate origin field — the script decides where to fetch from.
  3. Verify with the curl below.

AWS CloudFront

Origin domain
api.wrenda.ai
Origin path
/proxy/your-domain.com
Protocol
HTTPS only
Custom origin header
Authorization = Bearer <api key>
Behaviour
Route bot user agents to this origin (a cache policy that forwards the User-Agent and Host headers, or a CloudFront Function that rewrites the request).

Fastly

Add api.wrenda.ai as a backend with the Host header set to your domain, then prepend the proxy path for bot requests:

VCL snippet (vcl_miss / vcl_pass)
set bereq.url = "/proxy/your-domain.com" + bereq.url;
set bereq.http.Authorization = "Bearer <api key>";

KeyCDN

Origin URL
https://api.wrenda.ai/proxy/your-domain.com
SSL
Full (strict)
Forward Host header
Enabled
Origin request header
Authorization: Bearer <api key>

Any other CDN

Configure a second origin at api.wrenda.ai with the path prefix /proxy/your-domain.com, forward the Host and User-Agent headers, add Authorization: Bearer <api key> as an origin request header, use TLS to the origin, and route only bot user agents to it. If your CDN can't route by user agent, contact support with the provider name and your domain.

Verify

Wrenda cannot see your server's configuration, so it can't confirm the forwarding for you. After you click Save & Verify with Crawl Tester the domain is marked configured and you land in Bot Testing pre-loaded with GPTBot/1.0 and your domain.

bash
curl -I -H "User-Agent: GPTBot/1.0" "https://your-domain.com/"
curl -I -H "User-Agent: Googlebot/2.1" "https://your-domain.com/"
Expected result
  • X-Correlation-ID — the request was processed by Wrenda.
  • X-Crawler-Action: optimize (or pre-render) — a crawler rule matched.
  • X-AI-Optimized: true for GPTBot in Active mode; X-Pre-Rendered: true for Googlebot when a pre-render rule exists.
  • In Shadow mode, X-Wrenda-Mode: shadow and the original content.

If none of these headers appear, your server isn't forwarding crawler traffic to Wrenda yet. A 401 means the Authorization header is missing or wrong.

Next

Bot handlingActions, bot categories, recommended rules, priority and testing.Serving modesActive, Shadow and Paused — what each does and when to use it.Cache & purgeTTLs, what counts as a request, purge hooks, sitemap watch and warming.APIAPI keys and JWT auth, the purge hook, the status widget and the /proxy endpoint.

Was this helpful?

Missing a step, or found something that doesn't match what you see? Tell us and we'll fix the page.

Contact us