CrawlerToll

Web Bot Auth

Web Bot Auth is the IETF draft for cryptographic bot identity over HTTP. A bot operator publishes Ed25519 public keys at a well-known URL on their domain. Every request the bot makes carries an RFC 9421 HTTP Message Signature, signed with the matching secret key. Publishers verify the signature, look up the public key in the bot's directory, and gain cryptographic certainty about the requester's identity.

Adopters

| Operator | Role | |---|---| | Cloudflare | Verifier (network-wide) | | AWS WAF | Verifier (Nov 2025) | | Akamai | Verifier | | OpenAI | Signer — Operator + ChatGPT-agent traffic | | Google | Signer (experimental, agent.bot.goog, Mar 2026) | | Block (Goose) | Signer | | Browserbase | Signer | | Vercel | Verifier | | Shopify | Verifier (merchant-side crawler auth) |

Anthropic, Perplexity, Mistral — not yet publicly committed as of Q2 2026.

How it works

   Bot                                Publisher
   ───                                ─────────
 
1. Generate Ed25519 keypair
2. Publish public key as JWKS at
   /.well-known/http-message-signatures-directory
3. Sign each outbound request with secret key
 
                    ──HTTP request──▶
                    Signature-Input: sig1=("@authority" "signature-agent");
                                          keyid="<thumbprint>";alg="ed25519"
                    Signature: sig1=:<base64>:
                    Signature-Agent: "https://bot.example/"
 
                                          4. Parse Signature-Input header
                                          5. Fetch bot's JWKS directory
                                          6. Find key by thumbprint
                                          7. Reconstruct signature base
                                          8. Ed25519 verify
                                          9. allow / charge / block

How CrawlerToll uses it

CrawlerToll does not verify Web Bot Auth signatures. The WordPress plugin recognises a declared AI crawler by the user agent it sends, matched against its catalogue of 30 crawlers, and a signed request is treated like any other. Fetching a bot's key directory on every request would add an outbound call to each page view, which shared WordPress hosting often blocks or slows.

What this means for you:

  • A crawler that lies about its user agent is not recognised. That is why premium content is sealed: an unidentified scraper gets the encrypted body, whatever name it wears.
  • If you want signatures checked, do it where it is cheap, at the edge. Cloudflare, AWS WAF and others already verify Web Bot Auth for you, and CrawlerToll works alongside them. See the decision tree.

Want to be a verified bot operator?

The IETF draft is at datatracker.ietf.org. Cloudflare publishes an integration guide at developers.cloudflare.com. Stytch + Fingerprint also publish how-tos.

See also