CrawlerToll

HTTP 402

HTTP 402 — Payment Required — sat dormant in RFC 7231 for a decade. It became operational in 2025 when Cloudflare launched pay-per-crawl. Cloudflare reports >1 billion HTTP 402 responses per day as of Q1 2026 through its customisable 402 message feature.

The shape CrawlerToll uses is straightforward:

  • Status 402
  • Crawler-Price header announcing the price
  • Crawler-Price-Rail header announcing the settlement rail
  • Link headers pointing at payment URL + description + terms
  • Retry-After for cooperative bots
  • JSON body with a structured payment offer

What the WordPress plugin sends

On an ordinary page, a declared AI crawler that your policy turns away gets this, as long as your site can be paid:

HTTP/1.1 402 Payment Required
Content-Type: application/json; charset=utf-8
Crawler-Price: 5000 micros USD
Crawler-Price-Rail: x402
Retry-After: 60
Link: <https://pay.example.com/abc>; rel="payment"; type="x402",
      <https://example.com/.well-known/context-license.json>; rel="describedby"; type="application/json",
      <https://example.com/ai-terms>; rel="terms-of-service"
 
{
  "error": "payment_required",
  "message": "This is the price this site advertises to AI crawlers. Paid access is sold per article: sealed articles link to their unlock offer.",
  "offer": {
    "rail": "x402",
    "priceMicros": 5000,
    "currency": "USD",
    "paymentUrl": "https://pay.example.com/abc",
    "publisher": "example-com",
    "endpoint": "default",
    "advertised": true
  }
}

The payment and terms-of-service links appear only if you set a payment URL and a terms-of-use URL. The price is advertised, not charged: paid access is sold per sealed article. If the site cannot be paid yet, the crawler gets a 403 "not licensed" instead, with no price. See the decision tree.

On a sealed article, a declared AI crawler gets a 402 with that article's price and a Link: <...>; rel="payment" header pointing at the unlock service. The agent asks the unlock service for the key, receives a signed offer, and pays it with x402. See AI agents.

Settlement rail enum

The offer.rail field tells the buyer where to settle:

| Value | Description | |---|---| | x402 | USDC via x402, paid to the publisher's wallet | | tollbit | Label only: CrawlerToll does not settle this rail | | skyfire | Label only: CrawlerToll does not settle this rail | | cloudflare-ppc | Label only: CrawlerToll does not settle this rail | | stripe-acp | Cards, charged on the publisher's own Stripe account | | context-license | Per the publisher's /.well-known/context-license.json | | custom | Bring your own |

See settlement rails for a comparison.

Why micros?

A "micro" is one millionth of a currency unit. USDC has 6 decimals — 5000 atomic units = 0.005 USDC = $0.005. The same unit works across USDC, USD, EUR, GBP. Integer-safe at JS-number precision up to ~$9 quadrillion per call, which should be enough.

The Crawler-Price format

CrawlerToll writes the price as <micros> micros <CUR>, for example 5000 micros USD. Cloudflare's pay-per-crawl uses a different format for its own Crawler-Price header, so don't assume the two are interchangeable.

Cloudflare interop

If you run CrawlerToll at the application layer and pay-per-crawl at the edge, both can fire. The edge handles unauthenticated enforcement and CrawlerToll handles fine-grained per-route policy. They compose, they don't compete.

See also