CrawlerToll

Overview

CrawlerToll seals your premium articles and lets readers and AI agents pay you to read them, by card on your own Stripe account or in USDC to your own wallet. It also tells AI crawlers your terms, in robots.txt and in its HTTP responses.

It uses three of the standards that consolidated around AI-crawler monetization in 2025–2026: RSL-style licence lines, HTTP 402 and x402. It doesn't verify Web Bot Auth signatures; it recognises declared AI crawlers by their user agent. On ordinary pages the WordPress plugin advertises a price to AI crawlers. On sealed articles it sells access.

What it does

On every front-end request, the WordPress plugin:

  1. Recognises declared AI crawlers via a curated catalogue of 30 user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, CCBot, Cohere, Mistral, etc.). Search-engine crawlers and people always pass.
  2. Applies your RSL-style robots.txt policy.
  3. Answers a crawler the policy turns away. If the site can be paid, it gets HTTP 402 with Crawler-Price / Crawler-Price-Rail / Link headers and a structured JSON payment offer. If the site cannot be paid yet, it gets a 403 "not licensed". Everyone else gets the page.
  4. Seals premium articles: the body is encrypted on the page and unlocks only against a payment made directly to you.

What it doesn't do

  • Hold your money. The plugin seals content that unlocks against a payment made directly to you, by card on your own Stripe account or in USDC to your own wallet. See settlement rails and the unlock service.
  • Replace a CDN. It runs inside WordPress. Cloudflare / Akamai / Fastly handle edge-level concerns; CrawlerToll handles fine-grained policy where edge rules don't reach.
  • Identify scrapers that hide. User-agent recognition only covers crawlers that declare themselves. Sealing is the answer for premium content: an unidentified scraper gets the encrypted body. Edge tools such as Cloudflare Bot Management handle the rest of a site.

Where to start

  • WordPress (the plugin: sealed paywall, 30-crawler catalogue)
  • AI agents (how an agent pays for and reads a sealed article with @crawlertoll/sdk)
  • Unlock service (how a sealed article's key is released after payment)
  • Concepts (the decision tree, settlement rails, privacy)
  • Standards (RSL licence lines, HTTP 402, x402, Web Bot Auth, Context License)

How it's organised

| Part | What | |---|---| | WordPress plugin | Recognises declared AI crawlers, applies the RSL-style policy, answers with 402 or 403, and seals premium articles | | Unlock service | Holds the content keys and releases one after a payment settles. Never touches the money | | Settlement rails | USDC over x402 to the publisher's wallet, and cards on the publisher's own Stripe account | | @crawlertoll/sdk | For AI agents: pays for one sealed article and reads it, within a spending cap you set |

Want to know the design choices?

Read the decision tree, the settlement-rails comparison, and the privacy posture.