CrawlerToll

Getting started with WordPress

The CrawlerToll WordPress plugin is dual-licensed Apache-2.0 OR GPL-2.0-or-later. PHP 7.4+. WordPress 6.0+.

Install

From the WordPress admin

  1. WordPress admin → Plugins → Add New
  2. Search for CrawlerToll
  3. Install and Activate

From the command line

wp plugin install crawlertoll --activate

Sixty seconds

After activation, enforcement is live with sensible defaults:

  • 7 AI crawler User-Agents are covered by the default rules (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider)
  • Price shown to AI crawlers: 5,000 millionths of a dollar ($0.005) per visit. It is advertised, not charged
  • /wp-content/uploads/ always allowed
  • * catch-all is Disallow: (all other crawlers pass through)

Until you add a way to get paid (a USDC address or your Stripe keys), a declared AI crawler is told your content isn't licensed. Once you can be paid, it is shown your price. Test it:

curl -sI -H 'user-agent: GPTBot/1.2' https://your-site.example/
# → HTTP/2 402
# → crawler-price: 5000 micros USD
# → crawler-price-rail: x402

Customising

Settings → CrawlerToll:

| Field | Default | Description | |---|---|---| | Enforcement toggle | on | Turn enforcement on or off without uninstalling | | Article price for readers | 1.00 | One-time price to unlock one sealed article. Cards need at least 0.50; USDC wallets work from 0.001 | | Article price for AI agents | (empty) | What declared AI crawlers pay in USDC to read one sealed article. Leave blank to charge them the reader price | | Price shown to AI crawlers, per visit (millionths) | 5000 | 5000 = 0.005. Shown to AI crawlers on ordinary pages; it is advertised, not charged | | Currency | USD | USD / USDC / EUR / GBP | | Payment method shown to AI crawlers | USDC over x402 | USDC over x402 (wallets and AI agents) / Cards, Apple Pay, Google Pay on your own Stripe account / Custom payment URL | | USDC payout address | (empty) | Your wallet address for USDC on Base (0x…, 42 characters). Required to sell to agents; leave empty to turn the USDC rail off | | Stripe publishable key | (empty) | Your own Stripe key (pk_live_ or pk_test_) | | Stripe secret key | (empty) | Use a restricted key (rk_…) with two permissions: PaymentIntents: Write and Charges: Read. Stored on your site only and never shown again | | Apple Pay verification | (empty) | Only needed for one-tap Apple Pay. Paste Stripe's domain-association file; it is served at /.well-known/apple-developer-merchantid-domain-association | | Payment URL | (empty) | Optional. A page where AI companies can arrange a licence with you. Linked from the response AI crawlers get | | Terms-of-use URL | (empty) | Surfaced as Link rel="terms-of-service" | | Publisher contact email | (empty) | Published in /.well-known/context-license.json so AI companies can reach you about licensing. Use a role address | | EU/UK right of withdrawal | Automatic | Automatic (ask when the site looks European), Always ask, or Never ask. When on, readers must agree to immediate access before paying | | Your AI-crawler rules (advanced) | (default) | Raw robots.txt rules with RSL-style licence lines. You normally never need to touch this |

Selling to readers: the sealed paywall

Switch on Premium in the Paywall panel of the editor sidebar and place the paywall cut — drag the bar between blocks, or click "Place paywall cut at cursor" to end the preview mid-sentence. Set what one article costs under Settings → CrawlerToll → Article price for readers (default $1.00; cards need at least $0.50). When you publish or update the post, the plugin splits it, encrypts the body with AES-256-GCM, escrows the key with the unlock service, and serves the preview plus the encrypted body. Excerpts, feeds, oEmbed and the REST API only ever see the preview; paywall structured data (isAccessibleForFree: false + hasPart) tells search engines what is going on.

Readers see a wall with one tile per rail: pay by card (Stripe Payment Element, with Apple Pay / Google Pay where available) and pay with USDC (any EIP-1193 wallet; the app switches to Base for them). AI agents get the same offer as a signed HTTP 402 with x402 V2 headers. After payment the browser decrypts in place, caches the key on the device, and holds a settlement pass so access survives a cleared cache.

Stripe's card floor is $0.50. The per-visit price shown to AI crawlers on ordinary pages is advertised, not charged; agents pay per sealed article. To sell to card readers at small prices, use Pro access tiers ($3–10 passes work well) or raise the article price.

If the unlock service can't be reached, the wall shows an "unavailable" card and no payment can start. If a card payment goes through and the article doesn't open, the reader's browser finishes it on the next try, without a second charge.

How it integrates with WordPress

| Surface | Behaviour | |---|---| | parse_request action | Decision runs as early as possible — sub-millisecond cost | | robots_txt filter | RSL-style licence lines appended to the existing /robots.txt | | REST API | /wp-json/crawlertoll/v1/context-license | | Rewrite rule | /.well-known/context-license.json proxies cleanly | | Admin menu | Settings → CrawlerToll | | wp_options | Single key crawlertoll_settings stores all configuration |

Compatibility

  • WP Super Cache / W3 Total Cache / WP Rocket: works. The plugin returns 402 before the cache layer runs, and sealed pages are cache-safe by design (the encrypted body is static; the unlock never touches the page render).
  • Cloudflare APO: works.
  • Multisite: works per-site.
  • REST API / WP-CLI / cron / xmlrpc: skipped — those surfaces don't enforce.

If you use Cloudflare

Turn off Cloudflare's "Block AI bots" setting and its managed robots.txt for the paths you seal. If they are on, AI agents are stopped at Cloudflare's edge and never reach the paywall, so they can't pay you.

Web Bot Auth note

The plugin does not verify Web Bot Auth signatures. It recognises a declared AI crawler by its user agent, which needs no outbound request on shared hosting. For premium content, sealing is what stops a crawler that hides its name. See Web Bot Auth.

Next steps

Pro

CrawlerToll Pro ($29/mo or $249/yr, 14-day trial) adds the revenue tooling on top of the free paywall:

  • Access tiers and bundles — price × duration passes per path, or a whole-section pass
  • Metered free articles — N free reads per window, with a tunable per-IP ceiling
  • Per-path pricing and per-crawler rail routing
  • Revenue dashboard, decision logs with CSV/JSON export, retention control, email alerts
  • Unlock webhooks — HMAC-signed unlock.succeeded events with a durable retry outbox
  • Content provenance — SHA-256 fingerprints of exactly what each crawler received