CrawlerToll

Pay for and read sealed articles (AI agents)

This page is for developers who build AI agents. It shows how your agent pays for one sealed article and reads it, using @crawlertoll/sdk version 0.2.0 or later.

What you get

One paid request opens one sealed article. Your agent buys the full article and gets its full text. The publisher is paid directly, in USDC on Base. Charthouse never holds the money.

Where a publisher offers it, you can also buy an AI-training licence for an article. See AI-training licence below.

Before you start

  • Node 20 or later.
  • The SDK (@crawlertoll/sdk) is not on npm yet. Email hello@crawlertoll.com for the package, or follow the steps under "How it works" with your own HTTP client. The samples below assume the SDK and viem are installed.
  • A wallet key that holds USDC on Base. Your agent signs each payment with it. The SDK sends the signed payment, not the key.
  • A spending cap. You must pass maxMicros, the most you will pay for one article, in USDC micros (1,000,000 micros is 1 USDC). The SDK refuses to sign anything above it.

Practise first on Base Sepolia, where the USDC is free. Get test USDC from faucet.circle.com, then point your script at the test article at https://www.crawlertoll.com/demo/testnet.html. That article costs 0.10 test USDC, so raise maxMicros to 100_000 when you practise on it. The 50_000 cap in the script below would refuse it. Its key is sold by the test unlock service, not by registry.crawlertoll.com, so the SDK refuses it unless you also pass registryKeysUrl: "https://crawlertoll-registry-x402test.vgsv-office.workers.dev/.well-known/crawlertoll-keys.json".

The script

import { payAndRead } from "@crawlertoll/sdk";
import { privateKeyToAccount } from "viem/accounts";
 
const account = privateKeyToAccount(process.env.AGENT_PRIVATE_KEY as `0x${string}`);
const { html, paid } = await payAndRead("https://www.crawlertoll.com/demo/micro.html", account, {
  maxMicros: 50_000, // never pay more than 0.05 USDC
  userAgent: "MyAgent/1.0 (+https://example.com/agent)", // your real crawler user agent
});
console.log(`paid ${paid.micros / 1e6} USDC`);
console.log(html);

Set userAgent to the user agent your crawler really uses. The unlock service uses it to quote you the right price, and the SDK sends the same one on every request in the flow.

How it works

payAndRead runs three steps. You can call them one at a time if you want to look at the price before paying.

  1. readSealed(url) fetches the article. A sealed article is a normal page with its text encrypted. The SDK reads the encrypted text from the page and finds the payment link in the Link header, which looks like Link: <...>; rel="payment"; part="full". It also returns the listed price, if there is one. The article URL must be https, after any redirects, and the sealed article must belong to the site that served it.

  2. buyKey(keyUrl, account, { maxMicros, contentId }) asks the unlock service for the key. With no payment attached, the service answers 402 with a signed offer (ct_offer_v1). The SDK checks the offer before anything is signed:

    • the signature matches a key published at https://registry.crawlertoll.com/.well-known/crawlertoll-keys.json;
    • the offer is for the article you asked for, and is signed for the same site as the article (with or without www.);
    • the offer is still current;
    • the price is within your maxMicros.

    Then it signs a USDC transfer for the amount in the offer and sends it back as an x402 V2 PAYMENT-SIGNATURE header. The request also carries a replay_secret. If the response is lost, the same payment can be sent again with the same secret. The key comes back as cek. A full-price unlock also comes with a pass for the article; a micro unlock (a very small price) comes with none.

  3. unseal(blob, cek) decrypts the article on your side and gives you the HTML.

The SDK only talks to https registry URLs. It refuses a payment link that is not on registry.crawlertoll.com unless you pass registryKeysUrl yourself, and it refuses redirects while paying.

Prices

The offer quotes a USDC price. If the publisher sets prices in another currency, the unlock service converts the price to USDC at a daily reference rate. The amount you sign is exactly the amount in the offer, and paid.micros tells you what it was.

If something goes wrong

The SDK throws an Error. Before anything is signed, a 402 is simply the offer itself. After a payment has been signed, some errors carry two extra fields: err.retry and err.expiresAt.

  • 400: final. The service refused the request. Do not retry it.
  • 402 with err.retry: the service could not confirm the payment, for example because the payment network was unreachable. The payment may have gone through, so retry it. Do not start again with buyKey.
  • Any other failure after paying (a network error, a 409 already used, or a 5xx): the key may not have reached you. Call retryKey(err.retry) before err.expiresAt.
import { readSealed, buyKey, retryKey } from "@crawlertoll/sdk";
 
const userAgent = "MyAgent/1.0 (+https://example.com/agent)";
const page = await readSealed("https://www.crawlertoll.com/demo/micro.html", { userAgent });
let keyResult;
try {
  keyResult = await buyKey(page.keyUrl, account, { maxMicros: 50_000, contentId: page.contentId, userAgent });
} catch (err: any) {
  if (err.retry) keyResult = await retryKey(err.retry); // same payment, sent again
  else throw err;
}

retryKey sends the identical signed payment again, so you are not charged twice. That holds within 24 hours, while any pass it bought is still live. Retry before err.expiresAt. If you call buyKey again instead, you sign a new payment.

Don't log err.retry: it can unlock the article again within 24 hours. The SDK keeps it out of console.error(err) and JSON.stringify(err).

Other errors you may meet:

  • 403 from the site's CDN: the publisher blocks bots before the paywall, so your agent never reaches the article. The publisher has to allow agents on sealed articles.
  • Price above your cap: the SDK throws and signs nothing. Raise maxMicros only if you accept the price.

AI-training licence

Where a publisher offers it, the AI-training licence is offered to declared AI crawlers: the unlock service lists it, as the tier l0, only when your request's user agent matches a declared AI crawler. The SDK's default user agent (CrawlerToll-Agent) is one since unlock service 2.2.1, and a crawler's real, declared user agent sent with userAgent is too. Any other user agent sees only the read price, and buyKey(..., { tierId: "l0" }) throws no tier "l0". Buy it with the same call and a tier id:

const bought = await buyKey(page.keyUrl, account, { maxMicros: 500_000, tierId: "l0", contentId: page.contentId, userAgent });
console.log(bought.licence);

You can check whether your wallet already holds the licence for an article, before paying, at POST /v1/sealed/:id/licence on https://registry.crawlertoll.com. You prove the wallet address by signing a free message, so nobody else can see what your address has bought.

For the service behind these calls, see the unlock service.