How to get your bot verified by Cloudflare with Web Bot Auth

A step-by-step guide to becoming a Cloudflare verified bot: sign requests with Web Bot Auth, host a key directory, submit in BotBase, test it, and avoid the mistakes that fail verification.

If your crawler, monitor or AI agent visits websites, a large share of them sit behind Cloudflare. An unknown bot there can be challenged or blocked. A verified one is recognised by name, sorted into a category, and site owners can let that category in with one rule.

The short version: generate an Ed25519 key, publish its public half at /.well-known/http-message-signatures-directory on your own domain, submit the bot in the Cloudflare dashboard under BotBase, and once it’s accepted, sign every request with three headers: Signature-Agent, Signature-Input and Signature. Cloudflare calls this Web Bot Auth.

What “verified” means

Cloudflare verifies a bot that is honest about who it is and doesn’t abuse that honesty. In practice, two things:

  • It identifies itself the same way every time. That means a cryptographic signature (Web Bot Auth), or a published list of IP addresses with a stable user agent, or reverse DNS. Signatures are the method Cloudflare now encourages.
  • It behaves. It follows robots.txt, including Crawl-delay, keeps its request rate reasonable, and has never been seen evading a site owner’s choices.

Each verified bot is labelled by what it does, such as Search, Agent, Training, SEO, Monitoring or Data Collection. It is also labelled by who can run it:

  • Direct: only you can send its requests.
  • Intermediary: many end users can drive it, as with an agent platform.

Choose the labels that match your traffic. Cloudflare can change them if the bot’s behaviour says otherwise.

Before you apply

  • A public page about your bot. Say what it does, how often it visits, how to block it and how to reach you. The form asks for its address.
  • A stable user agent that names the bot and links to that page.
  • robots.txt respected, including Crawl-delay. Ignoring it is one of the listed reasons for losing verified status.
  • Your own domain, served over HTTPS, to host the key directory.

Step 1: generate a signing key

Cloudflare accepts Ed25519 keys. With OpenSSL:

openssl genpkey -algorithm ed25519 -out private-key.pem
openssl pkey -in private-key.pem -pubout -out public-key.pem

Convert the public key to a JSON Web Key (JWK) with any JWK library, or generate the pair with the WebCrypto API, which gives you the JWK directly. Keep the private key in your secrets, never in a repository.

Step 2: host a key directory

Serve a JSON Web Key Set at exactly /.well-known/http-message-signatures-directory, over HTTPS:

{ "keys": [{ "kty": "OKP", "crv": "Ed25519", "x": "<your public key, base64url, no padding>" }] }
  • Send it with Content-Type: application/http-message-signatures-directory+json.
  • Sign the response itself, so nobody can copy your directory and register it as theirs:
    • one signature per key;
    • tag http-message-signatures-directory;
    • the key’s JWK thumbprint as keyid;
    • created and expires timestamps;
    • the @authority component.
  • Never publish the private part of the key (the JWK’s d).

Cloudflare publishes a small command-line tool, http-signature-directory, that checks a directory before you submit.

Step 3: submit the bot

In the Cloudflare dashboard, open BotBase and its Submission form:

  • the bot’s name, its documentation page, the operator and a short description;
  • its classification: Direct or Intermediary, what it does, and how it uses content;
  • Request Signature as the verification method, with your key directory’s address.

Follow it under BotBase › Submission history:

Status Meaning
Waiting For Review Received and queued
Accepted Verified: it appears in BotBase and in Cloudflare Radar’s bot directory
Rejected Declined; the submission shows why

Review takes days to a couple of weeks. A well-formed signature gets prioritised.

Step 4: sign every request

Once accepted, add three headers to each request your bot sends:

  • Signature-Agent: your key directory’s origin, as a quoted string: "https://example.com".
  • Signature-Input: which components you sign and how:
    • at least @authority and signature-agent;
    • tag web-bot-auth;
    • your key’s JWK thumbprint as keyid;
    • created, and a short expires. A minute is enough, and it limits replays.
  • Signature: the Ed25519 signature over those components.

Here Signature-Input is wrapped for reading; on the wire each header is one line:

Signature-Agent: "https://example.com"
Signature-Input: sig1=("@authority" "signature-agent")
  ;created=1735689600;expires=1735689660
  ;keyid="<thumbprint>";alg="ed25519";tag="web-bot-auth"
Signature: sig1=:<base64 signature>:

You don’t have to write the signing yourself: Cloudflare maintains open-source libraries, the web-bot-auth package on npm and a Rust crate.

Test it

Send a signed request to https://crawltest.com/cdn-cgi/web-bot-auth:

  • 200: your key is known and the signature checks out.
  • 401: the message is well formed, but the key isn’t registered yet or the signature failed.
  • 400: something in the headers is malformed.

Mistakes that fail verification

  • A Signature-Agent that isn’t quoted, isn’t https://, or uses the dictionary form from later drafts (sig1="https://…").
  • Leaving signature-agent out of the signed components while sending the header.
  • Signing a header with non-ASCII characters. Cloudflare doesn’t support the parameters that would encode them.
  • Signing Content-Digest through a proxy that may change the body.
  • A long expires. It works, but a replayed request stays valid for that long.

After approval

Your bot appears in Cloudflare Radar’s bots directory, and site owners can allow it by its category rather than by guessing at user agents. Verification can be withdrawn: traffic that doesn’t match the declared purpose, unpatched vulnerabilities or ignoring Crawl-delay are all listed reasons. An organisation that breaks the rules repeatedly can lose verified status for all its bots.

Our own scanner works this way: here is its entry in Cloudflare Radar, and what it does and how to block it.

To see the other side, which bots a site lets in and which it keeps out, check any site’s robots.txt for AI crawlers.

Put it to work

Something here out of date or wrong? Tell us and we’ll fix it, with the date of the change.