What it does
- Reads robots.txt first, as its standard (RFC 9309) reads it, and doesn’t fetch your page if your rules ask it not to, or if robots.txt answers with a server error.
- Fetches the home page once, over HTTPS first (then the www. address, then plain HTTP, when one doesn’t answer), following up to 5 redirects, and reads at most 5 MB. Beside it, one request to plain HTTP shows whether it sends visitors to HTTPS. If your site sends visitors to sign in first (a sign-in page such as Vercel’s, Cloudflare Access or Google’s, or a password prompt), it stops there and never opens the sign-in page.
- After the result is shown, and only when the question asked needs them, opens up to 5 pages your home page links (contact, privacy, terms…) to see that they answer and reads /llms.txt and /.well-known/security.txt when they exist; it also reads the one stylesheet your home page links on its own address (up to 1.5 MB) to name the CSS framework it uses. Each is checked against robots.txt.
- When the page is drawn by JavaScript, reads up to 2 of your own scripts, each checked against robots.txt.
- Fetches your site’s icon to show next to the result: the one your page links (up to 3 tries when one is missing or too big), each up to 32 KB. We show our copy, so visitors’ browsers never load it from your site.
- Reuses a result for 24 hours, so repeated requests don’t reach your site. A question about one thing reads only what its answer needs: a question about a site’s traffic or history reads nothing from your site at all.
- Identifies itself as CheckAnySiteBot/0.1 with a link to this page, and never pretends to be a browser.
- Signs its requests with Web Bot Auth: its public keys are at /.well-known/http-message-signatures-directory, so you can check that a request really comes from us.
How to block it
Add this to your robots.txt:
User-agent: CheckAnySiteBot Disallow: /
It then never fetches your pages. A result for your domain can still show what public records say about it: its registry record, DNS records, certificate logs, popularity lists and security filters.
Already scanned? Ask us to remove the result.