AI Crawler Checker

Is this website ready for AI?

See which AI crawlers a site lets in, to train their models and to answer people, what they can read, and what it offers AI agents, with the evidence behind every answer.

Free, no sign-up
Evidence on every answer
All eight in one full scan

How to

How to check if AI crawlers can access a website

Enter the site’s address in the box above: a domain such as example.com is enough.
See which AI companies it lets in, to train their models and to answer people, and what their crawlers can read.
Open the evidence: each answer shows the line of robots.txt or the file it comes from, so you can check it yourself.

What we check

Where the answer comes from

Seen on the site
What it tells AI crawlers
Its rules for 51 AI crawlers, as their makers name them, split into training and answering people, and any Content Signals it gives.
Seen on the site
What AI can find and read
The page as AI crawlers and AI tools get it, whether its text needs scripts to show, whether search engines may list it, its structured data and its llms.txt.
Seen on the site
What it offers AI agents
The files and DNS records AI agents look for to find a site’s API, how to sign in or register, and the tools it offers them.

Reading the answer

What the answers mean

Answer	What it means
Let in	Its robots.txt lets the crawler read the site.
Partly	It keeps the crawler from some pages, or keeps out only some of the crawlers.
Kept out	It asks the crawler to stay away from the whole site.
Found	The site gives AI tools what the row names; the evidence shows where.
Not found	We looked, and the site doesn’t have it.
Couldn’t check	We couldn’t read it, and say why; it never counts as missing.

A real example

wikipedia.org · It lets every AI crawler in.

Its result shows each AI company’s crawlers with the line of robots.txt that decides them, and what AI can find and read there.

See the evidence

For site owners

Write your AI rules

Pick what AI companies may do with your site, and paste the lines into your robots.txt. Training and answering people are separate choices: keeping training crawlers out doesn’t keep you out of AI answers.

Let every AI inTraining and answers. Most popular sites do this.Answers yes, training noStay in AI assistants’ answers and AI search; keep out of training.Keep all AI outNo training, no AI answers: AI assistants won’t read or quote the site.

Add Cloudflare’s Content Signals line too

robots.txt

# AI training: no. AI answers and AI search: yes.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: Claude-Web
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: FacebookBot
User-agent: Bytespider
User-agent: cohere-ai
User-agent: cohere-training-data-crawler
User-agent: Diffbot
User-agent: Omgilibot
User-agent: Omgili
User-agent: ImagesiftBot
User-agent: Timpibot
User-agent: AI2Bot
User-agent: Amazonbot
User-agent: YouBot
User-agent: PanguBot
User-agent: Webzio-Extended
User-agent: img2dataset
User-agent: Ai2Bot-Dolma
User-agent: TikTokSpider
User-agent: DeepSeekBot
User-agent: ChatGLM-Spider
User-agent: ICC-Crawler
User-agent: Cotoyogi
User-agent: SBIntuitionsBot
User-agent: VelenPublicWebCrawler
User-agent: Brightbot
User-agent: Factset_spyderbot
User-agent: laion-huggingface-processor
Disallow: /

# Content Signals (contentsignals.org)
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no

robots.txt asks; well-behaved crawlers follow it. To enforce a rule, turn crawlers away at your firewall or CDN as well. These lines name all 51 AI crawlers we know.

Honest limits

What it can’t tell

robots.txt asks; each crawler decides whether to follow it.
A firewall can turn AI crawlers away while letting us in, and we can’t see that.
We read the site as our own bot: AI companies’ crawlers may get a different answer.
The files for AI agents matter only for sites with an API or a sign-in.
We don’t run the page’s scripts, so tools a page offers AI agents in the browser (WebMCP) don’t show.
AI companies add and rename crawlers often; we update our list and log every change.

How we keep the data reliable

Questions

Common questions

How do I check if AI crawlers can access my website?

Enter it above. We read its robots.txt for the 51 AI crawlers we know and show, company by company, which it lets in to train their models and which to answer people, each with the line it comes from.

Does blocking GPTBot keep my site out of ChatGPT?

Not on its own. GPTBot collects pages to train OpenAI’s models, OAI-SearchBot finds pages for ChatGPT’s search, and ChatGPT-User fetches a page when a person asks for it. robots.txt names each one separately, and OpenAI says its rules may not apply to ChatGPT-User, since a person asked.

Does blocking Google-Extended keep my site out of Google’s AI answers?

No. Google’s AI Overviews and AI Mode are part of Search: Googlebot reads pages for them, and snippet rules decide what they show. Google-Extended only decides whether Google may use pages to train its Gemini models and ground some of its other AI products.

Why doesn’t my website show up in ChatGPT?

The check shows the usual reasons: robots.txt keeps OAI-SearchBot out, the page asks search engines not to list it (noindex), or its text only shows once scripts run. A new site may also not have been read yet. None of them guarantees a mention when they’re fixed: the AI decides what it cites.

Should I block AI crawlers?

That’s your choice. Keeping training crawlers out asks that your pages aren’t used to train models; keeping AI assistants out also keeps you out of their answers. A third of the popular sites that block AI training still let AI assistants in.

Can AI read a website made with Lovable?

Often not from its HTML: many sites made with AI builders draw their text with scripts, which most AI crawlers don’t run. Some send a Markdown copy to AI tools that ask for one. The check shows what this site sends.

What is llms.txt?

A short summary of a site in Markdown, for AI models, at /llms.txt. It was proposed in 2024 and is still a proposal; the big AI companies haven’t said they read it. 9% of the 2,125 popular sites we read in October 2026 had one.

Live