AI companies send crawlers across the web. Some collect pages to train their models, such as OpenAI’s GPTBot, Anthropic’s ClaudeBot and Common Crawl’s CCBot. Others fetch a page when a person asks an AI assistant or an AI search engine a question, such as ChatGPT-User and PerplexityBot. The one widely used way for a website to say no is a line in its robots.txt file.
So how many of the web’s most popular websites block AI crawlers, and which ones? We read the robots.txt of sites in the Chrome UX Report’s top 10,000, the sites people visit most in Google’s Chrome browser. That is 2,125 sites, scanned between 10 and 21 October 2026. We could read a robots.txt on 1,691 of them.
The short version:
- One in three popular websites (32%) blocks at least one AI training crawler from the whole site.
- GPTBot is the most blocked AI crawler (23%), followed by CCBot (22.5%), Bytespider (21%) and ClaudeBot (20%).
- Half of news and blog sites (51%) block AI training, against 20% of online shops, 18% of software sites and 8% of government sites.
- 22% also block AI assistants and AI search, led by PerplexityBot (14.5%) and ChatGPT-User (13%). A third of the sites that block training still let assistants in.
- Only 9% have an llms.txt file, and 3% use Cloudflare’s new Content Signals.
Training crawlers and AI assistants are not the same
The crawlers fall into two groups, and sites treat them differently:
- AI training crawlers collect pages to build and improve AI models. Blocking them asks that a site’s content isn’t used for training.
- AI assistant and AI search crawlers fetch pages on demand, when someone asks ChatGPT, Perplexity or Claude about a topic, or to build an AI search index. Blocking them keeps those assistants from reading the site for their users, and from quoting it.
The AI training crawlers blocked most, as a share of the 1,691 sites whose robots.txt we read:
| Crawler | Run by | Blocked by |
|---|---|---|
| GPTBot | OpenAI | 23.2% |
| CCBot | Common Crawl | 22.5% |
| Bytespider | ByteDance | 20.8% |
| ClaudeBot | Anthropic | 20.3% |
| Google-Extended | 16.7% | |
| meta-externalagent | Meta | 16.2% |
| Applebot-Extended | Apple | 15.0% |
| Amazonbot | Amazon | 14.4% |
And the AI assistant and search crawlers blocked most:
| Crawler | Run by | Blocked by |
|---|---|---|
| PerplexityBot | Perplexity | 14.5% |
| ChatGPT-User | OpenAI | 13.0% |
| OAI-SearchBot | OpenAI | 9.3% |
| Perplexity-User | Perplexity | 7.6% |
| Claude-SearchBot | Anthropic | 7.1% |
| Claude-User | Anthropic | 7.0% |
Common Crawl isn’t an AI company, but its open archive of the web is one of the most used sources of AI training data, which is why so many sites block CCBot alongside GPTBot.
Who blocks AI crawlers
The kind of site matters far more than how popular it is. Sites in Chrome’s top 1,000 block AI training as often as those ranked 1,000 to 10,000 (31.7% against 31.8%). But news sites and online shops are worlds apart:
| Kind of site | Sites | Block AI training | Block AI assistants | Have an llms.txt |
|---|---|---|---|---|
| Blog or news site | 469 | 51% | 42% | 8% |
| Forum or community | 60 | 42% | 24% | 5% |
| Online shop | 126 | 20% | 14% | 13% |
| Company or service site | 250 | 19% | 11% | 14% |
| Software or app | 168 | 18% | 6% | 24% |
| Government or public body | 34 | 8% | 4% | 0% |
The pattern follows what each kind of site lives on. News publishers and forums live on their writing: half of news sites block AI training, and most of those block AI assistants too. Software companies want AI assistants to recommend them and to answer questions about their products: they rarely block assistants (6%), and they publish an llms.txt more than any other kind of site (24%).
Blocking AI training, but not AI search
Of the 537 sites that block AI training, 178 (33%) let every AI assistant and AI search crawler in. They refuse to be training data, but still want to be read, summarised and cited when someone asks an AI. That’s the trade-off: blocking every AI crawler also means fewer chances to appear in AI answers.
Google is a special case. Google-Extended isn’t a separate crawler but a robots.txt token: it tells Google whether pages Googlebot already fetched may be used for its Gemini models. Google says it doesn’t affect a site’s inclusion or ranking in Google Search (Search Engine Roundtable, reporting Google’s crawler documentation). 16.7% of the sites we read block it.
llms.txt: 9% of popular sites have one
An llms.txt file is a short Markdown summary of a site, written for AI models, at /llms.txt. Jeremy Howard proposed it in September 2024 (llmstxt.org). It is still a proposal, not a standard the big AI companies have said they read. We found an llms.txt on 189 of the 2,125 sites (8.9%), led by software and app sites (24%). No government site we scanned has one.
Cloudflare’s Content Signals
In September 2025 Cloudflare introduced the Content Signals Policy, a robots.txt line that says what a site’s content may be used for: search, ai-input (used to answer a question) and ai-train (Cloudflare’s announcement). Cloudflare writes search=yes, ai-train=no into the robots.txt files it manages for its customers. A year on, 49 of the 1,691 popular sites (2.9%) carry a Content Signals line: 41 say ai-train=no, 8 say ai-train=yes, and all of them say search=yes.
How to block AI crawlers in robots.txt
To ask a crawler to stay out of your whole site, name it and disallow everything:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Three things to know before you do:
- robots.txt is a request, not a lock. Well-behaved crawlers follow it; others don’t. To enforce a block, sites turn AI crawlers away at their firewall or CDN, which robots.txt doesn’t show.
- Decide about AI search separately. Blocking training crawlers such as GPTBot is a different choice from blocking ChatGPT-User, OAI-SearchBot or PerplexityBot, which read your pages to answer people.
- Names change. AI companies add and rename crawlers often; a list written a year ago misses some of today’s.
Check any website’s AI crawler rules
Every CheckAnySite result shows what a site tells AI crawlers: which training crawlers and AI assistants its robots.txt blocks, each with the exact line we found, any Content Signals line or noai tag, and what its llms.txt says.
How we measured
- Sites: every site whose newest scan says the Chrome UX Report ranks it, under its own name, in its top-10,000 bucket (the August 2026 list): 2,125 sites, scanned 10–21 October 2026, most popular first.
- robots.txt: a crawler counts as blocked when a group naming it disallows the whole site (
Disallow: /). Rules for some paths only are counted apart: 8.4% of sites have only those. Shares are of the 1,691 sites whose robots.txt we could read; 3.4% of them block every crawler. - Crawler lists: 34 AI training crawlers and 17 AI assistant and search crawlers, as their makers name them.
- Kinds: the kind of site our scan named, for kinds with at least 30 sites.
- llms.txt: a file we could read at
/llms.txt, counted on all 2,125 sites.
What this doesn’t tell you
- robots.txt only. A site can also block AI crawlers at its firewall, which we can’t see; these are the rules sites publish, not what reaches them.
- A sample of the top 10,000. The first 2,125 sites our crawl reached, most popular first; 434 had no robots.txt we could read.
- Kinds are named for about half the sites, and government sites are few (34).
- llms.txt is a floor: a file we couldn’t read counts as none.
- One snapshot, of October 2026. Sites change their robots.txt often; we’ll count again.