AI tools read websites in more ways than search engines did. AI crawlers collect pages to train models. Assistants such as ChatGPT, Claude and Perplexity fetch a page when someone asks about it. And AI agents, which act for a person, look for files that tell them what a site offers: an llms.txt, a Markdown copy of a page, an MCP server, the details they need to sign in.
So what do the web’s most popular websites actually offer them? On 29 October 2026 we checked 3,577 sites in the Chrome UX Report’s top 10,000, the sites people visit most in Google’s Chrome browser, with the same checks as our AI Crawler Checker. We could read the home page of 2,439 of them. Of the rest, 889 turned automated visitors away, 114 asked bots like ours to stay out, 134 didn’t answer and one asked visitors to sign in first.
The short version:
- Most popular sites are easy for AI to read. 89% send their text in the page itself, before any script runs; 88% have both a title and a description; 54% describe themselves in structured data.
- The new files for AI are still rare. 13.5% publish an llms.txt, and 1.9% send a Markdown copy of a page when asked for one.
- Almost none offer AI agents anything yet. 3.7% publish any of the files agents look for, nearly all of them sign-in details (3.3%). Nine sites publish an MCP server card, and 5 of 3,412 name their AI agents in DNS.
- Online shops and software sites lead the way. About 30% of them have an llms.txt, and more than one in ten offer agents a file.
- A third keep AI training crawlers out: 32% block at least one by name in robots.txt, and half of news sites do.
Can AI find your pages?
AI tools find pages much as search engines do: through links, sitemaps and the search indexes some of them draw on.
- A sitemap: 74% of the sites we could check have one that opens, listed in robots.txt or at the usual address,
/sitemap.xml. - Links for AI tools: 21 sites (0.9%) point AI tools to their files in the headers of their home page, a newer way to say where an llms.txt or an API description is.
- A noindex rule: 23 home pages (0.9%) ask search engines not to list them. A page search engines leave out also stays out of the AI answers that draw on them, whatever robots.txt allows.
Can AI read them?
| What AI tools read | Popular sites with it |
|---|---|
| The page’s text before any script runs | 88.7% |
| A title and a description | 87.6% |
| Structured data (schema.org JSON-LD) | 54.2% |
| An llms.txt file | 13.5% |
| A Markdown copy of the page, when asked | 1.9% |
Text before scripts matters most. A study of crawler traffic by Vercel and MERJ found that the crawlers behind ChatGPT, Claude and Perplexity fetch pages but don’t run their scripts (The rise of the AI crawler, December 2024). A page that only appears once a browser runs its code can look empty to them. Popular sites mostly send their words in the page: one in nine doesn’t. Sites made with AI builders are another story: two in three are blank before their scripts run.
Structured data says in a form machines parse what a page is: an organisation, a product, an article. News sites use it most (82%), government sites least (17.5%).
An llms.txt is a short Markdown summary of a site at /llms.txt, written for AI models. Jeremy Howard proposed it in September 2024 (llmstxt.org); it is still a proposal, not a standard the big AI companies have said they read. 324 of the 2,396 sites we could check have one.
A Markdown copy: some sites answer a request that asks for Markdown (Accept: text/markdown) with a plain-text version of the page, lighter for AI tools to read than the HTML. 47 do.
What do sites offer AI agents?
An AI agent doesn’t only read a page: it may search a catalogue, book, buy or sign in for its user. A handful of new standards tell agents where to start, each a file at an agreed address. Very few popular sites publish any of them yet:
| What a site offers agents | Popular sites | Share |
|---|---|---|
| Any of the files below | 90 | 3.7% |
| Sign-in details for agents (OAuth, RFC 9728 and RFC 8414) | 79 | 3.3% |
| An API catalog (RFC 9727) | 14 | 0.6% |
| Agent Skills | 14 | 0.6% |
| An MCP server card | 9 | 0.4% |
| An AI catalog | 8 | 0.3% |
| Auth.md | 3 | 0.1% |
| An agent card (Agent2Agent) | 3 | 0.1% |
| Agents named in DNS (DNS-AID) | 5 | 0.1% |
Most of the sign-in details predate AI agents: they describe any OAuth sign-in, and agents built on MCP use them to sign in. Each of the files written for agents alone is on fewer than one popular site in a hundred.
Agents in DNS is the newest of them: DNS-AID, an IETF draft, names a site’s AI agents in DNS records under _agents. Five sites publish such records: Linktree, CoinGecko and Padlet for an MCP server, Hostinger for its AI catalog, and faymas.in. Counting them taught us something too. Many domains answer every name in DNS, with an alias or a mail record, so our first count found agents on one site in eleven. The checker now asks a name no site publishes beside DNS-AID’s own, and ignores the records that name gets too (changelog).
Which sites are ready, by kind
| Kind of site | Pages read | Text before scripts | Structured data | llms.txt | An agent file | Keep AI training out |
|---|---|---|---|---|---|---|
| Blog or news site | 618 | 99.2% | 82.4% | 10.4% | 4.1% | 50.0% |
| Software or app | 194 | 84.0% | 55.7% | 31.4% | 13.5% | 19.6% |
| Company or service site | 169 | 85.2% | 50.9% | 23.2% | 3.6% | 18.7% |
| Online shop | 147 | 93.2% | 68.0% | 30.3% | 11.1% | 19.9% |
| Government or public body | 63 | 84.1% | 17.5% | 0% | 0% | 7.5% |
The pattern follows what each kind of site wants from AI. Shops and software companies want assistants to recommend them and agents to use them: they publish an llms.txt three times as often as news sites, and offer agents a file three times as often. News sites are the most readable of all: almost every one sends its text in the page, and four in five add structured data. They are also the most closed to training: half block AI training crawlers. Government sites are readable but rarely say what they are in structured data, and none we read has an llms.txt.
How popular a site is matters little: the top 1,000 and the sites ranked 1,000 to 10,000 are within two points of each other on everything AI can read and everything sites offer agents. The top 1,000 have a sitemap a little less often (69% against 75%) and keep AI training crawlers out a little more often (38% against 32%).
What sites tell AI crawlers
Of the 2,842 sites whose robots.txt we read, 922 (32.4%) block at least one AI training crawler from the whole site, close to the 31.8% we counted in mid-October on a smaller sample. Which crawlers they block, and which AI assistants they let in, is in How many websites block AI crawlers?
222 sites (7.8%) carry a Content Signals line in robots.txt, which says what their content may be used for: search, AI answers, AI training. 169 of them read search=yes,ai-train=no,use=reference, the line Cloudflare’s managed robots.txt writes. Its comments now define a fourth signal beside the three Cloudflare announced in September 2025 (Content Signals Policy): use, how AI systems may consume the content, immediately, as a reference or in full. 13 sites mark their pages noai, in a robots tag or header.
How to make your website ready for AI
- Send your text in the page. If scripts draw your content, render it on the server or prerender it, so crawlers that don’t run JavaScript read it too.
- Give every page a title and a description, and add structured data that says what the site is: an organisation, a shop, a product.
- List your sitemap in robots.txt, and check that no
noindexwas left on a page you want found. - Decide about AI training and AI answers separately. Blocking a training crawler such as GPTBot is a different choice from blocking ChatGPT-User or PerplexityBot, which read your pages to answer people.
- Add an llms.txt if you like. It is cheap to write, but still a proposal no big AI company has said it reads.
- If agents could use your service, publish what they need where they look: an MCP server card, an API catalog, sign-in details. Today almost no one does, so the sites that do stand out.
Check your own website
The AI Crawler Checker runs these checks on any site: what its robots.txt lets each AI company’s crawlers do, whether AI can find and read its pages, and which files it offers AI agents, each with what we found and where.
How we measured
- Sites: every site whose newest scan says the Chrome UX Report ranks it, under its own name, in its top-10,000 bucket (the August 2026 list): 3,577 sites our crawl had reached, read on 29 October 2026.
- Checks: the AI Crawler Checker’s, as of that day. Each share counts the sites where the check could be made: a page we couldn’t read can’t be checked for what’s on it, and a file robots.txt keeps us from isn’t asked for.
- Files for agents count only when their address answers with the format their standard gives them, or when the site’s AI catalog lists them.
- Kinds: the kind of site our scan named, for kinds with enough pages read.
What this doesn’t tell you
- What we could read. A quarter of the sites turned our bot away; the shares are of the sites that answered.
- What isn’t at an agreed address. An MCP server listed only in a site’s documentation, or tools a page offers agents through its scripts (WebMCP), aren’t counted.
- Whether AI companies use any of it. We count what sites publish, not what AI tools read.
- One snapshot, of October 2026. These standards are weeks or months old and change fast; we’ll count again.