AI companies
Its robots.txt names none of the 24 crawlers of the 10 AI companies we know, so it lets every one in.
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
If this is your site
- Decide what AI may do with the site, and say it in robots.txt. Write the lines
Like 71% of the 319 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 392 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 3% send a Markdown copy and 10% have an llms.txt. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.