AI companies
Its robots.txt names none of the 24 crawlers of the 10 AI companies we know, so it lets every one in.
1 of the 27 other AI crawlers we know is kept out: ExaBot.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 75% of the 995 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 1,218 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 3% send a Markdown copy, 10% have an llms.txt and 4% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.