AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsKept out
GPTBotline 28To answer people and searchKept outOAI-SearchBotline 166ChatGPT-Userline 31 - AnthropicClaudeTo train its modelsKept out
ClaudeBotline 10anthropic-ailine 34Claude-Webline 13To answer people and searchKept outClaude-SearchBotline 163Claude-Userline 97 - GoogleGeminiTo train its modelsKept out
Google-Extendedline 22To answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchKept out
PerplexityBotline 37Perplexity-Userline 76 - MetaMeta AITo train its modelsPartly
meta-externalagentline 103, kept outFacebookBotlet inTo answer people and searchKept outmeta-externalfetcherline 25meta-webindexerline 136 - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 4To answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 1To answer people and searchKept outAmzn-SearchBotline 94Amzn-Userline 100 - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 43To answer people and searchNone named - ByteDanceTo train its modelsKept out
Bytespiderline 7TikTokSpiderline 130To answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchKept out
MistralAI-Userline 52
13 of the 27 other AI crawlers we know are kept out: cohere-ai, Diffbot, Timpibot, AI2Bot, YouBot and 8 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 30% of the 408 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps at least one AI training crawler out. Of the 517 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 2% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.