AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotTo answer people and searchLet inOAI-SearchBotChatGPT-User - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 63, kept outanthropic-ailet inClaude-Weblet inTo answer people and searchKept outClaude-SearchBotline 61Claude-Userline 62 - GoogleGeminiTo train its modelsKept out
Google-Extendedline 71To answer people and searchKept outGoogle-CloudVertexBotline 70 - PerplexityTo train its modelsNone namedTo answer people and searchKept out
PerplexityBotline 82Perplexity-Userline 81 - MetaMeta AITo train its modelsPartly
meta-externalagentline 76, kept outFacebookBotlet inTo answer people and searchKept outmeta-externalfetcherline 77meta-webindexerline 78 - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 55To answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 54To answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 60To answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 59, kept outTikTokSpiderline 50, let inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchKept out
MistralAI-Userline 79
10 of the 27 other AI crawlers we know are kept out: cohere-ai, cohere-training-data-crawler, Diffbot, Timpibot, YouBot and 5 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 32% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps at least one AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.