AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsKept out
GPTBotline 51To answer people and searchPartlyOAI-SearchBotline 52, kept outChatGPT-Userline 88, let in - AnthropicClaudeTo train its modelsKept out
ClaudeBotline 59anthropic-ailine 57Claude-Webline 58To answer people and searchPartlyClaude-SearchBotline 60, kept outClaude-Userline 90, let in - GoogleGeminiTo train its modelsKept out
Google-Extendedline 53To answer people and searchKept outGoogle-CloudVertexBotline 65 - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 61, kept outPerplexity-Userline 89, let in - MetaMeta AITo train its modelsKept out
meta-externalagentline 54FacebookBotline 78To answer people and searchLet inmeta-externalfetchermeta-webindexerline 97 - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 64To answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 55To answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 75To answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 79, kept outTikTokSpiderlet inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
6 of the 27 other AI crawlers we know are kept out: cohere-ai, Diffbot, Omgilibot, Omgili, YouBot and 1 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 32% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps at least one AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.