AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsKept out
GPTBotline 108To answer people and searchLet inOAI-SearchBotline 150ChatGPT-Userline 132 - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 98, kept outanthropic-ailine 92, kept outClaude-Weblet inTo answer people and searchLet inClaude-SearchBotline 134Claude-Userline 136 - GoogleGeminiTo train its modelsLet in
Google-Extendedline 140To answer people and searchKept outGoogle-CloudVertexBotline 106 - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 154Perplexity-Userline 152 - MetaMeta AITo train its modelsKept out
meta-externalagentline 110FacebookBotline 102To answer people and searchLet inmeta-externalfetcherline 146meta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-ExtendedTo answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 88To answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 96To answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 94, kept outTikTokSpiderlet inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-Userline 148
2 of the 27 other AI crawlers we know are kept out: cohere-ai and Timpibot.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 32% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps at least one AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.