AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsKept out
GPTBotline 1, for every crawlerTo answer people and searchKept outOAI-SearchBotline 1, for every crawlerChatGPT-Userline 1, for every crawler - AnthropicClaudeTo train its modelsKept out
ClaudeBotline 1, for every crawleranthropic-ailine 1, for every crawlerClaude-Webline 1, for every crawlerTo answer people and searchKept outClaude-SearchBotline 1, for every crawlerClaude-Userline 1, for every crawler - GoogleGeminiTo train its modelsKept out
Google-Extendedline 1, for every crawlerTo answer people and searchKept outGoogle-CloudVertexBotline 1, for every crawler - PerplexityTo train its modelsNone namedTo answer people and searchKept out
PerplexityBotline 1, for every crawlerPerplexity-Userline 1, for every crawler - MetaMeta AITo train its modelsKept out
meta-externalagentline 1, for every crawlerFacebookBotline 1, for every crawlerTo answer people and searchKept outmeta-externalfetcherline 1, for every crawlermeta-webindexerline 1, for every crawler - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 1, for every crawlerTo answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 1, for every crawlerTo answer people and searchKept outAmzn-SearchBotline 1, for every crawlerAmzn-Userline 1, for every crawler - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 1, for every crawlerTo answer people and searchNone named - ByteDanceTo train its modelsKept out
Bytespiderline 1, for every crawlerTikTokSpiderline 1, for every crawlerTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchKept out
MistralAI-Userline 1, for every crawler
27 of the 27 other AI crawlers we know are kept out: cohere-ai, cohere-training-data-crawler, Diffbot, Omgilibot, Omgili and 22 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 68% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps no AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.