AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsKept out
GPTBotline 103To answer people and searchKept outOAI-SearchBotline 72ChatGPT-Userline 73 - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 104, kept outanthropic-ailet inClaude-Webline 105, kept outTo answer people and searchKept outClaude-SearchBotline 77Claude-Userline 76 - GoogleGeminiTo train its modelsKept out
Google-Extendedline 106To answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchKept out
PerplexityBotline 74Perplexity-Userline 75 - MetaMeta AITo train its modelsKept out
meta-externalagentline 111FacebookBotline 280To answer people and searchPartlymeta-externalfetcherlet inmeta-webindexerline 79, kept out - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 107To answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 108To answer people and searchNone named - ByteDanceTo train its modelsKept out
Bytespiderline 109TikTokSpiderline 110To answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
18 of the 27 other AI crawlers we know are kept out: cohere-ai, cohere-training-data-crawler, Diffbot, Omgilibot, Omgili and 13 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 28% of the 394 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps at least one AI training crawler out. Of the 503 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 2% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.