AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotline 302To answer people and searchLet inOAI-SearchBotline 283ChatGPT-Userline 284 - AnthropicClaudeTo train its modelsKept out
ClaudeBotline 271anthropic-ailine 272Claude-Webline 273To answer people and searchLet inClaude-SearchBotline 291Claude-Userline 290 - GoogleGeminiTo train its modelsLet in
Google-ExtendedTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 296Perplexity-Userline 297 - MetaMeta AITo train its modelsPartly
meta-externalagentline 268, kept outFacebookBotlet inTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsKept out
Applebot-Extendedline 270To answer people and searchNone named - AmazonTo train its modelsKept out
Amazonbotline 269To answer people and searchLet inAmzn-SearchBotline 312Amzn-Userline 313 - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 251To answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 252, kept outTikTokSpiderlet inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
7 of the 27 other AI crawlers we know are kept out: cohere-ai, Diffbot, Omgilibot, Omgili, ImagesiftBot and 2 more.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 28% of the 598 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps at least one AI training crawler out. Of the 735 popular sites we’ve checked since 29 Oct 2026, 8% have a page drawn by scripts, 3% send a Markdown copy, 10% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.