AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotline 26To answer people and searchLet inOAI-SearchBotChatGPT-Userline 29 - AnthropicClaudeTo train its modelsLet in
ClaudeBotline 33anthropic-ailine 36Claude-WebTo answer people and searchLet inClaude-SearchBotClaude-User - GoogleGeminiTo train its modelsLet in
Google-Extendedline 44To answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 40Perplexity-User - MetaMeta AITo train its modelsLet in
meta-externalagentline 48FacebookBotTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-ExtendedTo answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsLet in
CCBotTo answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 72% of the 351 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 446 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 3% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.