AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotline 7To answer people and searchLet inOAI-SearchBotline 1ChatGPT-Userline 4 - AnthropicClaudeTo train its modelsLet in
ClaudeBotline 16anthropic-aiClaude-WebTo answer people and searchLet inClaude-SearchBotline 10Claude-Userline 13 - GoogleGeminiTo train its modelsLet in
Google-Extendedline 34To answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 28Perplexity-Userline 31 - MetaMeta AITo train its modelsLet in
meta-externalagentFacebookBotTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-ExtendedTo answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsLet in
CCBotTo answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 71% of the 435 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 555 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 2% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.