AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotline 48To answer people and searchLet inOAI-SearchBotline 50ChatGPT-Userline 46 - AnthropicClaudeTo train its modelsLet in
ClaudeBotline 64anthropic-ailine 66Claude-Webline 62To answer people and searchLet inClaude-SearchBotline 58Claude-Userline 60 - GoogleGeminiTo train its modelsLet in
Google-Extendedline 72To answer people and searchLet inGoogle-CloudVertexBotline 70 - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 92Perplexity-Userline 90 - MetaMeta AITo train its modelsLet in
meta-externalagentline 84FacebookBotline 82To answer people and searchLet inmeta-externalfetcherline 86meta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-Extendedline 78To answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsLet in
CCBotTo answer people and searchNone named - ByteDanceTo train its modelsLet in
Bytespiderline 96TikTokSpiderline 98To answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 68% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps no AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.