AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsPartly
GPTBotline 7, some pathsTo answer people and searchPartlyOAI-SearchBotline 5, some pathsChatGPT-Userline 6, some paths - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 11, some pathsanthropic-ailet inClaude-Weblet inTo answer people and searchLet inClaude-SearchBotClaude-User - GoogleGeminiTo train its modelsLet in
Google-ExtendedTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 8, some pathsPerplexity-Userlet in - MetaMeta AITo train its modelsPartly
meta-externalagentline 10, some pathsFacebookBotlet inTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-ExtendedTo answer people and searchNone named - AmazonTo train its modelsPartly
Amazonbotline 9, some pathsTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsPartly
CCBotline 13, some pathsTo answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 71% of the 425 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 542 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 2% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.