AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsPartly
GPTBotline 78, some pathsTo answer people and searchPartlyOAI-SearchBotline 129, some pathsChatGPT-Userline 105, some paths - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 136, some pathsanthropic-ailet inClaude-Weblet inTo answer people and searchPartlyClaude-SearchBotline 181, some pathsClaude-Userline 160, some paths - GoogleGeminiTo train its modelsPartly
Google-Extendedline 214, some pathsTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 187, some pathsPerplexity-Userline 194, some paths - MetaMeta AITo train its modelsPartly
meta-externalagentline 241, some pathsFacebookBotlet inTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsPartly
Applebot-Extendedline 278, some pathsTo answer people and searchNone named - AmazonTo train its modelsPartly
Amazonbotline 260, some pathsTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsLet in
CCBotTo answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 72% of the 588 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 724 popular sites we’ve checked since 29 Oct 2026, 8% have a page drawn by scripts, 3% send a Markdown copy, 10% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.