AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsPartly
GPTBotline 22, some pathsTo answer people and searchPartlyOAI-SearchBotline 23, some pathsChatGPT-Userline 24, some paths - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 25, some pathsanthropic-ailet inClaude-Weblet inTo answer people and searchPartlyClaude-SearchBotline 27, some pathsClaude-Userline 26, some paths - GoogleGeminiTo train its modelsPartly
Google-Extendedline 21, some pathsTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 28, some pathsPerplexity-Userline 29, some paths - MetaMeta AITo train its modelsPartly
meta-externalagentline 33, some pathsFacebookBotlet inTo answer people and searchPartlymeta-externalfetcherline 34, some pathsmeta-webindexerlet in - AppleApple IntelligenceTo train its modelsPartly
Applebot-Extendedline 20, some pathsTo answer people and searchNone named - AmazonTo train its modelsPartly
Amazonbotline 30, some pathsTo answer people and searchPartlyAmzn-SearchBotline 31, some pathsAmzn-Userline 32, some paths - Common Crawlopen data for trainingTo train its modelsPartly
CCBotline 35, some pathsTo answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchPartly
MistralAI-Userline 37, some paths
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 71% of the 520 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 650 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 2% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.