AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsPartly
GPTBotline 4, some pathsTo answer people and searchPartlyOAI-SearchBotline 9, some pathsChatGPT-Userline 14, some paths - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 29, some pathsanthropic-ailine 44, some pathsClaude-Weblet inTo answer people and searchPartlyClaude-SearchBotline 39, some pathsClaude-Userline 34, some paths - GoogleGeminiTo train its modelsPartly
Google-Extendedline 49, some pathsTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 19, some pathsPerplexity-Userline 24, some paths - MetaMeta AITo train its modelsLet in
meta-externalagentFacebookBotTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsPartly
Applebot-Extendedline 54, some pathsTo answer people and searchNone named - AmazonTo train its modelsPartly
Amazonbotline 64, some pathsTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsKept out
CCBotline 69To answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 72, kept outTikTokSpiderlet inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
3 of the 27 other AI crawlers we know are kept out: Diffbot, Omgili and ImagesiftBot.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
Like 26% of the 685 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps at least one AI training crawler out. Of the 842 popular sites we’ve checked since 29 Oct 2026, 8% have a page drawn by scripts, 3% send a Markdown copy, 9% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.