AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsLet in
GPTBotline 9To answer people and searchLet inOAI-SearchBotChatGPT-User - AnthropicClaudeTo train its modelsLet in
ClaudeBotline 12anthropic-aiClaude-WebTo answer people and searchLet inClaude-SearchBotClaude-User - GoogleGeminiTo train its modelsLet in
Google-ExtendedTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchLet in
PerplexityBotline 15Perplexity-User - MetaMeta AITo train its modelsLet in
meta-externalagentFacebookBotTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsLet in
Applebot-Extendedline 24To answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsLet in
CCBotline 21To answer people and searchNone named - ByteDanceTo train its modelsLet in
BytespiderTikTokSpiderTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
None of the 27 other AI crawlers we know is kept out.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
If this is your site
- Give the Markdown copy the page’s own title: AI tools that ask for Markdown read another one.
Like 73% of the 754 popular sites whose robots.txt we’ve read since 29 Oct 2026, it keeps no AI training crawler out. Of the 935 popular sites we’ve checked since 29 Oct 2026, 9% have a page drawn by scripts, 3% send a Markdown copy, 10% have an llms.txt and 3% offer a file for AI agents. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.