AI companies
CompanyTo train its modelsTo answer people and search
- OpenAIChatGPTTo train its modelsPartly
GPTBotline 142, some pathsTo answer people and searchPartlyOAI-SearchBotline 140, some pathsChatGPT-Userline 246, some paths - AnthropicClaudeTo train its modelsPartly
ClaudeBotline 145, some pathsanthropic-ailet inClaude-Weblet inTo answer people and searchPartlyClaude-SearchBotline 146, some pathsClaude-Userlet in - GoogleGeminiTo train its modelsPartly
Google-Extendedline 143, some pathsTo answer people and searchLet inGoogle-CloudVertexBot - PerplexityTo train its modelsNone namedTo answer people and searchPartly
PerplexityBotline 147, some pathsPerplexity-Userlet in - MetaMeta AITo train its modelsLet in
meta-externalagentFacebookBotTo answer people and searchLet inmeta-externalfetchermeta-webindexer - AppleApple IntelligenceTo train its modelsPartly
Applebot-Extendedline 144, some pathsTo answer people and searchNone named - AmazonTo train its modelsLet in
AmazonbotTo answer people and searchLet inAmzn-SearchBotAmzn-User - Common Crawlopen data for trainingTo train its modelsPartly
CCBotline 148, some pathsTo answer people and searchNone named - ByteDanceTo train its modelsPartly
Bytespiderline 250, kept outTikTokSpiderlet inTo answer people and searchNone named - MistralLe ChatTo train its modelsNone namedTo answer people and searchLet in
MistralAI-User
1 of the 27 other AI crawlers we know is kept out: img2dataset.
What it tells AI crawlers
It gives no Content Signals and no “noai” tag.
If this is your site
- Give the Markdown copy the page’s own title: AI tools that ask for Markdown read another one.
Like 32% of the 1,691 popular sites whose robots.txt we read in October 2026, it keeps at least one AI training crawler out. How many websites block AI crawlers?
We read the site once, as our named bot: its robots.txt, read as crawlers read it, its home page and the files AI tools ask sites for. What we couldn’t read says so. robots.txt asks and doesn’t enforce: a firewall can treat AI crawlers differently from us.