How many AI crawlers are there?

There are 61 widely recognized AI crawlers, grouped into four types: search indexers for AI answer engines, assistant fetchers, agent browsers, and training-data collectors. New ones appear every few months as AI companies launch products.

Which AI crawlers ignore robots.txt?

A few crawlers — including Bytespider, DeepSeekBot, LAIONDownloader and ISSCyberRiskCrawler — have been reported to ignore robots.txt. For these, enforce blocks at the CDN or WAF level instead of relying on robots.txt.

The AI crawler directory

61 known AI crawlers that may visit your site — who runs them, what they do with your content, and whether a robots.txt Disallow actually stops them. Every profile links back to blocking instructions and the free audit tool.

AI search crawlers (10)

OAI-SearchBotOpenAI Claude-SearchBotAnthropic PerplexityBotPerplexity AI GooglebotGoogle ApplebotApple AmazonbotAmazon YouBotYou.com DuckDuckBotDuckDuckGo BingbotMicrosoft PetalBotHuawei

AI assistant fetchers (11)

ChatGPT-UserOpenAI Claude-UserAnthropic Claude-WebAnthropic Perplexity-UserPerplexity AI Gemini-Deep-ResearchGoogle Amzn-UserAmazon MistralAI-UserMistral AI cohere-aiCohere DuckAssistBotDuckDuckGo Diffbot-UserDiffbot Copilot/MicrosoftMicrosoft

AI agent browsers (9)

ChatGPT AgentOpenAI Claude-CodeAnthropic AmazonBuyForMeAmazon CursorCursor (Anysphere) DevinCognition Google-AgentGoogle Google-Gemini-CLIGoogle WindsurfWindsurf viberankVibeRank

AI training data crawlers (31)

GPTBotOpenAI ClaudeBotAnthropic anthropic-aiAnthropic Google-ExtendedGoogle GoogleOtherGoogle GoogleOther-ImageGoogle GoogleOther-VideoGoogle Google-CloudVertexBotGoogle Applebot-ExtendedApple BytespiderByteDance · ⚠ ignores robots.txt TikTokSpiderByteDance CCBotCommon Crawl FacebookBotMeta meta-externalagentMeta BedrockbotAmazon DeepSeekBotDeepSeek · ⚠ ignores robots.txt MistralAI-TrainingMistral AI cohere-training-data-crawlerCohere DiffbotDiffbot AI2BotAllen Institute for AI Ai2Bot-DolmaAllen Institute for AI img2datasetimg2dataset project LAIONDownloaderLAION · ⚠ ignores robots.txt ImagesiftBotHive AI ICC-CrawlerNICT (Japan) ISSCyberRiskCrawlerISS-Corporate · ⚠ ignores robots.txt FriendlyCrawlerUnknown PanguBotHuawei YandexAdditionalYandex OmgilibotWebz.io OmgiliWebz.io

Check your site against all 61 crawlers

Paste your robots.txt and get a per-crawler verdict, a score, and a fix list — free, no signup, runs in your browser.

Audit my robots.txt Run a live scan