The AI crawler directory
61 known AI crawlers that may visit your site — who runs them, what they do with your content, and whether a robots.txt Disallow actually stops them. Every profile links back to blocking instructions and the free audit tool.
What counts as an AI crawler?
An AI crawler is any automated client that fetches pages for something other than classic search ranking: assembling a training corpus, building the citation index behind an AI answer engine, pulling a single page because a user asked an assistant about it, or browsing on a user's behalf inside an AI tool. Googlebot, Bingbot and similar classic crawlers still belong to ordinary search — they are listed here because the same robots.txt file governs both worlds, and a rule written for one frequently catches the other.
The practical difficulty is that these clients identify themselves by a product token in the User-Agent header, and there is no central registry. A token that looks like a scraper may be a legitimate assistant fetcher, while a familiar-looking name can be spoofed by anything. Every profile below records the operator, the documented purpose, whether the crawler is known to honor robots.txt, and the exact rules to allow or block it.
The four types at a glance
The type matters more than the vendor, because it determines what you lose by blocking. These are the current counts in the directory:
| Type | Count | What blocking costs you |
|---|---|---|
| AI search crawler | 10 | Removal from that engine's citation pool — no crawl means no citation and no referral visit |
| AI assistant fetcher | 11 | Live fetches for users who explicitly asked about your page; low volume, high intent |
| AI agent browser | 9 | Agent-driven visits and task completion inside AI tools |
| AI training data crawler | 31 | Future model knowledge; anything already collected usually stays collected |
robots.txt is a request, not a wall
A Disallow line tells a well-behaved crawler to stay out, and most of the major AI crawlers do honor it. It is still a voluntary protocol: nothing enforces it, and a handful of crawlers are known to ignore it outright. For those, the only reliable block is at the edge — a Cloudflare WAF rule, an Nginx deny, or your host's firewall — which is why every profile that needs it carries a warning mark.
If you are unsure where your site currently stands, paste your robots.txt into the free auditor and get a verdict for all 61 crawlers at once, including the specific rules that are silently blocking AI engines.
Most-checked AI crawlers
Start with the crawlers that decide whether AI engines can read and cite your site.
Which crawlers should you actually allow?
Blocking everything is the fastest way to disappear from AI answers. Blocking nothing puts your content into training sets you may not want. The right policy differs by crawler type — search indexers feed citations, assistant fetchers drive live answers, and training collectors only take data. These guides walk through the trade-offs per category.
- robots.txt for AI Crawlers: Every Rule That Matters — A 30-second robots.txt refresher, what each AI bot costs you when blocked, three copy-paste policies and the syntax mistakes that silently break your rules.
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- llms.txt: The Proposed Standard for AI-Friendly Sites — llms.txt is a proposed markdown file at your site root that helps AI models find your best pages. Full format spec, a copy-paste example, and the tradeoffs.
- How to Measure AI Visibility: Metrics That Actually Matter — AI visibility has no industry-standard score yet. Track citation share, referral traffic, bot access rate and answer accuracy instead — here is how to do it.
Check your site against all 61 crawlers
Paste your robots.txt and get a per-crawler verdict, a score, and a fix list — free, no signup, runs in your browser.