Every page on CrawlCheck
A plain HTML index of the whole site — 61 crawler profiles, 10 guides, four tools and the policy pages. Useful if a link somewhere is broken, and useful for any crawler that prefers links to a sitemap file.
Tools
- CrawlCheck home — robots.txt auditor, policy generator and JSON-LD schema builder
- AI visibility scanner — live-fetch your site as 8 AI crawlers
- AI crawler directory — all 61 profiles in one page
- GEO guides — 10 practical playbooks
AI crawler profiles
Each profile covers who operates the crawler, what it does with your content, whether a robots.txt rule is actually honored, and how to allow or block it.
AI search crawlers (10)
- OAI-SearchBot OpenAI
- Claude-SearchBot Anthropic
- PerplexityBot Perplexity AI
- Googlebot Google
- Applebot Apple
- Amazonbot Amazon
- Bingbot Microsoft
- DuckDuckBot DuckDuckGo
- PetalBot Huawei
- YouBot You.com
AI assistant fetchers (11)
- ChatGPT-User OpenAI
- Claude-User Anthropic
- Perplexity-User Perplexity AI
- Gemini-Deep-Research Google
- DuckAssistBot DuckDuckGo
- MistralAI-User Mistral AI
- Amzn-User Amazon
- Claude-Web Anthropic
- cohere-ai Cohere
- Copilot/Microsoft Microsoft
- Diffbot-User Diffbot
AI agent browsers (9)
- ChatGPT Agent OpenAI
- AmazonBuyForMe Amazon
- Claude-Code Anthropic
- Cursor Cursor (Anysphere)
- Devin Cognition
- Google-Agent Google
- Google-Gemini-CLI Google
- viberank VibeRank
- Windsurf Windsurf
AI training data crawlers (31)
- GPTBot OpenAI
- ClaudeBot Anthropic
- Google-Extended Google
- Applebot-Extended Apple
- Bytespider ByteDance · ignores robots.txt
- CCBot Common Crawl
- meta-externalagent Meta
- TikTokSpider ByteDance
- PanguBot Huawei
- AI2Bot Allen Institute for AI
- Ai2Bot-Dolma Allen Institute for AI
- anthropic-ai Anthropic
- Bedrockbot Amazon
- cohere-training-data-crawler Cohere
- DeepSeekBot DeepSeek · ignores robots.txt
- Diffbot Diffbot
- FacebookBot Meta
- FriendlyCrawler Unknown
- Google-CloudVertexBot Google
- GoogleOther Google
- GoogleOther-Image Google
- GoogleOther-Video Google
- ICC-Crawler NICT (Japan)
- ImagesiftBot Hive AI
- img2dataset img2dataset project
- ISSCyberRiskCrawler ISS-Corporate · ignores robots.txt
- LAIONDownloader LAION · ignores robots.txt
- MistralAI-Training Mistral AI
- Omgili Webz.io
- Omgilibot Webz.io
- YandexAdditional Yandex
Guides
- What is Generative Engine Optimization (GEO)? — GEO is the practice of making your site readable, extractable and citable by AI answer engines. What it covers, why it matters, and the first five steps to take.
- GEO vs SEO: What Changes and What Doesn't — SEO is not dead, but AI answer engines added a second set of rules. A practical comparison of goals, crawl access, content format and how to measure success.
- How AI Crawlers Work (And What They Want From Your Site) — AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.
- robots.txt for AI Crawlers: Every Rule That Matters — A 30-second robots.txt refresher, what each AI bot costs you when blocked, three copy-paste policies and the syntax mistakes that silently break your rules.
- llms.txt: The Proposed Standard for AI-Friendly Sites — llms.txt is a proposed markdown file at your site root that helps AI models find your best pages. Full format spec, a copy-paste example, and the tradeoffs.
- How to Measure AI Visibility: Metrics That Actually Matter — AI visibility has no industry-standard score yet. Track citation share, referral traffic, bot access rate and answer accuracy instead — here is how to do it.
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- Content Structure for AI: Pages That Get Extracted and Cited — AI answer engines extract pages in chunks, not wholes. Structure rules that make content quotable: question headings, answer-first paragraphs, tables, FAQs.
- Structured Data for AI: The Schema.org Markup Answer Engines Read — JSON-LD explained in 30 seconds, the schema types answer engines actually use, a copy-paste FAQPage example, and how to validate your markup for free.
- 10 GEO Mistakes That Keep Your Site Out of AI Answers — Ten fixable mistakes that keep sites out of AI answers: blanket crawler bans, JS-only content, prose walls, stale pages, schema spam and more. Each with a fix.
About this site
- About CrawlCheck
- CrawlCheck.dev vs crawlcheck.io
- Contact
- Privacy policy
- Terms
- Refund policy
- XML sitemap
- llms.txt
Sitemap last generated October 19, 2026.
Start with the audit
Paste your robots.txt and see a verdict for all 61 AI crawlers — free, no signup, runs in your browser.