About CrawlCheck
A free toolkit for Generative Engine Optimization — because an uncrawled page cannot be cited by AI answers.
What CrawlCheck is
CrawlCheck is a free toolkit for Generative Engine Optimization (GEO) — making sure your website stays visible when people ask ChatGPT, Perplexity, Claude, Gemini or Copilot about your topic. Search engines needed robots.txt; AI answers need it even more, because a page that is never crawled is a page that is never cited.
Why it exists
Most site owners have no idea that dozens of AI crawlers visit their robots.txt every day — some building search indexes, some collecting training data, some fetching pages for live assistants. The settings are buried, the trade-offs are undocumented, and enterprise SEO suites charge monthly fees to explain them. CrawlCheck gives the same clarity for free, instantly, in your browser.
What's inside
Auditor — paste your robots.txt and get a verdict for every AI crawler, per path, plus a score and a fix list.
Generator — pick your visibility strategy per crawler and get a clean, valid policy.
Schema — generate FAQ and HowTo JSON-LD that answer engines parse well.
Scanner — live-scan your site as 8 AI crawlers: real fetches, per-bot robots.txt verdicts, one visibility score.
Crawler directory — 61 AI bots profiled: operator, purpose, robots.txt behavior, and blocking instructions.
Where the data comes from
The crawler database is built from the open-source ai.robots.txt community project and vendor documentation, then reviewed and extended with practical blocking recipes. CrawlCheck is not affiliated with any AI vendor. Spotted something outdated? Tell us — the directory is updated as vendors change their bots.
How the project sustains itself
The core tools stay free. We may add optional paid services later (such as automated monitoring) and, if the site grows, unobtrusive advertising. Whatever changes: the free audit stays free, and the site keeps its cookie-free, privacy-first approach.
Who builds it
CrawlCheck is an independent project run by web developers who have spent their careers making sites findable — and who think the AI-answer era deserves the same open tooling the search era had.
See what AI engines can see on your site
Audit your robots.txt against 61 AI crawlers in seconds — free, no signup, runs entirely in your browser.