Can AI engines actually read your site?
We fire real requests at your site as GPTBot, ClaudeBot, PerplexityBot and 5 more AI crawlers — then grade what they can see, fetch and cite.
3 free scans per day · no signup · results in ~10 seconds
AI crawler access — 8 engines
| Crawler | Org | Role | robots.txt | Live fetch | Verdict |
|---|
Findings
What this scanner does
Path-level allow/deny judgment for 8 AI crawlers, including training, search and browsing agents.
We actually fetch your page as each bot — catching server-level blocks and 403s that robots.txt never shows.
Compares bot vs. browser content. If an AI crawler gets far less text than Chrome, your score says so.
Structured data (JSON-LD), llms.txt, noindex tags and extractable text — the things AI engines quote from.
Scanner FAQ
What does the scanner check?
Eight things at once: your robots.txt as seen by each AI crawler, a live fetch of your page as each bot, server-level blocks, a bot-vs-browser cloaking comparison, JSON-LD structured data, llms.txt, noindex tags and extractable text. Everything rolls up into one AI visibility score from 0 to 100.
How is this different from the Auditor?
The Auditor analyzes the robots.txt you paste — instantly, entirely in your browser. The Scanner goes further: it fetches your live site as each AI crawler, so it catches 403s, server-level blocks and cloaking that no robots.txt file can reveal. Use the Auditor when you are editing a policy; use the Scanner to see the real world effect.
Is it really free?
Yes — 3 scans per day, no signup, no email required. The engine runs on our own infrastructure at api.crawlcheck.dev.
Fix the policy too
The Scanner shows what is blocked today. The Auditor grades your robots.txt line by line — and the Generator writes the policy you actually want.