Control what AI sees on your site
ChatGPT, Perplexity and Claude now decide who gets discovered. Audit your robots.txt against 60+ known AI crawlers, then generate a policy in seconds — free, no signup, runs entirely in your browser.
AI Crawler Auditor
Paste your robots.txt below. We parse it against our database of 60+ AI crawlers and show exactly who is allowed, who is blocked, and what it costs you.
robots.txt Generator
Pick a preset or fine-tune crawler by crawler. The file updates live — copy it to https://yourdomain.com/robots.txt when you're happy.
Generated robots.txt
Structured Data Generator
FAQ, HowTo and Article schema are what AI answer engines quote most easily. Fill the form, paste the JSON-LD into your page's <head>.
Generated JSON-LD
Why this matters now
AI referral traffic is exploding
Referrals from LLM apps grew faster than any channel in 2025–2026. Early sites are capturing citations that compound as AI search becomes the default discovery layer.
Your robots.txt decides it
One Disallow: / can make you invisible across ChatGPT, Perplexity and Claude at once. Most sites were configured years ago, before AI crawlers existed.
Dozens of crawlers, no visibility
GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended… nobody audits them by hand. CrawlCheck parses your file against a maintained database in one click.
GEO guides — from audit to fix
Every guide ends with something you can check or change with the tools above. Start here, then work through the rest in order.
What is Generative Engine Optimization?
What GEO covers, why AI answers ignore most sites, and the first five steps to take.
robots.txt for AI Crawlers
Every rule that matters, what each blocked bot costs you, and three copy-paste policies.
How to Get Cited by ChatGPT & Perplexity
The fetch–retrieve–extract pipeline, what each stage rewards, and a 10-point checklist.
llms.txt Explained
The proposed AI-friendly index file: full format spec, example, and the tradeoffs.
How to Measure AI Visibility
Citation share, referral traffic, bot access rate — the metrics that actually move.
10 GEO Mistakes to Avoid
Blanket crawler bans, JS-only content, prose walls, schema spam — each with a fix.
FAQ
What is GEO (Generative Engine Optimization)?
GEO is the practice of making your content easy for AI answer engines — ChatGPT, Perplexity, Claude, Gemini — to find, trust and cite. The foundation is technical accessibility: a robots.txt policy that lets the right AI crawlers in.
Does blocking GPTBot hurt my Google rankings?
No. GPTBot only trains OpenAI models. Google Search uses Googlebot, a separate crawler. Blocking GPTBot has zero effect on your Google rankings, and blocking Google-Extended does not affect Search either.
Some AI crawlers ignore robots.txt — what can I do?
robots.txt is a request, not a wall. For crawlers known to ignore it (like Bytespider), enforce blocks at the CDN or WAF level, e.g. Cloudflare user-agent rules.
Is my data sent anywhere?
No. All parsing, auditing and generation runs locally in your browser. Nothing is uploaded, logged or tracked.