Control what AI sees on your site
ChatGPT, Perplexity and Claude now decide who gets discovered. Audit your robots.txt against 60+ known AI crawlers, then generate a policy in seconds — free, no signup, runs entirely in your browser.
AI Crawler Auditor
Paste your robots.txt below. We parse it against our database of 60+ AI crawlers and show exactly who is allowed, who is blocked, and what it costs you.
robots.txt Generator
Pick a preset or fine-tune crawler by crawler. The file updates live — copy it to https://yourdomain.com/robots.txt when you're happy.
Generated robots.txt
Structured Data Generator
FAQ, HowTo and Article schema are what AI answer engines quote most easily. Fill the form, paste the JSON-LD into your page's <head>.
Generated JSON-LD
Why this matters now
AI referral traffic is exploding
Referrals from LLM apps grew faster than any channel in 2025–2026. Early sites are capturing citations that compound as AI search becomes the default discovery layer.
Your robots.txt decides it
One Disallow: / can make you invisible across ChatGPT, Perplexity and Claude at once. Most sites were configured years ago, before AI crawlers existed.
Dozens of crawlers, no visibility
GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot, Google-Extended… nobody audits them by hand. CrawlCheck parses your file against a maintained database in one click.
GEO guides — from audit to fix
Every guide ends with something you can check or change with the tools above. Start here, then work through the rest in order.
What is Generative Engine Optimization?
What GEO covers, why AI answers ignore most sites, and the first five steps to take.
robots.txt for AI Crawlers
Every rule that matters, what each blocked bot costs you, and three copy-paste policies.
How to Get Cited by ChatGPT & Perplexity
The fetch–retrieve–extract pipeline, what each stage rewards, and a 10-point checklist.
llms.txt Explained
The proposed AI-friendly index file: full format spec, example, and the tradeoffs.
How to Measure AI Visibility
Citation share, referral traffic, bot access rate — the metrics that actually move.
10 GEO Mistakes to Avoid
Blanket crawler bans, JS-only content, prose walls, schema spam — each with a fix.
The crawlers that decide who gets cited
Each profile shows who runs the bot, what it does with your content, whether it actually honors robots.txt, and the exact rule to allow or block it.
GPTBot
OpenAI's training crawler. Blocking it removes your pages from future ChatGPT knowledge.
OAI-SearchBot
Powers ChatGPT Search citations. Blocking it makes your site invisible in ChatGPT answers.
ChatGPT-User
Fetches pages a user explicitly asked about. Blocking it breaks live browsing.
ChatGPT Agent
Browses on a user's behalf during agent tasks. Same allow/block logic as ChatGPT-User.
ClaudeBot
Anthropic's training crawler. Separate from Claude-User, which only answers live queries.
PerplexityBot
Builds the index behind Perplexity answers. A blocked page can never be a cited source.
Perplexity-User
Fetches pages for user-initiated queries. Not used for training.
Google-Extended
Controls Gemini training only. Blocking it does not touch Google Search rankings.
Applebot-Extended
Opt-in crawler for Apple Intelligence training, separate from Applebot Search.
CCBot
Feeds Common Crawl, the corpus most LLMs were trained on. Block it and you leave the corpus.
Bytespider
ByteDance's crawler. Known to ignore robots.txt — you need a WAF rule, not a text file.
Claude-User
Fetches pages when a Claude user asks about them. Not used for training.
Free scan vs Pro Deep Report
Free scan
$0 · no signupAudit one URL: 8 AI crawlers live-tested against your robots.txt, plus llms.txt and sitemap health — with a shareable report and concrete fixes.
Pro Deep Report
$49 · one-time per siteSweeps up to 50 pages of your site, live-tests 8 AI crawlers on every page, catches cloaking, broken pages and noindex traps — and returns a prioritized P0–P3 fix list at a private URL you can hand straight to your developer.
One-time payment · no subscription · 30-day money-back guarantee · your license key appears the moment payment clears.
FAQ
What is GEO (Generative Engine Optimization)?
GEO is the practice of making your content easy for AI answer engines — ChatGPT, Perplexity, Claude, Gemini — to find, trust and cite. The foundation is technical accessibility: a robots.txt policy that lets the right AI crawlers in.
Does blocking GPTBot hurt my Google rankings?
No. GPTBot only trains OpenAI models. Google Search uses Googlebot, a separate crawler. Blocking GPTBot has zero effect on your Google rankings, and blocking Google-Extended does not affect Search either.
Some AI crawlers ignore robots.txt — what can I do?
robots.txt is a request, not a wall. For crawlers known to ignore it (like Bytespider), enforce blocks at the CDN or WAF level, e.g. Cloudflare user-agent rules.
Is my data sent anywhere?
It depends on the tool. The Auditor and Generator run entirely in your browser, so nothing is uploaded. The Scanner needs one server round-trip to fetch your page as each AI crawler, and the Pro Deep Report does the same across up to 50 pages; in both cases results are returned to your browser and not stored.