robots.txt for AI Crawlers: Every Rule That Matters

The 30-second refresher

robots.txt is a plain text file at the root of your domain — https://example.com/robots.txt — that tells crawlers which paths they may fetch. The syntax fits in four lines of theory: a User-agent line opens a rule group, Disallow and Allow lines define paths, and each crawler obeys the single most specific group whose name matches its user agent. Disallow: / blocks the entire site, an empty Disallow value allows everything, and paths are case-sensitive on most servers. That is ninety percent of the syntax. The rest is knowing which bots exist and what each one is worth to you.

Two rules worth knowing before you write anything. A Sitemap: line tells every compliant crawler where your sitemap lives, independent of group order. And a crawler matches exactly one group — the best-fitting one — so a broad blanket group never silently stacks on top of a specific allow group. Each bot reads one group and one group only.

Who wants in: the three categories

Category	Who	What blocking costs you
Training bots	GPTBot, ClaudeBot, Bytespider, CCBot, Google-Extended, Applebot-Extended	Your content stops shaping future models. The loss is gradual and invisible: engines know and recommend you less over time.
Search-index bots	OAI-SearchBot, PerplexityBot	The immediate one. These bots feed live answers — block them and AI engines stop citing and linking to you today.
User-triggered fetchers	ChatGPT-User	Real-time reads when a user explicitly asks the assistant to open your page. Low volume, high intent.

The practical takeaway: most sites want categories two and three in, and decide consciously about category one. Blocking everything because a training-panic headline told you to also blocks the citations that send referral traffic. The crawler mechanics guide explains what each bot does with your content, and the directory lists every bot worth knowing.

Three policies you can paste today

Policy 1 — block training crawlers, keep AI search visibility. The balanced default for most businesses:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Bytespider
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
Disallow:

Several User-agent lines in one group apply the rules to all of them at once — no need to repeat the Disallow per bot. The empty Disallow in the second group explicitly allows those bots the whole site.

Policy 2 — allow everything. Maximum visibility, training included:

User-agent: *
Disallow:

An empty Disallow means fetch anything. Note that this is also exactly what happens when you have no robots.txt at all, and that removing a disallow rule takes effect only after each engine's next crawl — not the second you save the file.

Policy 3 — block all AI crawlers. For private, paywalled or legally sensitive content:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Bytespider
User-agent: CCBot
Disallow: /

Honest warning: this makes you invisible in AI answers, not merely untrained-on. And because Bytespider has a documented history of ignoring robots.txt, a file-only block is not enforcement — pair it with server-level user-agent or network rules if blocking genuinely matters.

robots.txt is the floor, not the ceiling. It can only forbid. To actively help AI engines understand your site — recommended pages, a map of your content, contact points — add an llms.txt file alongside it.

Syntax mistakes that silently break your rules

• Case does not matter for directives — it matters enormously for bot names. User-agent and Disallow can be written in any case. The bot name cannot: user-agent: gptbot matches nothing, because the crawler identifies as GPTBot. A misspelled name produces no error and no warning; the group simply applies to nobody, and the block you intended never happens.
• One directive per line. Disallow: /a /b is parsed as a single strange path, not two rules. Two paths need two lines. Same for Allow and Sitemap.
• Crawl-delay is not a universal standard. Google ignores it entirely, and several AI bots publish no support for it at all. Do not treat Crawl-delay as a load-protection strategy — throttle at the server or CDN level, which works against every client.
• Empty groups and empty values do opposite things. A group with a User-agent line but no rules does nothing, while Disallow: with no path means allow everything. People add the first expecting a block and write the second expecting a block, and get the opposite of both.

The cost of blocking, honestly

A word on scope first: robots.txt lives per host. A file on your main domain says nothing about a docs subdomain or an old staging domain, and the only supported wildcards are the asterisk for any character sequence and the dollar sign for end-of-URL — enough for patterns like blocking every query URL, but everything fancier belongs at the server layer.

Blocking is not free, and neither is allowing. Block the search-index bots and you lose the citation channel that links back with real referral traffic — a loss you can measure within weeks using the AI visibility metrics. Allow the training bots and your best content helps a model answer questions about your domain without you being named. There is no universally correct policy; there is only the one that matches what your content is worth and how you make money.

So decide per category, write the file, then verify it says what you think it says. Paste your robots.txt into the free Auditor for a per-bot verdict, or use the generator to produce a correct file from checkboxes instead of memorizing syntax. Both take under a minute and catch every silent mistake listed above.