Claude-SearchBot

What is Claude-SearchBot?

Powers Claude's search citation index. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.

Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.

Does Claude-SearchBot respect robots.txt?

Yes. Claude-SearchBot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.

How to block Claude-SearchBot

Add this group to your robots.txt to deny Claude-SearchBot across your whole site:

User-agent: Claude-SearchBot
Disallow: /

Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.

How to allow Claude-SearchBot

Claude-SearchBot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:

User-agent: Claude-SearchBot
Disallow:

User-agent: *
Disallow: /