Claude-SearchBot
Anthropic uses this crawler to index pages that can be cited in its AI answers. Powers Claude's search citation index.
Crawler data last reviewed September 18, 2026 · all 61 crawlers
Claude-SearchBot at a glance
| Operator | Anthropic |
|---|---|
| Type | AI search |
| Respects robots.txt | Yes — a Disallow rule is honored |
| If you allow it | Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from. |
| If you block it | Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited. |
What is Claude-SearchBot?
Powers Claude's search citation index. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.
Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.
Primary source: Anthropic's official crawler documentation.
Does Claude-SearchBot respect robots.txt?
Yes. Claude-SearchBot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block Claude-SearchBot
Add this group to your robots.txt to deny Claude-SearchBot across your whole site:
User-agent: Claude-SearchBot Disallow: /
Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.
How to allow Claude-SearchBot
Claude-SearchBot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: Claude-SearchBot Disallow: User-agent: * Disallow: /
Does blocking Claude-SearchBot hurt your rankings?
Not in Google — but it does cost you AI visibility. Claude-SearchBot is not used by Google Search, so classic rankings are unaffected. What changes is whether Anthropic can cite your pages in AI answers. That is a separate discovery channel, and for a AI search token, blocking it removes you from it completely.
How to verify Claude-SearchBot is really hitting your site
robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:
grep -i "Claude-SearchBot" /var/log/nginx/access.log | tail -50
On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.
User agents are trivially spoofable, which is why "I saw Claude-SearchBot in my logs" is not proof. Because Claude-SearchBot is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.
Should you block Claude-SearchBot?
- Allow it — a search crawler is the only route from your site into Anthropic citations, and citations are where AI referral traffic comes from.
- Do not block it casually. There is no partial state: an uncrawled page cannot be cited, and the engine will simply answer from a competitor's page.
- Review it if you recently tightened a global User-agent: * block. Global blocks are the most common reason a site silently disappears from AI answers.
Is Claude-SearchBot currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.
Other AI crawlers from Anthropic
Anthropic runs 6 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.
| Crawler | Type | robots.txt | What it does |
|---|---|---|---|
| ClaudeBot | AI training | yes | Crawls content to train Claude models. |
| Claude-User | AI assistant fetcher | yes | Fetches pages when a Claude user asks about them. Not used for training. |
| Claude-SearchBot this page | AI search | yes | Powers Claude's search citation index. |
| Claude-Web | AI assistant fetcher | yes | Legacy web-fetch user agent for Claude. |
| Claude-Code | AI agent browser | yes | Fetches docs/pages requested during Claude Code sessions. |
| anthropic-ai | AI training | yes | Legacy training crawler. |
Related AI crawlers
Further reading
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- How AI Crawlers Work (And What They Want From Your Site) — AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.