ClaudeBot
Anthropic uses this crawler to collect content for AI model training. Crawls content to train Claude models.
Crawler data last reviewed September 18, 2026 · all 61 crawlers
ClaudeBot at a glance
| Operator | Anthropic |
|---|---|
| Type | AI training |
| Respects robots.txt | Yes — a Disallow rule is honored |
| If you allow it | If you want AI engines to know your brand and quote your pages from memory, this is the crawler that gets you there. |
| If you block it | If you monetize content behind a paywall or care about control, allowing broad training crawlers means giving that material away with no attribution loop. |
What is ClaudeBot?
Crawls content to train Claude models. Training-data crawlers collect publicly available pages to build future AI models. Allowing one means your content can shape how models write and reason about your topics, and your site may be quoted from model memory. Blocking one keeps future training runs away, though anything already collected typically stays collected.
If you want AI engines to know your brand and quote your pages from memory, this is the crawler that gets you there.
Primary source: Anthropic's official crawler documentation.
Does ClaudeBot respect robots.txt?
Yes. ClaudeBot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block ClaudeBot
Add this group to your robots.txt to deny ClaudeBot across your whole site:
User-agent: ClaudeBot Disallow: /
If you monetize content behind a paywall or care about control, allowing broad training crawlers means giving that material away with no attribution loop.
How to allow ClaudeBot
ClaudeBot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: ClaudeBot Disallow: User-agent: * Disallow: /
Does blocking ClaudeBot hurt your rankings?
No — not for classic search. ClaudeBot is not a Google user agent, so a robots.txt rule for it has no effect on Google rankings or indexing. The only thing you give up is what this specific AI training does with your pages, described above.
How to verify ClaudeBot is really hitting your site
robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:
grep -i "ClaudeBot" /var/log/nginx/access.log | tail -50
On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.
User agents are trivially spoofable, which is why "I saw ClaudeBot in my logs" is not proof. Because ClaudeBot is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.
Should you block ClaudeBot?
- Allow it if you want your brand and expertise inside future model knowledge — being in the training corpus is how an engine learns to describe what you do.
- Block it if you sell the content itself (paywall, course, licensed dataset) and want to keep it out of training runs. Remember that a block stops future collection only; anything already crawled stays in the dataset.
- Split the decision by keeping training crawlers out while leaving the operator's search crawler open, so you stay citable in AI answers without feeding model training.
Is ClaudeBot currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.
Other AI crawlers from Anthropic
Anthropic runs 6 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.
| Crawler | Type | robots.txt | What it does |
|---|---|---|---|
| ClaudeBot this page | AI training | yes | Crawls content to train Claude models. |
| Claude-User | AI assistant fetcher | yes | Fetches pages when a Claude user asks about them. Not used for training. |
| Claude-SearchBot | AI search | yes | Powers Claude's search citation index. |
| Claude-Web | AI assistant fetcher | yes | Legacy web-fetch user agent for Claude. |
| Claude-Code | AI agent browser | yes | Fetches docs/pages requested during Claude Code sessions. |
| anthropic-ai | AI training | yes | Legacy training crawler. |
Related AI crawlers
Further reading
- robots.txt for AI Crawlers: Every Rule That Matters — A 30-second robots.txt refresher, what each AI bot costs you when blocked, three copy-paste policies and the syntax mistakes that silently break your rules.
- llms.txt: The Proposed Standard for AI-Friendly Sites — llms.txt is a proposed markdown file at your site root that helps AI models find your best pages. Full format spec, a copy-paste example, and the tradeoffs.