Googlebot
Google uses this crawler to index pages that can be cited in its AI answers. Classic Google Search crawler. NEVER block unless you want out of Google entirely.
Crawler data last reviewed September 18, 2026 · all 61 crawlers
Googlebot at a glance
| Operator | |
|---|---|
| Type | AI search |
| Respects robots.txt | Yes — a Disallow rule is honored |
| If you allow it | Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from. |
| If you block it | Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited. |
What is Googlebot?
Classic Google Search crawler. NEVER block unless you want out of Google entirely. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.
Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.
Primary source: Google's official crawler documentation.
Does Googlebot respect robots.txt?
Yes. Googlebot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block Googlebot
Add this group to your robots.txt to deny Googlebot across your whole site:
User-agent: Googlebot Disallow: /
Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.
How to allow Googlebot
Googlebot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: Googlebot Disallow: User-agent: * Disallow: /
Does blocking Googlebot hurt your Google rankings?
Yes — this is the one crawler where the answer is unambiguous. Googlebot builds the main Google Search index. A robots.txt rule that blocks it removes your pages from Google Search and no other crawler compensates. If your goal is to keep AI training out while keeping search traffic, block Google-Extended instead and leave Googlebot fully open.
How to verify Googlebot is really hitting your site
robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:
grep -i "Googlebot" /var/log/nginx/access.log | tail -50
On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.
User agents are trivially spoofable, which is why "I saw Googlebot in my logs" is not proof. Because Googlebot is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.
Should you block Googlebot?
- Allow it — a search crawler is the only route from your site into Google citations, and citations are where AI referral traffic comes from.
- Do not block it casually. There is no partial state: an uncrawled page cannot be cited, and the engine will simply answer from a competitor's page.
- Review it if you recently tightened a global User-agent: * block. Global blocks are the most common reason a site silently disappears from AI answers.
Is Googlebot currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.
Other AI crawlers from Google
Google runs 9 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.
| Crawler | Type | robots.txt | What it does |
|---|---|---|---|
| Google-Extended | AI training | yes | Lets Google use content to train Gemini. Does NOT affect Search ranking. |
| Googlebot this page | AI search | yes | Classic Google Search crawler. NEVER block unless you want out of Google entirely. |
| Gemini-Deep-Research | AI assistant fetcher | yes | Fetches sources for Gemini Deep Research reports. |
| GoogleOther | AI training | yes | Google data collection outside the main search index. |
| GoogleOther-Image | AI training | yes | Google image data collection for research. |
| GoogleOther-Video | AI training | yes | Google video data collection for research. |
| Google-CloudVertexBot | AI training | yes | Builds models on customer's own Vertex AI instances (site-owner opted in). |
| Google-Agent | AI agent browser | yes | Fetches pages during Google AI agent tasks on a user's behalf. |
| Google-Gemini-CLI | AI agent browser | yes | Fetches pages explicitly requested by developers in Gemini CLI. |
Related AI crawlers
Further reading
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- How AI Crawlers Work (And What They Want From Your Site) — AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.