Google-Agent

Google-Agent at a glance

Operator	Google
Type	AI agent browser
Respects robots.txt	Yes — a Disallow rule is honored
If you allow it	Allowing it keeps your site reachable to AI agents acting for users — an emerging traffic channel worth keeping open.
If you block it	Blocking it stops agent-driven visits but provides no real content protection, since agents only fetch pages the user asked for.

What is Google-Agent?

Fetches pages during Google AI agent tasks on a user's behalf. Agent crawlers browse the web on a user's behalf inside AI tools: opening pages, reading documentation, and completing tasks. Traffic arrives from engaged users who asked for exactly what you offer. Blocking agents does not protect content from training, but it does cut off this kind of high-intent visit.

Allowing it keeps your site reachable to AI agents acting for users — an emerging traffic channel worth keeping open.

Primary source: Google's official crawler documentation.

Does Google-Agent respect robots.txt?

Yes. Google-Agent honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.

How to block Google-Agent

Add this group to your robots.txt to deny Google-Agent across your whole site:

User-agent: Google-Agent
Disallow: /

Blocking it stops agent-driven visits but provides no real content protection, since agents only fetch pages the user asked for.

How to allow Google-Agent

Google-Agent is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:

User-agent: Google-Agent
Disallow:

User-agent: *
Disallow: /

Does blocking Google-Agent hurt your Google rankings?

No. Google-Agent is a distinct user agent from Googlebot and is evaluated separately in robots.txt. Blocking it does not change your Google Search ranking, indexing or classic organic traffic. Google documents this separation whenever it introduces a new agent, which is exactly why these AI-specific tokens exist.

How to verify Google-Agent is really hitting your site

robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:

grep -i "Google-Agent" /var/log/nginx/access.log | tail -50

On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.

User agents are trivially spoofable, which is why "I saw Google-Agent in my logs" is not proof. Because Google-Agent is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.

Should you block Google-Agent?

• Allow it — agent traffic is a user asking an AI tool to do something on the open web, which is closer to a real visitor than a crawler is.
• Block it if the agent could trigger writes, purchases or state changes on your site, and you cannot tell agent traffic apart downstream.
• Expect growth here — agent browsers are the newest category and the least covered by classic bot management.

Other AI crawlers from Google

Google runs 9 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.

Crawler	Type	robots.txt	What it does
Google-Extended	AI training	yes	Lets Google use content to train Gemini. Does NOT affect Search ranking.
Googlebot	AI search	yes	Classic Google Search crawler. NEVER block unless you want out of Google entirely.
Gemini-Deep-Research	AI assistant fetcher	yes	Fetches sources for Gemini Deep Research reports.
GoogleOther	AI training	yes	Google data collection outside the main search index.
GoogleOther-Image	AI training	yes	Google image data collection for research.
GoogleOther-Video	AI training	yes	Google video data collection for research.
Google-CloudVertexBot	AI training	yes	Builds models on customer's own Vertex AI instances (site-owner opted in).
Google-Agent this page	AI agent browser	yes	Fetches pages during Google AI agent tasks on a user's behalf.
Google-Gemini-CLI	AI agent browser	yes	Fetches pages explicitly requested by developers in Gemini CLI.

Further reading

• How AI Crawlers Work (And What They Want From Your Site) — AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.
• robots.txt for AI Crawlers: Every Rule That Matters — A 30-second robots.txt refresher, what each AI bot costs you when blocked, three copy-paste policies and the syntax mistakes that silently break your rules.

All 14 GEO guides · Full AI crawler directory