Gemini-Deep-Research
Google uses this crawler to fetch pages in real time when a user asks about them. Fetches sources for Gemini Deep Research reports.
Crawler data last reviewed September 18, 2026 · all 61 crawlers
Gemini-Deep-Research at a glance
| Operator | |
|---|---|
| Type | AI assistant fetcher |
| Respects robots.txt | Yes — a Disallow rule is honored |
| If you allow it | Allowing it means users who ask about you get an accurate, current answer with your page as the source. |
| If you block it | Blocking it only affects user-requested fetches, so the traffic cost is small — but the users you do lose are the highest-intent ones. |
What is Gemini-Deep-Research?
Fetches sources for Gemini Deep Research reports. Assistant crawlers fetch a page in real time when a user explicitly asks an AI assistant about it. They do not build a standing index of the web. Blocking them makes the assistant say it cannot access your page instead of summarizing it — a poor experience for users who were actively looking for you.
Allowing it means users who ask about you get an accurate, current answer with your page as the source.
Primary source: Google's official crawler documentation.
Does Gemini-Deep-Research respect robots.txt?
Yes. Gemini-Deep-Research honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block Gemini-Deep-Research
Add this group to your robots.txt to deny Gemini-Deep-Research across your whole site:
User-agent: Gemini-Deep-Research Disallow: /
Blocking it only affects user-requested fetches, so the traffic cost is small — but the users you do lose are the highest-intent ones.
How to allow Gemini-Deep-Research
Gemini-Deep-Research is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: Gemini-Deep-Research Disallow: User-agent: * Disallow: /
Does blocking Gemini-Deep-Research hurt your Google rankings?
No. Gemini-Deep-Research is a distinct user agent from Googlebot and is evaluated separately in robots.txt. Blocking it does not change your Google Search ranking, indexing or classic organic traffic. Google documents this separation whenever it introduces a new agent, which is exactly why these AI-specific tokens exist.
How to verify Gemini-Deep-Research is really hitting your site
robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:
grep -i "Gemini-Deep-Research" /var/log/nginx/access.log | tail -50
On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.
User agents are trivially spoofable, which is why "I saw Gemini-Deep-Research in my logs" is not proof. Because Gemini-Deep-Research is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.
Should you block Gemini-Deep-Research?
- Allow it — it only fetches pages a user explicitly asked about, so the traffic is high-intent and the volume is low.
- Block it only if a page must never be summarized by an assistant, and accept that the user will be told the page is unavailable.
- Check it after CDN changes — assistant fetchers are frequently caught by bot-fighting rules aimed at scrapers.
Is Gemini-Deep-Research currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.
Other AI crawlers from Google
Google runs 9 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.
| Crawler | Type | robots.txt | What it does |
|---|---|---|---|
| Google-Extended | AI training | yes | Lets Google use content to train Gemini. Does NOT affect Search ranking. |
| Googlebot | AI search | yes | Classic Google Search crawler. NEVER block unless you want out of Google entirely. |
| Gemini-Deep-Research this page | AI assistant fetcher | yes | Fetches sources for Gemini Deep Research reports. |
| GoogleOther | AI training | yes | Google data collection outside the main search index. |
| GoogleOther-Image | AI training | yes | Google image data collection for research. |
| GoogleOther-Video | AI training | yes | Google video data collection for research. |
| Google-CloudVertexBot | AI training | yes | Builds models on customer's own Vertex AI instances (site-owner opted in). |
| Google-Agent | AI agent browser | yes | Fetches pages during Google AI agent tasks on a user's behalf. |
| Google-Gemini-CLI | AI agent browser | yes | Fetches pages explicitly requested by developers in Gemini CLI. |
Related AI crawlers
Further reading
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- Content Structure for AI: Pages That Get Extracted and Cited — AI answer engines extract pages in chunks, not wholes. Structure rules that make content quotable: question headings, answer-first paragraphs, tables, FAQs.