Googlebot
Google uses this crawler to index pages that can be cited in its AI answers. Classic Google Search crawler. NEVER block unless you want out of Google entirely.
What is Googlebot?
Classic Google Search crawler. NEVER block unless you want out of Google entirely. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.
Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.
Does Googlebot respect robots.txt?
Yes. Googlebot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block Googlebot
Add this group to your robots.txt to deny Googlebot across your whole site:
User-agent: Googlebot Disallow: /
Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.
How to allow Googlebot
Googlebot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: Googlebot Disallow: User-agent: * Disallow: /
Is Googlebot currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.