YouBot

What is YouBot?

You.com search and answer engine indexer. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.

Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.

Does YouBot respect robots.txt?

Yes. YouBot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.

How to block YouBot

Add this group to your robots.txt to deny YouBot across your whole site:

User-agent: YouBot
Disallow: /

Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.

How to allow YouBot

YouBot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:

User-agent: YouBot
Disallow:

User-agent: *
Disallow: /