OAI-SearchBot
OpenAI uses this crawler to index pages that can be cited in its AI answers. Powers ChatGPT Search citations. Blocking = invisible in ChatGPT search results.
Crawler data last reviewed September 18, 2026 · all 61 crawlers
OAI-SearchBot at a glance
| Operator | OpenAI |
|---|---|
| Type | AI search |
| Respects robots.txt | Yes — a Disallow rule is honored |
| If you allow it | Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from. |
| If you block it | Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited. |
What is OAI-SearchBot?
Powers ChatGPT Search citations. Blocking = invisible in ChatGPT search results. Search crawlers build the index behind AI answer engines. When a user asks a question, the engine searches this index and cites matching pages with a link. No crawl means no citation, no link, no referral traffic — allowing search crawlers is the single highest-impact step for AI-era visibility.
Allowing it puts your pages in the citation pool for AI answers, which is where referral traffic increasingly comes from.
Primary source: OpenAI's official crawler documentation.
Does OAI-SearchBot respect robots.txt?
Yes. OAI-SearchBot honors robots.txt directives, including wildcard rules and Allow lines. A Disallow rule for this crawler is respected — but remember robots.txt is a voluntary protocol across the whole industry, not a wall.
How to block OAI-SearchBot
Add this group to your robots.txt to deny OAI-SearchBot across your whole site:
User-agent: OAI-SearchBot Disallow: /
Blocking it removes you from that engine's answers entirely. There is no middle ground: uncrawled pages cannot be cited.
How to allow OAI-SearchBot
OAI-SearchBot is allowed by default on every site. It only gets blocked if a matching rule exists — most often a global User-agent: * group with Disallow: /. To exempt it while blocking other bots, put an empty Disallow group for it before the global block:
User-agent: OAI-SearchBot Disallow: User-agent: * Disallow: /
Does blocking OAI-SearchBot hurt your rankings?
Not in Google — but it does cost you AI visibility. OAI-SearchBot is not used by Google Search, so classic rankings are unaffected. What changes is whether OpenAI can cite your pages in AI answers. That is a separate discovery channel, and for a AI search token, blocking it removes you from it completely.
How to verify OAI-SearchBot is really hitting your site
robots.txt states what a compliant crawler may fetch; your access log shows what actually happened. Search the log for the token:
grep -i "OAI-SearchBot" /var/log/nginx/access.log | tail -50
On Cloudflare the free plan does not expose raw request logs, so use Security Events to see blocked requests, or add a console.log line in a Worker to record the user agent. On Vercel or Netlify, filter the request log by user agent string.
User agents are trivially spoofable, which is why "I saw OAI-SearchBot in my logs" is not proof. Because OAI-SearchBot is documented to honor robots.txt, a request from it that ignores your rules is worth investigating — but a single stray hit is usually a misconfigured tool, not the real crawler.
Should you block OAI-SearchBot?
- Allow it — a search crawler is the only route from your site into OpenAI citations, and citations are where AI referral traffic comes from.
- Do not block it casually. There is no partial state: an uncrawled page cannot be cited, and the engine will simply answer from a competitor's page.
- Review it if you recently tightened a global User-agent: * block. Global blocks are the most common reason a site silently disappears from AI answers.
Is OAI-SearchBot currently allowed on your site?
Paste your robots.txt into the free auditor and see verdicts for all 61 AI crawlers — including the specific paths that matter.
Other AI crawlers from OpenAI
OpenAI runs 4 distinct user agents. They are configured independently in robots.txt, so blocking one does not block the others — a detail that catches people out when they assume a single rule covers a whole vendor.
| Crawler | Type | robots.txt | What it does |
|---|---|---|---|
| GPTBot | AI training | yes | Feeds GPT model training. Blocking removes your content from future ChatGPT knowledge. |
| OAI-SearchBot this page | AI search | yes | Powers ChatGPT Search citations. Blocking = invisible in ChatGPT search results. |
| ChatGPT-User | AI assistant fetcher | yes | Fetches pages a ChatGPT user explicitly requested. Blocking breaks user-initiated browsing. |
| ChatGPT Agent | AI agent browser | yes | Browses the web on a user's behalf during ChatGPT agent tasks. |
Related AI crawlers
Further reading
- How to Get Cited by ChatGPT, Perplexity and Copilot — ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- How AI Crawlers Work (And What They Want From Your Site) — AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.