OAI-SearchBot: what it is, what it does with your content, and what blocking it costs
What OAI-SearchBot is
OAI-SearchBot builds the retrieval index that OpenAI answers from. This is the category that decides whether you can be cited right now: when someone asks a question and the assistant goes looking for sources, it searches an index this crawler built. Blocking it removes you from those answers.
Operator: OpenAI
robots.txt token: OAI-SearchBot
User-agent on the wire: Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot
Category: Live retrieval / search index
What blocking it actually costs
You are removed from the index ChatGPT search draws on. This is the one that costs you citations today.
How to block OAI-SearchBot
Add this to your robots.txt:
User-agent: OAI-SearchBot
Disallow: /
The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.
How to allow OAI-SearchBot
User-agent: OAI-SearchBot
Allow: /
An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.
robots.txt is not the only thing that can block it
A permissive robots.txt does not mean OAI-SearchBot can reach you. WAF rules, Cloudflare's bot-management settings, rate limits and country blocks all sit in front of robots.txt and answer first. A site whose robots.txt welcomes OAI-SearchBot and whose edge returns 403 to it is blocked in every way that matters — and nothing in robots.txt will tell you so.
This is the gap the AI Crawler Access Checker was built to close: it reads robots.txt and sends a real request carrying the crawler's user-agent, so you see what the crawler sees.
Official documentation
OpenAI documents OAI-SearchBot at https://platform.openai.com/docs/bots.
Common questions
- What user-agent does OAI-SearchBot send?
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot- How do I block OAI-SearchBot?
Add a group to robots.txt naming the token exactly:
User-agent: OAI-SearchBot Disallow: /The group must name
OAI-SearchBoton its own line. AUser-agent: *group does not combine with a named one — under RFC 9309 the most specific matching group replaces the wildcard entirely, it does not inherit from it.- What does blocking OAI-SearchBot cost me?
You are removed from the index ChatGPT search draws on. This is the one that costs you citations today.
- How do I verify a request really came from OAI-SearchBot?
OpenAI publishes the IP ranges it crawls from at https://openai.com/searchbot.json. A user-agent string is trivially spoofed, so anything acting on OAI-SearchBot traffic — rate limits, firewall rules, analytics segments — should check the address against that list rather than trusting the header.
Is OAI-SearchBot blocked on your site right now?
Reading your own robots.txt only answers half of it — a firewall rule can block OAI-SearchBot while robots.txt says it is welcome. The checker tests both.
Check your site free