OpenAI crawler guide
GPTBot vs OAI-SearchBot
The important OpenAI choice is not "allow OpenAI or block OpenAI." It is whether you want ChatGPT Search discovery, model training access, user-triggered fetches, or some mix of those.
The short version
Use OAI-SearchBot for ChatGPT Search visibility. Use GPTBot for OpenAI model training access. Treat ChatGPT-User as user-triggered access, not automatic search discovery.
What each OpenAI user agent means
| User agent | Main use | What to decide |
|---|---|---|
OAI-SearchBot |
Search crawling for ChatGPT search features. | Allow it if you want eligible pages to appear in ChatGPT Search answers. |
GPTBot |
Crawling content that may be used for training OpenAI foundation models. | Allow or block it based on your training-data policy. |
ChatGPT-User |
User-triggered visits when a ChatGPT user asks for or acts on web content. | Do not use it as your ChatGPT Search opt-out switch. |
A balanced robots.txt pattern
This pattern keeps ChatGPT Search discovery open while signaling that GPTBot should not use the site for model training. Review it against your content rights and business goals before publishing.
User-agent: OAI-SearchBot Allow: / User-agent: GPTBot Disallow: /
How CrawlerSignal uses this
CrawlerSignal checks whether your public robots.txt separates search, training, and user-triggered crawler intent. A medium score often means the site has not made this choice explicit yet.