Google crawler guide
Google-Extended is not Googlebot
Google-Extended is a robots.txt product token for Gemini-related use. It is not the same thing as Googlebot, and blocking it should not be described as blocking Google Search.
The short version
Googlebot controls Google Search crawling. Google-Extended controls whether content Google crawls may be used for Gemini model training and grounding use cases. Google says Google-Extended does not affect Search inclusion or ranking.
What to separate
| Token | Controls | Common mistake |
|---|---|---|
Googlebot |
Google Search crawling and Search features. | Blocking it when you only meant to control AI model use. |
Google-Extended |
Gemini model training and grounding use cases described by Google. | Assuming it is a crawler user-agent string visible in every HTTP request. |
A common protect pattern
If you want Google Search visibility but do not want Google-Extended use, keep Googlebot open and set a separate Google-Extended rule.
User-agent: Googlebot Allow: / User-agent: Google-Extended Disallow: /
How CrawlerSignal uses this
CrawlerSignal flags confusing policies where Google Search crawling and Google-Extended controls are mixed together. The goal is not to tell every site to block Google-Extended, but to make the choice explicit.