Google-Extended: what it is, what it does with your content, and what blocking it costs
What Google-Extended is
Google-Extended collects pages to train future models. It does not decide whether Google can cite you in an answer today — that is a different crawler with a different token. Blocking Google-Extended is a decision about your content being learned from, not about your visibility.
Operator: Google
robots.txt token: Google-Extended
User-agent on the wire: none — this is a control token, not a crawler
Category: Model training
Google-Extended is the entry people most often get wrong. Blocking it does not remove you from Google's search results, and allowing it does not add you to anything. It governs one thing: whether content Google has already crawled may be used for AI.
What blocking it actually costs
Opts your content out of Gemini training and grounding. It does NOT remove you from Google Search or from AI Overviews, which are served from the regular Googlebot index.
How to block Google-Extended
Add this to your robots.txt:
User-agent: Google-Extended
Disallow: /
The token must appear on its own User-agent: line. A named group replaces the User-agent: * group rather than adding to it, so anything you also want disallowed for this crawler has to be repeated inside its group.
How to allow Google-Extended
User-agent: Google-Extended
Allow: /
An explicit allow is worth writing even when you have no blanket block: it documents the decision, and it survives someone later adding a restrictive User-agent: * rule without thinking about AI crawlers.
How to confirm your rule is actually in place
Because nothing ever arrives carrying Google-Extended, you cannot confirm this rule from your server logs — there is no request to look for. The only way to check is to read robots.txt back the way Google does: confirm the Google-Extended group exists, is spelled exactly right, and is not silently overridden by a more specific group further down the file.
The AI Crawler Access Checker does that for Google-Extended and every other AI token in one pass, and names the exact line that decides each verdict.
Official documentation
Google documents Google-Extended at https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers.
Common questions
- Does Google-Extended actually crawl my site?
No. Google-Extended is a robots.txt control token, not a crawler. No request ever arrives carrying it as a user-agent. Google crawls with its regular crawler and then reads the Google-Extended group to decide whether that content may be used for AI. This is why you will never see Google-Extended in your server logs, and why any tool that lists a user-agent string for it is wrong.
- How do I block Google-Extended?
Add a group to robots.txt naming the token exactly:
User-agent: Google-Extended Disallow: /The group must name
Google-Extendedon its own line. AUser-agent: *group does not combine with a named one — under RFC 9309 the most specific matching group replaces the wildcard entirely, it does not inherit from it.- What does blocking Google-Extended cost me?
Opts your content out of Gemini training and grounding. It does NOT remove you from Google Search or from AI Overviews, which are served from the regular Googlebot index.
- Can I tell a real Google-Extended request from a fake one?
Not reliably. Google does not publish verified IP ranges for Google-Extended, so the user-agent string is the only signal, and anyone can send it. Treat robots.txt as a statement of policy rather than as enforcement.
Is Google-Extended blocked on your site right now?
Reading your own robots.txt only answers half of it — a firewall rule can block Google-Extended while robots.txt says it is welcome. The checker tests both.
Check your site free