Google has completely reorganized its crawlers and user-triggered fetchers documentation. It used to be all on one page and now it is in several pages. Most of the changes were just moving content around but Google did add sections for what product each crawler affects, and added a robots.txt snippet for each crawler to demonstrate how to use the user agent tokens.
Google wrote, “The documentation grew very long which limited our ability to extend the content about our crawlers and user-triggered fetchers,” so that is why they redid it.
If you dig into each crawl, for example Googlebot you will see two new sections:
(1) Affected products
(2) Example robots.txt group
Here is a sample screenshot but what I highlighted in red was added for every crawler:
Here is how each crawler affects products:
- Googlebot: Crawling preferences addressed to the Googlebot user agent affect Google Search (including Discover and all Google Search features), as well as other products such as Google Images, Google Video, Google News, and Discover.
