Google announced that it is sunsetting the search console crawl rate limiter tool, scheduled to be removed on January 8, 2024, citing improvements to crawling that has essentially made it unnecessary.
Search Console Crawl Rate Limiter Tool
The crawl rate limiter tool was introduced to search console fifteen years ago in 2008. The purpose of the tool was to provide publishers a way to control Googlebot crawling so that it didn’t overwhelm the server.
There was a time when some publishers experienced too much crawling, which could result in the server being unable to server webpages to users.
Enough people complained that Google eventually released the tool within search console.
The impact of the tool was to provide Google with data. According to Google, requests to limit crawling typically took about a day to go into effect and remained in effect for 90 days.
Why Google Is Removing Rate Limiter Tool
The announcement stated that crawling algorithms have reached a state where Googlebot can automatically sense when a server is reaching capacity and take immediate action to slow down the crawl rate.
Furthermore, Google stated that the tool was rarely used and when it was used, the crawl rate was generally set to the lowest setting.
Moving forward, the minimum crawl rate will by default be set to a lower rate similar to what publishers tended to request.
According to the announcement: