Google has officially declared the impending sunset of the Search Console Crawl Rate Limiter Tool, set for removal on January 8, 2024. This decision stems from advancements in crawling technology that have rendered its utility obsolete.
The Crawl Rate Limiter Tool was integrated into Search Console fifteen years ago, in 2008. Its primary objective was to offer publishers a mechanism to regulate Googlebot's crawling activities, preventing server overload.
During a certain period, certain publishers encountered excessive crawling, leading to potential server inability to serve webpages to users. The volume of complaints prompted Google to address this issue, culminating in the release of the tool within Search Console.
Upon implementation, the tool's primary function was to furnish Google with pertinent data. As outlined by Google, requests to curtail crawling typically underwent activation within approximately a day, maintaining their efficacy for a span of 90 days.
In the recent announcement, Google emphasized the advancements in crawling algorithms that have enabled Googlebot to autonomously detect instances when a server approaches its capacity limit. This heightened capability empowers Googlebot to promptly adjust and moderate the crawl rate, mitigating potential strain on the server.
Additionally, Google highlighted the infrequent utilization of the tool, noting that when employed, the crawl rate tended to be consistently set at its lowest setting. As a strategic move, Google indicated that going forward, the default minimum crawl rate will align more closely with the rates commonly requested by publishers, ensuring a smoother operational pace for web servers.
Read Google’s official announcement - Upcoming deprecation of Crawl Rate Limiter Tool in Search Console
The removal of this tool streamlines the Search Console interface by eliminating seldom-used features, resulting in a more intuitive user experience. This enhancement aims to bolster the overall usability of Search Console for its users. Publishers encountering ongoing challenges related to Googlebot's crawl rate can opt to utilize the Googlebot report form, an alternative avenue enabling them to submit feedback directly to Google for assistance.
Comments