In a recent statement on LinkedIn, Google Analyst Gary Illyes shared his mission for the year: to figure out how to crawl the web even less.
This comes on the heels of a Reddit post discussing the perception that Google is crawling less than in previous years.
While Illyes clarifies that Google is crawling roughly the same amount, he emphasizes the need for more intelligent scheduling and a focus on URLs that are more likely to deserve crawling.
Illyes’ statement aligns with the ongoing discussion among SEO professionals about the concept of a “crawl budget,” which assumes that sites must stay within a limited number of pages that search engines can crawl daily to get their pages indexed.
However, Google’s Search Relations team recently debunked this misconception in a podcast, explaining how Google prioritizes crawling based on various factors.
Crawling Prioritization & Search Demand
In a podcast published two weeks ago, Illyes explained how Google decides how much to crawl:
“If search demand goes down, then that also correlates to the crawl limit going down.”
While he didn’t provide a clear definition of “search demand,” it likely refers to search query demand from Google’s perspective. In other words, if there is a decrease in searches for a particular topic, Google may have less reason to crawl websites related to that topic.