Google’s Gary Illyes and Lizzi Sassman discussed three factors that trigger increased Googlebot crawling. While they downplayed the need for constant crawling, they acknowledged there a ways to encourage Googlebot to revisit a website.
1. Impact of High-Quality Content on Crawling Frequency
One of the things they talked about was the quality of a website. A lot of people suffer from the discovered not indexed issue and that’s sometimes caused by certain SEO practices that people have learned and believe are a good practice. I’ve been doing SEO for 25 years and one thing that’s always stayed the same is that industry defined best practices are generally years behind what Google is doing. Yet, it’s hard to see what’s wrong if a person is convinced that they’re doing everything right.
Gary Illyes shared a reason for an elevated crawl frequency at the 4:42 minute mark, explaining that one of triggers for a high level of crawling is signals of high quality that Google’s algorithms detect.
Gary said it at the 4:42 minute mark:
“…generally if the content of a site is of high quality and it’s helpful and people like it in general, then Googlebot–well, Google–tends to crawl more from that site…”
There’s a lot of nuance to the above statement that’s missing, like what are the signals of high quality and helpfulness that will trigger Google to decide to crawl more frequently?
Well, Google never says. But we can speculate and the following are some of my educated guesses.
We know that there are patents about branded search that count branded searches made by users as implied links. Some people think that “implied links” are brand mentions, but “brand mentions” are absolutely not what the patent talks about.
Then there’s the Navboost patent that’s been around since 2004. Some people equate the Navboost patent with clicks but if you read the actual patent from 2004 you’ll see that it never mentions click through rates (CTR). It talks about user interaction signals. Clicks was a topic of intense research in the early 2000s but if you read the research papers and the patents it’s easy to understand what I mean when it’s not so simple as “monkey clicks the website in the SERPs, Google ranks it higher, monkey gets banana.”