Google’s Gary Illyes posted on LinkedIn with two common examples of when a spike in Googlebot activity, crawling, is a bad thing. The short answer is when Googlebot gets to crawling an infinite section of your site (like calendar pages that goes on forever) and when your site is hacked with a ton of new hacked pages.
This is probably very basic SEO stuff for most of you but there might be a reason he is sharing this now.
Gary wrote, “Don’t get happy prematurely when search engines unexpectedly start to crawl like crazy from your site.” “A sudden increase in crawling can mean good things, sure, but it can also mean something is wrong,” he added.
In these cases, he said, “Treat unexpected sharp increases in crawling as a symptom of an issue until you can prove otherwise.” Then he joked, “Or, you know, maybe I’m just a hardline pessimist.”
Here are two issues that come up way too often when looking at sharp increases in crawling, he wrote:
Infinite Spaces:
The example on infinite spaces that he provided is one of the most common. When “you have a calendar thingie on your site, or an infinitely filterable product listings page. If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time. robots.txt is your friend, use it.”
Hacked Content:
Another common example is hacked content. He said, “if a no-good-doer somehow managed to get access to your server’s file system or your content management system, they might flood your otherwise dandy site with, well, crap. If your site generally has pages that search users find helpful, crawlers will get excited about these new pages for a time and happily crawl them. https://web.dev/hacked has great resources about what to do in these cases. (tangent: yes, this is more cracking than hacking but apparently the internet is fine with the misnomer).”
Forum discussion at LinkedIn.
