Google’s Gary Illyes clarified on the Search Off The Record podcast that Google technically does not follow links. Instead, Google will extract the links, collect them in a database, and then check them later. Of course, most of you know this already and it doesn’t really matter much for SEO to know the difference but hey.
Gary Illyes from Google said at the 25:26 mark in that podcast:
Well, yeah, it’s my pet peeve. On onesie [Google Search Central Site], we keep saying Googlebot is following links, like, no, it’s not following links. It’s collecting links, and then it goes back to those links. It’s not like properly following links. The picture that we are painting is that Googlebot is like hopping from–
Gary then did a bit of a post on this on LinkedIn, explaining more. “You probably heard it before that Googlebot “follows” links. It doesn’t. But it’s a pretty illustrative way to describe what Googlebot does,” he said.
He wrote:
A recent Search Off the Record episode (https://lnkd.in/eG566yve) caused some ruckus because we apparently “leaked” that Googlebot doesn’t just “follow” links it finds in a page it just downloaded. If you ever spent some time analyzing your server’s access logs in the past, say, 15 years, you already knew that that’s not the case. There’s more involved than just blindly making a request to URLs found in a elements; there’s deduplication across protocol variants, there’s prioritization of URLs, there’s coffee or lack of, thereof.
So why “follow” then? As much as I don’t like it, it is a very simple way to explain what Googlebot actually does. There’s value in using simple analogies (similes?), but there’s also a place for going for more indepth explanations. You choose the one that you think will work for the audience you’re talking to at the time.
Here is the embed to listen to it:
