In a recent LinkedIn post, Google Analyst Gary Illyes raised awareness about two issues plaguing web crawlers: soft 404 and other “crypto” errors.
These seemingly innocuous mistakes can negatively affect SEO efforts.
Soft 404 errors occur when a web server returns a standard “200 OK” HTTP status code for pages that don’t exist or contain error messages. This misleads web crawlers, causing them to waste resources on non-existent or unhelpful content.
Illyes likened the experience to visiting a coffee shop where every item is unavailable despite being listed on the menu. While this scenario might be frustrating for human customers, it poses a more serious problem for web crawlers.
As Illyes explains:
“Crawlers use the status codes to interpret whether a fetch was successful, even if the contents of the page is basically just an error message. They might happily go back to the same page again and again wasting your resources, and if there are many such pages, exponentially more resources.”
The consequences of soft 404 errors extend beyond the inefficient use of crawler resources.
According to Illyes, these pages are unlikely to appear in search results because they are filtered out during indexing.
To combat this issue, Illyes advises serving the appropriate HTTP status code when the server or client encounters an error.
This allows crawlers to understand the situation and allocate their resources more effectively.
Illyes also cautioned against rate-limiting crawlers with messages like “TOO MANY REQUESTS SLOW DOWN,” as crawlers cannot interpret such text-based instructions.
Soft 404 errors can impact a website’s crawlability and indexing.
By addressing these issues, crawlers can focus on fetching and indexing pages with valuable content, potentially improving the site’s visibility in search results.
Eliminating soft 404 errors can also lead to more efficient use of server resources, as crawlers won’t waste bandwidth repeatedly visiting error pages.
To identify and resolve soft 404 errors on your website, consider the following steps:
Proactively addressing soft 404 errors can improve your website’s crawlability, indexing, and SEO.
Featured Image: Julia Tim/Shutterstock
Bing added a new guideline to its Bing Webmaster Guidelines named Prompt Injection. A prompt…
A Google community manager, Alistair D., posted a Reviews troubleshooting and FAQs document in the…
Gary Illyes from Google explained two reasons why soft 404 errors are bad. Soft 404s…
There are reports that some sites hit by the Google September 2023 Helpful Content Update…
Google’s latest Search Off the Record shared a wealth of insights on how Google Search…
It’s been a year of considerable disruptions in digital marketing so far. Right now, the…
This website uses cookies.
Leave a Comment