Back in mid-June, I noticed that Google was not showing many of my images in Google Search and Discover and also some readers were pointing it out to me. So I used the handy Google Search Console URL Inspection tool to find out those S3 URLs I was using to host my images were blocking Googlebot from crawling. Here is a bit of a case study from yours truly of an indexing/crawling issue I had for my image URLs.
This AWS bug led to an 83% drop in the impressions my images were getting from Google Search and Google Images. It led to a 76% drop in image search related clicks to this site. I am still down several weeks later by about 16% in impressions and 26% in clicks from image search but it is a huge improvement.
Here is the Google Search Console Search Performance report showing the impressions and clicks chart over time. You will see the drop around June 15th, then it start to pick back up around July 8th. You will also see that my image traffic has still not fully returned to its normal numbers pre-AWS bug, even after two months:
When Googlebot was trying to access my image URLs on S3, Google was getting a 404 not found error. But when I visited the URLs with my computer, they loaded just fine. These are the same image URLs I have been using on this site for well over a decade and poof, one day, AWS decided to block Googlebot. I reached out to both Google and AWS about the issue and I suspect it was a pretty big issue. Tons of sites use S3 for image and file storage, so Googlebot was likely getting tons of 404 errors. The weird part is that I saw zero public complaints about the issue.
In any event, this is what Googlebot saw when they tried to crawl those URLs:
AWS fixed it after several days:
