Google addresses the subject of robots.txt files and whether it’s a good SEO practice to keep them within a reasonable size.
This topic is discussed by Google’s Search Advocate John Mueller during the Google Search Central SEO office-hours hangout recorded on January 14.
David Zieger, an SEO manager for a large news publisher in Germany, joins the livestream with concerns about a “huge” and “complex” robots.txt file.
How huge are we talking here?
Zieger says there’s over 1,500 lines with a “multitude” of disallows that keeps growing over the years.
The disallows prevent Google from indexing HTML fragments and URLs where AJAX calls are used.
Zieger says it’s not possible to set a noindex, which is another way to keep the fragments and URLs out of Google’s index, so he’s resorted to filling the site’s robots.txt with disallows.
Are there any negative SEO effects that can result from a huge robots.txt file?
Here’s what Mueller says.