Google’s John Mueller answered a question on Reddit about a commonly used robots meta tag and what would happen if it was missing. Mueller’s answer, while it makes sense and is documented, may still come as a surprise to many publishers and SEOs.
Robots Meta Tag
The HTML meta element communicates metadata. Metadata is machine readable information that a crawler like Googlebot can read.
There are many kinds of meta elements like the meta description element but the Robots Meta Element is different in that it can control the search engine crawlers.
The information communicated by the robots meta tag is called a directive, which means that robot crawlers are obligated to obey the instructions in the robots meta tag.
There are many directions to pass along through the robots meta but the following meta tag is one that is relevant to the question John Mueller answered.
The noindex, no follow meta tag:
<meta name="robots" content="noindex, nofollow">
The above meta tag tells the search engine crawlers to not index the content on the webpage and to not follow any links.
One of the most common meta tags is this one, which commands search engines to index the content and follow all the links:
<meta name="robots" content="index, follow">
While the above meta tag is common, there is a significant amount of misunderstanding about it. There’s a line of reasoning that because Google supports nofollow then it must imply that Google supports the follow directive.