Google’s John Mueller recently shares some suggestions on blocking robot.txt and sitemap files from being indexed in the search results.
Reason For Advice:
Garry Illyes generally noted as any other URL robot.txt files can also be technically indexed. Sometimes it provides special directions as well and there is no method for not being indexed.
Tweet By Garry Illyes:
Triggered by an internal question: robots.txt from indexing point of view is just a url whose content can be indexed. It can become canonical or it can be deduped, just like any other URL.
It only has special meaning for crawling, but there its index status doesn't matter at all. pic.twitter.com/bBMXy1XcRF
— Gary "鯨理／경리" Illyes (@methode) November 6, 2019
Responsive Tweet By John Mueller:
Use the x-robots-tag HTTP header to block indexing of the robots.txt or sitemaps files. Also, if your robots.txt or sitemap file is ranking for normal queries (not site:), that's usually a sign that your site is really bad off and should be improved instead. https://t.co/DpWz6sYanN
— 🍌 John 🍌 (@JohnMu) November 7, 2019