.Google's John Mueller answered a concern about why Google marks pages that are prohibited coming from crawling by robots.txt as well as why the it's safe to ignore the associated Browse Console documents regarding those crawls.Robot Web Traffic To Inquiry Criterion URLs.The individual inquiring the question documented that crawlers were producing hyperlinks to non-existent question guideline Links (? q= xyz) to webpages along with noindex meta tags that are likewise obstructed in robots.txt. What triggered the inquiry is actually that Google.com is creeping the hyperlinks to those web pages, getting obstructed through robots.txt (without seeing a noindex robots meta tag) at that point acquiring turned up in Google Browse Console as "Indexed, though obstructed by robots.txt.".The individual talked to the complying with question:." However listed below is actually the big question: why would Google mark webpages when they can not also view the material? What's the advantage because?".Google.com's John Mueller verified that if they can not crawl the page they can not see the noindex meta tag. He likewise helps make an exciting reference of the site: hunt operator, recommending to dismiss the outcomes since the "normal" consumers will not view those results.He created:." Yes, you are actually proper: if our company can't creep the page, our company can't find the noindex. That mentioned, if our team can't crawl the webpages, at that point there's certainly not a whole lot for our company to mark. Therefore while you might observe a few of those pages along with a targeted internet site:- concern, the ordinary user won't find all of them, so I would not fuss over it. Noindex is actually likewise great (without robots.txt disallow), it merely indicates the Links are going to find yourself being actually crept (and also wind up in the Search Console document for crawled/not indexed-- neither of these statuses lead to issues to the remainder of the site). The vital part is actually that you don't produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer affirms the restrictions being used the Site: search evolved hunt operator for analysis explanations. Some of those causes is given that it's certainly not linked to the normal search mark, it's a different thing completely.Google.com's John Mueller discussed the web site hunt driver in 2021:." The quick answer is that a website: query is actually not meant to become total, neither made use of for diagnostics objectives.A web site inquiry is actually a details sort of hunt that restricts the results to a certain site. It is actually primarily only words web site, a digestive tract, and afterwards the website's domain name.This inquiry restricts the outcomes to a specific web site. It's certainly not suggested to become a detailed assortment of all the webpages from that internet site.".2. Noindex tag without using a robots.txt is great for these type of conditions where a robot is actually connecting to non-existent web pages that are obtaining discovered by Googlebot.3. URLs with the noindex tag are going to produce a "crawled/not recorded" item in Look Console and that those will not have a bad result on the rest of the website.Review the concern and respond to on LinkedIn:.Why would Google mark pages when they can't even observe the information?Included Photo by Shutterstock/Krakenimages. com.