.Google.com's John Mueller addressed a concern about why Google.com indexes pages that are prohibited from crawling by robots.txt as well as why the it is actually risk-free to overlook the associated Browse Console documents regarding those crawls.Bot Traffic To Query Criterion URLs.The individual asking the question documented that bots were actually producing links to non-existent question specification Links (? q= xyz) to pages along with noindex meta tags that are actually also blocked out in robots.txt. What motivated the inquiry is actually that Google.com is actually crawling the hyperlinks to those webpages, getting blocked through robots.txt (without envisioning a noindex robotics meta tag) after that acquiring turned up in Google Browse Console as "Indexed, though blocked out by robots.txt.".The individual talked to the observing inquiry:." Yet listed below's the big concern: why would certainly Google.com index web pages when they can't even see the content? What's the perk in that?".Google.com's John Mueller confirmed that if they can not creep the page they can't see the noindex meta tag. He additionally produces an exciting reference of the web site: search driver, advising to disregard the outcomes considering that the "typical" customers won't observe those results.He composed:." Yes, you're proper: if our company can not creep the page, our team can not observe the noindex. That claimed, if our company can't creep the webpages, after that there's certainly not a lot for us to index. Therefore while you may observe a few of those webpages along with a targeted site:- query, the normal customer will not find all of them, so I wouldn't fuss over it. Noindex is actually also fine (without robots.txt disallow), it only means the URLs will definitely find yourself being actually crept (and also wind up in the Browse Console document for crawled/not listed-- neither of these standings lead to issues to the remainder of the internet site). The fundamental part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's solution affirms the constraints in operation the Website: hunt advanced search operator for diagnostic main reasons. One of those reasons is actually due to the fact that it's not linked to the routine hunt index, it's a separate factor entirely.Google's John Mueller discussed the internet site search operator in 2021:." The brief response is that a web site: query is certainly not meant to be full, nor utilized for diagnostics reasons.A website question is actually a certain sort of search that confines the results to a specific web site. It is actually generally just the word internet site, a bowel, and afterwards the internet site's domain.This query restricts the outcomes to a details site. It is actually certainly not suggested to be a thorough collection of all the web pages from that website.".2. Noindex tag without using a robots.txt is great for these sort of scenarios where a robot is connecting to non-existent webpages that are actually acquiring found through Googlebot.3. Links along with the noindex tag are going to create a "crawled/not listed" entry in Look Console which those won't have an adverse result on the rest of the website.Go through the question and address on LinkedIn:.Why would Google index web pages when they can not also find the information?Featured Graphic through Shutterstock/Krakenimages. com.