Google States “Don’t Block GoogleBot From Crawling 404s”


Google’s John Mueller states that ” it is really a bad idea which will create all sorts of problems” if you happen to block Google or any other search engine from crawling webpages that return a 404 server status code. He says “every day billions of 404 pages are crawled” by Google. One webmaster David Michaels stated that his “site block user agents automatically who get more than 10 404 errors, that includes the GoogleBot, so that is kind of a problem.” To which John replied that seems like a really bad idea which will create problems.

Here are the Tweets

Recommendation: Google Updates Shopping Ads Policy Against Scamming & Phishing

So automatically blocking Google from having access to 404 pages without even knowing how Google access those pages could be a really bad idea.

Google, David Michaels, John Mueller,

Leave a Reply