Google’s John Mueller states that ” it is really a bad idea which will create all sorts of problems” if you happen to block Google or any other search engine from crawling webpages that return a 404 server status code. He says “every day billions of 404 pages are crawled” by Google. One webmaster David Michaels stated that his “site block user agents automatically who get more than 10 404 errors, that includes the GoogleBot, so that is kind of a problem.” To which John replied that seems like a really bad idea which will create problems.
Here are the Tweets
That sounds like a really bad idea which will cause all sorts of problems.. You can't avoid that Googlebot & all other search engines will run into 404s. Crawling always includes URLs that were previously seen to be 404.
— ? John ? (@JohnMu) July 15, 2020
Recommendation: Google Updates Shopping Ads Policy Against Scamming & Phishing
Billions of 404 pages are crawled every day – it's a normal part of the web, it's the proper way to signal that a URL doesn't exist. That's not something you need to, or can, suppress.
— ? John ? (@JohnMu) July 15, 2020
So automatically blocking Google from having access to 404 pages without even knowing how Google access those pages could be a really bad idea.