Recently Google’s John Mueller commented on twitter that Google never supports crawl delay in robot.txt file. But this does not mean that user can’t slow down the crawling process of Google. On the other hand, Bing’s Frédéric Dubut replies in tweet by saying Bingbot supports crawl delay. 

Recommended: Google Introduce Interpreter Mode Language Translator To Assistant

Tweet By John Mueller

Tweet By Frédéric Dubut

The crawl settings are still present in the Google Search Console. The user can set the rate of Google’s crawling rate under search console.

 

Leave a Reply

WE ARE TECHNIANS

Latest Articles & video

Good quality, informative articles about the industry

Looking to
Achieve your Goals?

Let us help you
get there!

We're a digital agency focused on
creative and results-driven solutions.

Start With
Submit Form Below