Recently Google’s John Mueller commented on twitter that Google never supports crawl delay in robot.txt file. But this does not mean that user can’t slow down the crawling process of Google. On the other hand, Bing’s Frédéric Dubut replies in tweet by saying Bingbot supports crawl delay. 

Recommended: Google Introduce Interpreter Mode Language Translator To Assistant

Tweet By John Mueller

Tweet By Frédéric Dubut

The crawl settings are still present in the Google Search Console. The user can set the rate of Google’s crawling rate under search console.

 

Leave a Reply

EXPLORE THE LATEST

EXPLORE THE LATEST

That answer every ifs and buts of digital marketing

Stay ahead with deeper understanding of the trends and up-to-date marketing tactics; all tried & tested in the ever-growing digital space.

Looking to
Achieve your Goals?

Let us help you
get there!

We're a digital agency focused on
creative and results-driven solutions.

Start With
Submit Form Below