How to increase Google Search Engine crawl rate using Google Webmaster Tools


Are you a webmaster tired of waiting for Google Search Engine to index your web pages? Here are some tips to configure the Google crawl rate for your websites using Google Webmaster Tools.

I was working on shifting this website from the old domain "MyWindowsClub" to the new domain "Techulator.com". I followed all general webmaster guidelines and successfully setup 301 redirect in Google Webmaster Tools as well as in the website. All traffic to the old site started redirecting to the new domain with virtually no lose in traffic.

Even though I was not losing any significant traffic to the new domain, I experienced multiple issues in the domain name change related SEO. I would cover most of them in another post. In this post, I will discuss a specific issue related to crawl rate and indexing by Search Engines.



Google crawl rate for new sites is slower than rate for established sites


After we moved to the new domain, Google was pretty nice to us. It was indexing all new pages within 30 minutes or so. However, it was very slow in indexing all existing articles and other pages. The old domain had about 60,000 pages indexed by Google. However, even 4 days after we setup 301 redirect, Google had indexed only less than 500 pages. This triggered some red signal to me. At this rate, it could take several months to complete indexing all of our existing pages.

I started looking around for help. During my disparate search for options to increase the crawl rate by Google, I looked at the Google Webmaster Tools. I saw the crawl rate option in Google Webmaster tools which was set to Let Google determine my crawl rate (recommended). Google recommends this option and I do leave it like that in most cases.

However, in this case, I was desperate to get all my old pages indexed by Google as quickly as possible and did not really want to go by the recommendations. Recommendations are not meant to cover all cases but are meant for general cases.

I switched to the option "Set custom crawl rate". Then the tool gave me a slider to choose the crawl rate. The current value was set to 0.5 requests per second, which means it would take about 2 seconds delay between each request.

I moved the slider to the highest value possible, resulting in a crawl rate of 2 requests per second.

Please note that default value will be different for different sites. If your site is popular and Google index it frequently, then the default crawl rate would be high. In case of small sites with low traffic, the crawl rate would be automatically set to 0.001 requests per second.

By increasing the crawl rate in Google Webmaster Tools, you are instructing Google Search Engine spiders to crawl and index your pages as quickly as possible.

Impact of changing Crawl Rate on SEO


I changed the crawl rate 3 days ago through Google Webmaster tools and started seeing significant results within a day. Now the pages are crawled and indexed much faster than before. Even though it could cause an overload on the server, I really don't care about it for time being. My highest priority is, get all pages indexed by Google as quickly as possible.

The crawl rate plays a significant role in Search Engine Optimization. I think this is not really mentioned in many SEO tips that you see everywhere on the web. I have been spending a lot of time last few days figuring out how to increase quality backlinks for the new domain, but did not pay attention to this until recently. I am glad I used this option which helped a lot in SEO. (I would reset the value back to default once Google finish indexing all old pages in the new domain.)



How to change crawl rate for Google Search Engine spider


If you want Google Search Engine spiders to crawl your web pages faster than normal, you could request it through Google webmaster tools.

Increase Search Engine Crawl rate

Login to Google Webmaster Tools, select the site for which you want to make the changes and follow the steps below:

1. Click on "Site Configuration" on the navigation on the left.

2. Click on "Settings"

3. Check the option "Set custom crawl rate" near the crawl rate heading.

4. Move the slider to the right to increase the crawl rate. You may increase it to the highest possible value, but remember this could cause an overhead to your server.


Article by Tony John
Tony John is a professional blogger from India, who started his first Weblog in 1998 at Tripod.com. Tony switched to blogging as a passion blended business in the year 2000 and currently operates several popular web properties including IndiaStudyChannel.com, Techulator.com, dotnetspider.com and many more.

Follow Tony John or read 703 articles authored by Tony John

Comments

Guest Author: Anuj03 Aug 2012

I my case, my website is not able to be changed its crawl rate, because the webmasters tool says the following:

your website has been assigned a special crawl rate, it cant be changes

Author: Tony John04 Aug 2012 Member Level: Gold   Points : 0

Anuj,

I haven't seen it the crawl rate changing is disabled for any site. Is there anything special for your site - like too new, too old, too low traffic, too popular etc?

Guest Author: Anuj Sharma04 Aug 2012

Mine is a blogger blog with a custom domain. I have added it to Google Webmasters. As per your steps , when I visit the settings page, the crawl rate setting is disabled for me. Below is a screen of the situation

http://awesomescreenshot.com/0e6cbf8ec

What to do now, my default crawl rate is very slow, i.e. it takes somewhat two days to get my posts indexed

Author: Sachin Rambhau Kukale25 Sep 2012 Member Level: Gold   Points : 0

It is really nice and helpful article. Thank you for sharing such important information with us. Hope this will help to optimize SEO strategies for bloggers and website owners.

Guest Author: Roger P.25 Sep 2012

Very informative on how to increase Google Search Engine crawl rate by using Google Webmaster Tools, I am blogger and I was looking this info from a long time. Finally I got it here, it solved my problem. Thanks

Author: Tony John25 Sep 2012 Member Level: Gold   Points : 0

I think crawl rate is disabled in Google Webmaster toos for new sites. Let me check it and get back to you soon.



  • Do not include your name, "with regards" etc in the comment. Write detailed comment, relevant to the topic.
  • No HTML formatting and links to other web sites are allowed.
  • This is a strictly moderated site. Absolutely no spam allowed.
  • Name:
    Email: