How to increase Google Search Engine crawl rate using Google Webmaster Tools
Are you a webmaster tired of waiting for Google Search Engine to index your web pages? Here are some tips to configure the Google crawl rate for your websites using Google Webmaster Tools.
I was working on shifting this website from the old domain "MyWindowsClub" to the new domain "Techulator.com". I followed all general webmaster guidelines and successfully setup 301 redirect in Google Webmaster Tools as well as in the website. All traffic to the old site started redirecting to the new domain with virtually no lose in traffic.
Even though I was not losing any significant traffic to the new domain, I experienced multiple issues in the domain name change related SEO. I would cover most of them in another post. In this post, I will discuss a specific issue related to crawl rate and indexing by Search Engines.Google crawl rate for new sites is slower than rate for established sites
After we moved to the new domain, Google was pretty nice to us. It was indexing all new pages within 30 minutes or so. However, it was very slow in indexing all existing articles and other pages. The old domain had about 60,000 pages indexed by Google. However, even 4 days after we setup 301 redirect, Google had indexed only less than 500 pages. This triggered some red signal to me. At this rate, it could take several months to complete indexing all of our existing pages.
I started looking around for help. During my disparate search for options to increase the crawl rate by Google, I looked at the Google Webmaster Tools. I saw the crawl rate option in Google Webmaster tools which was set to Let Google determine my crawl rate (recommended). Google recommends this option and I do leave it like that in most cases.
However, in this case, I was desperate to get all my old pages indexed by Google as quickly as possible and did not really want to go by the recommendations. Recommendations are not meant to cover all cases but are meant for general cases.
I switched to the option "Set custom crawl rate". Then the tool gave me a slider to choose the crawl rate. The current value was set to 0.5 requests per second, which means it would take about 2 seconds delay between each request.
I moved the slider to the highest value possible, resulting in a crawl rate of 2 requests per second.
Please note that default value will be different for different sites. If your site is popular and Google index it frequently, then the default crawl rate would be high. In case of small sites with low traffic, the crawl rate would be automatically set to 0.001 requests per second.
By increasing the crawl rate in Google Webmaster Tools, you are instructing Google Search Engine spiders to crawl and index your pages as quickly as possible.Impact of changing Crawl Rate on SEO
I changed the crawl rate 3 days ago through Google Webmaster tools and started seeing significant results within a day. Now the pages are crawled and indexed much faster than before. Even though it could cause an overload on the server, I really don't care about it for time being. My highest priority is, get all pages indexed by Google as quickly as possible.
The crawl rate plays a significant role in Search Engine Optimization. I think this is not really mentioned in many SEO tips that you see everywhere on the web. I have been spending a lot of time last few days figuring out how to increase quality backlinks for the new domain, but did not pay attention to this until recently. I am glad I used this option which helped a lot in SEO. (I would reset the value back to default once Google finish indexing all old pages in the new domain.)
How to change crawl rate for Google Search Engine spider
If you want Google Search Engine spiders to crawl your web pages faster than normal, you could request it through Google webmaster tools.
Login to Google Webmaster Tools, select the site for which you want to make the changes and follow the steps below:
1. Click on "Site Configuration" on the navigation on the left.
2. Click on "Settings"
3. Check the option "Set custom crawl rate" near the crawl rate heading.
4. Move the slider to the right to increase the crawl rate. You may increase it to the highest possible value, but remember this could cause an overhead to your server.
I my case, my website is not able to be changed its crawl rate, because the webmasters tool says the following:
your website has been assigned a special crawl rate, it cant be changes