Default 10 Ways to Increase Your Site Crawl Rate

usedcarforsale · #1 (**permalink**) 09-10-2010, 05:51 AM

Regular and frequent visits by the crawler is the first sign that your site appeals to Google. Thus the most efficient way to get frequent and deep crawls is to develop a website that search engines see as important and valuable.

Note that you can’t force Googlebot to visit you more often – what you can do is to invite it to come. Possible measures to take to increase the crawl rate may include:

- Update your content often and regularly (and ping Google once you do) – well, an obvious one, so not much to describe here; in a word, try to add new unique content as often as you can afford and do it regularly (3 times a week can be the best solution if you can’t update your site daily and are looking for the optimal update rate).
- Make sure your server works correctly: mind the uptime and Google Webmaster tools reports of the unreached pages. Two tools I can recommend here are Pingdom and Mon.itor.us.
- Mind your page load time: note that the crawl works on a budget – if it spends too much time crawling your huge images or PDFs, there will be no time left to visit your other pages.
- Check the site internal link structure: make sure there is no duplicate content returned via different URLs: again, the more time the crawler spends figuring your duplicate content, the fewer useful and unique pages it will manage to visit.
- Get more back links from regularly crawled sites.
- Adjust the crawl speed via Google Webmaster tools.
- Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).
- Make sure your server returns the correct header response. Does it handle your error pages properly? Don’t make the bot figure out what has happened: explain it clearly.
- Make sure you have unique title and meta tags for each of your pages.
- Monitor Google crawl rate for your site and see what works and what not:

Source: searchenginejournal

Marcus Right · #2 (**permalink**) 09-11-2010, 02:05 PM

Every time when Im posting new content on my blog Im always writing a short message on the Twitter about it and I leave the link to that post in the message. Thanks to that after a 10 to 20 min my blog post is indexed.

Canonical · #3 (**permalink**) 09-15-2010, 02:36 AM

The best ways to increase crawl rate are:

1) Increase the number of inbound links and
2) Add new pages to your site frequently

I disagree with several of "Anns" suggestions. Many of them have nothing to do with influencing crawl rates or they are just bad suggestions.

For example:

Quote:

Adjust the crawl speed via Google Webmaster tools.

While this "can" affect crawl rate, adjusting the crawl speed via Google's WMT is an absolute terrible idea. This was added to WMT not to allow you to speed up your crawl rate, but to slow it down. It should ONLY be used for sites whose server cannot handle the load of Googlebot requesting pages (i.e. sites hosted on dinosaur servers). You should never set the crawl rate in WMT in hopes of speeding up the crawling. Nor should you use meta elements to attempt to set the crawl rates.

Quote:

Add a sitemap (though it’s up for a debate whether the sitemap can help with crawling and indexing issues, many webmasters report they have seen increased crawl rate after adding it).

Sitemap.xml files do not make Googlebot visit your site more often. It can get a site with zero back links indexed, which is pretty much useless since without links it will not likely rank for squat and could be just as easily indexed a few days later.

There is no "debate". The real value of sitemap.xml files is twofold:

1) It can be used to help Google and other engines find URLs that might not otherwise be found due to crawlability issues and
2) It can be used by sites with hundreds of thousands or millions of URLs to prioritize which pages are most important, second in importance, third in importance, etc. using the <priority> element.

Other than those two reasons, sitemaps do nothing. They are used by Google to "assist" them with the crawl. They don't make them crawl more frequently. They don't make them index the pages...

99% of the sites out there that use them are just wasting their time building, maintaining, and constantly submitting sitemap.xmls that could be better spent building links, writing quality content, optimizing the site, etc.

Quote:

Make sure you have unique title and meta tags for each of your pages

Give me a break... has absolutely NO influence on crawl rate. Rankings (well title element anyway)? yes But crawl rate? no way.