Controlling Googlebot
For some webmasters Google crawls too often (and consumes too much bandwidth). For others it visits too infrequently. Some complain that it doesn’t visit their entire site and others get upset when areas that they didn’t want accessible via search engines appear in the Google index.
To a certain extent, it is not possible to attract robots. Google will visit your site often if the site has excellent content that is updated frequently and cited often by other sites. No amount of shouting will make you popular! However, it is certainly possible to deter robots. You can control both the pages that Googlebot crawls and (should you wish) request a reduction in the frequency or depth of each crawl.
No comments yet.
-
Archives
- September 2008 (1)
- July 2008 (2)
- June 2008 (1)
- April 2008 (1)
- March 2008 (4)
- February 2008 (4)
- January 2008 (1)
- October 2007 (8)
- September 2007 (17)
- August 2007 (22)
- July 2007 (15)
- June 2007 (33)
-
Categories
-
RSS
Entries RSS
Comments RSS
Leave a Reply