preCharge Forums - View Single Post

07-23-2006

On paper, the answer is that you can use robots.txt to diallow Googlebot while allowing the Adsense bot.
http://www.google.com/support/webmas...366&topic=8846

However,

There's recently been a change where all the various google bots feed a common cache, to reduce the need for multiple spiderings of the same page. So it really depends on your goal for keeping out the bots. If all you want to do is stop caching, there's a meta tag for that.

see also
http://www.mattcutts.com/blog/crawl-caching-proxy/
http://www.mattcutts.com/blog/bot-ob...ing-googlebot/

07-23-2006	#2 (permalink)
Rob Posts: n/a	Re: w/o access from SE On paper, the answer is that you can use robots.txt to diallow Googlebot while allowing the Adsense bot. http://www.google.com/support/webmas...366&topic=8846 However, There's recently been a change where all the various google bots feed a common cache, to reduce the need for multiple spiderings of the same page. So it really depends on your goal for keeping out the bots. If all you want to do is stop caching, there's a meta tag for that. see also http://www.mattcutts.com/blog/crawl-caching-proxy/ http://www.mattcutts.com/blog/bot-ob...ing-googlebot/