Maine Munchies Ad

Thursday, April 20, 2006

AdSense Spider is a Googlebot

I'm noticing a few blog posts recently about Google's AdSense spider. Notable posts include AdSense Spider Cross-Pollinates for Google and Matt Cutts confirms AdSense media bot in natural search index. What's strange is that this bot has visited my firm's site even though we don't use AdSense. These aren't recent visits either. Parsing our raw web server log file from last month:

$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1,$4,$7}'
66.249.72.141 [12/Mar/2006:20:36:53 /tools/keyword_tool.php
66.249.66.35 [23/Mar/2006:20:05:02 /tagman/
66.249.66.35 [24/Mar/2006:06:27:39 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:40 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:48 /tagman/FAQ.html
66.249.66.35 [24/Mar/2006:06:27:56 /tagman/flickr-bitty.html

Looking at the IP addresses, these are definitely Googlebots:

$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1}' |sort -u |nslookup |grep ^Name:
Name: crawl-66-249-66-35.googlebot.com
Name: crawl-66-249-72-141.googlebot.com

So, what's a Google AdSense bot (Media Bot, MediaBot, Mediapartners or whatever it's called) doing crawling a site that doesn't use Google AdSense? Is it crawling from a site that uses AdSense?

Also, I wonder if it'd be worth adding this new Googlebot to my firm's free, open source spider tracking tool.

Technorati tags: , , , , , ,

0 Comments:

Post a Comment

<< Home