AdSense Spider is a Googlebot
I'm noticing a few blog posts recently about Google's AdSense spider. Notable posts include AdSense Spider Cross-Pollinates for Google and Matt Cutts confirms AdSense media bot in natural search index. What's strange is that this bot has visited my firm's site even though we don't use AdSense. These aren't recent visits either. Parsing our raw web server log file from last month:
$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1,$4,$7}'
66.249.72.141 [12/Mar/2006:20:36:53 /tools/keyword_tool.php
66.249.66.35 [23/Mar/2006:20:05:02 /tagman/
66.249.66.35 [24/Mar/2006:06:27:39 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:40 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:48 /tagman/FAQ.html
66.249.66.35 [24/Mar/2006:06:27:56 /tagman/flickr-bitty.html
Looking at the IP addresses, these are definitely Googlebots:
$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1}' |sort -u |nslookup |grep ^Name:
Name: crawl-66-249-66-35.googlebot.com
Name: crawl-66-249-72-141.googlebot.com
So, what's a Google AdSense bot (Media Bot, MediaBot, Mediapartners or whatever it's called) doing crawling a site that doesn't use Google AdSense? Is it crawling from a site that uses AdSense?
Also, I wonder if it'd be worth adding this new Googlebot to my firm's free, open source spider tracking tool.
Technorati tags: google adsense, googlebot, matt cutts, mediabot, mediapartners bot, bots, track spiders
$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1,$4,$7}'
66.249.72.141 [12/Mar/2006:20:36:53 /tools/keyword_tool.php
66.249.66.35 [23/Mar/2006:20:05:02 /tagman/
66.249.66.35 [24/Mar/2006:06:27:39 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:40 /tagman/tagblog.html
66.249.66.35 [24/Mar/2006:06:27:48 /tagman/FAQ.html
66.249.66.35 [24/Mar/2006:06:27:56 /tagman/flickr-bitty.html
Looking at the IP addresses, these are definitely Googlebots:
$ grep Mediapartners-Google apogee-web-consulting.com-Mar-2006 |awk '{print $1}' |sort -u |nslookup |grep ^Name:
Name: crawl-66-249-66-35.googlebot.com
Name: crawl-66-249-72-141.googlebot.com
So, what's a Google AdSense bot (Media Bot, MediaBot, Mediapartners or whatever it's called) doing crawling a site that doesn't use Google AdSense? Is it crawling from a site that uses AdSense?
Also, I wonder if it'd be worth adding this new Googlebot to my firm's free, open source spider tracking tool.
Technorati tags: google adsense, googlebot, matt cutts, mediabot, mediapartners bot, bots, track spiders





1 Comments:
HEEL LIFTS
NIPPLE COVERS
HEEL LIFT
ADJUSTABLE HEEL LIFTS
HEIGHT INCREASE INSOLES
HEEL PADS
HEEL PADS
A HEEL LIFT
SHOES INSOLES
SHOE INSOLES
BRAS
NIPPLE COVERS
NIPPLE PASTIES
SOLE PROTECTORS
ARCH SUPPORTS
SHOE STRAPS
HEEL GRIPS
3 X SHOES INSOLES
2 X SHOES INSOLES
ARCH SUPPORT
2 X SHOE STRAPS
STICK ON SOLE PADS
GLITTER NIPPLE COVERS
SILICONE NIPPLE COVERS
WHOLESALE NIPPLE COVERS
TOE SPREADERS
LIFTS SHOE
SHOE INSERTS
SHOE LIFTS
FABRIC NIPPLE COVERS
Post a Comment
<< Home