Track Search Engine Spiders - Free Web Analytics Software for Apache Web Server

FREE CGI SCRIPTS

	trackSpiders - Info ==> - Download v1.1 - tar file (UNIX) - zip file (Windows) - Demo
	Instructions - Download Script - Edit 1 Line - Upload - Set Permissions - Enjoy!
	Related Pages - SEM Tools - custom reports - track search hits - SEM Articles - About Us

TRACK SEARCH ENGINE SPIDERS

Use trackSpiders.cgi to track spiders from major search engines (Google, Yahoo, MSN). This free web analytics software will examine an Apache web server log and will display visits from Googlebot, Yahoo Slurp, and MSNBot. It takes an optional query string in the format s=ygm where each letter corresponds to the search engine spider for the major search engines. For example, specifying s=g will only display pages crawled by Googlebot.

Output:
Each hit from a search engine spider will be displayed in a row with these columns:

Time (timestamp of the spider visit)
IP Address (remote ip address of the spider)
Spider (name of the spider)
Status (http error code)
Page Crawled (url of the page visited)

If more than one spider visit is found, a bar chart is displayed comparing the number of hits from each search engine spider.

Demo:
trackSpiders.cgi - Show all three search engine spider visits
trackSpiders.cgi?s=gy - Show Googlebot and Yahoo Slurp spider visits
trackSpiders.cgi?s=m - Show only MSNbot spider visits

Instructions:
Download script [UNIX] [Windows] to your machine.
Edit the line that reads:
LOG_FILE="/var/log/httpd/access_log"
so the CGI script will know where your Apache web server log file resides. Upload trackSpiders.cgi to a directory on your web server where you can run CGI scripts. Change permissions to make the file executable (chmod +x or chmod 755). If any of these instructions do not make sense, ask your webmaster or web hosting company to handle this for you. Or, contact us to become a client and we will install this free web analytics software and other custom search marketing reports on your site.

Notes:
This free CGI script only works on web sites using the Apache web server on a UNIX platform. It will parse a web server log that is in Apache combined log format. The script is open source code.

Maine Munchies Ad