Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Here’s a free script for analyzing spider behavior (you’ll need some entry-level command line skills to run it). It’s mighty handy for log analysis - do things like list all 302s, 404s, 500s, or any other status code segmented by user agent.... find out how Slurp crawls your site chronologically, or if MSN behaves with 301s. Great for really big files, fast and efficient.
Comments6 Comments  

Comments

Avatar
from trainsem 2290 Days ago #
Votes: 0

Always nice to see people sharing useful scripts. Thank you.

Avatar
from onreact 2289 Days ago #
Votes: 0

Advanced SEO for nerds.

Avatar
from g1smd 2289 Days ago #
Votes: 1

There are several more IP blocks that Google uses for spidering.     <div></div><div></div><div>I found these new IP blocks when looking for the IP addresses being used by all that new datacentre hardware (all USD 3 Billion of it) that has come online in the last couple of years.    A quick Google search easily finds the post.  The new datacentres serve search results, but also spider the web.</div><div>   </div>

Avatar
from AdamAudette 2289 Days ago #
Votes: 0

@q1smd do you have the IP range? thanks for the tip

Avatar
from g1smd 2285 Days ago #
Votes: 0

Check my posts [datacentre lists] from June 1st for the list.

Avatar
from SenHu 2122 Days ago #
Votes: 0

I use biterscripting. I believe it is available for all windows versions. I believe it is a free download. There are couple of sample scripts available on their website. I like the one script at http://www.biterscripting.com/Download/SS_WebLogParser.txt . Sen

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.



Join us at an upcoming SMX event: