Sphinn Home » SEO
Avatar Went Hot: July 18, 2008 - 2:58 am
Posted By: AdamAudette 83 days ago
Topic Type: News Story (Jump to http://www.audettemedia.com)
Category: SEO
Here's a free script for analyzing spider behavior (you'll need some entry-level command line skills to run it). It's mighty handy for log analysis - do things like list all 302s, 404s, 500s, or any other status code segmented by user agent.... find out how Slurp crawls your site chronologically, or if MSN behaves with 301s. Great for really big files, fast and efficient.
5 Comments     

Comments

from trainsem 83 days ago #
Votes: 0 | Vote:
+ -

Always nice to see people sharing useful scripts. Thank you.

from onreact 82 days ago #
Votes: 0 | Vote:
+ -

Advanced SEO for nerds.

from g1smd 82 days ago #
Votes: 1 | Vote:
+ -

There are several more IP blocks that Google uses for spidering.     

I found these new IP blocks when looking for the IP addresses being used by all that new datacentre hardware (all USD 3 Billion of it) that has come online in the last couple of years.    A quick Google search easily finds the post.  The new datacentres serve search results, but also spider the web.   

from AdamAudette 82 days ago #
Votes: 0 | Vote:
+ -

@q1smd do you have the IP range? thanks for the tip

from g1smd 78 days ago #
Votes: 0 | Vote:
+ -

Check my posts [datacentre lists] from June 1st for the list.


Log in to comment or register here.

Top 10 Most Sphunn
In What's New

  1. StumbleUpon Without the Toolbar - w... (21)
  2. Google social profiling for online ... (20)
  3. 10 Epic Website Failures (20)
  4. Social Media Metrics Greasemonkey P... (18)
  5. Guess the SEO's Tattoo in 'Whose Ta... (16)
  6. SEO, Small Business Websites & Down... (15)
  7. How To Diagnose Your Site with Goog... (15)
  8. Using Search To Mitigate/Reduce Bus... (15)
  9. 16 Web Based Handy Web Designer Too... (14)
  10. Link Building 101: How to Evaluate ... (14)

Latest Comments

Search Marketing Expo

Save the date for:
• SMX East (New York City) - Oct. 6-8: See the agenda or register today and save!
• SMX London - Nov. 4-5: Pre-agenda rate now available. Click here.
• SMX West - Feb. 10-12

Search Marketing Now

Learn more about search marketing through free online webcasts and webinars from our sister site Search Marketing Now.

Upcoming Webcasts: