Sphinn Home » SEO
A smart robots.txt logs crawler requests, maintains the crawler IP list, serves gibberish to nosy users, helps identifying new crawlers, is lean and easy to validate because it comes with crawler directives for the requesting search engine only.

Here is my step by step "How to create a smart robots.txt" guide. Adapt. Test. Enjoy.
8 Comments     

Comments

from floppy 587 days ago #
Votes: 0 | Vote:
+ -

That looks very interesting.  Thanks for the share.

from g1smd 587 days ago #
Votes: 0 | Vote:
+ -

Hot topic in two hours methinks.  

This needs to be used with care. There are several ways to catch yourself out if bots catch different versions of the file from the same site at different times.

Use with care, and test, test, test and test again.

from Sebastian 587 days ago #
Votes: 0 | Vote:
+ -

Well, too frequent changes of robots.txt can irritate search engines regardless whether the robots.txt is cloaked or not. I second your "Use with care, and test, test, test and test again.".

from ronwicker 587 days ago #
Votes: 0 | Vote:
+ -

Great tutorial.  You'd be a great, and popular, teacher.  Rough around the edges...but extremely knowledgeable.  Like the Howard Stern of Search.

from Sebastian 587 days ago #
Votes: 0 | Vote:
+ -

Ron, thank you, but  Howard Stern is in another league.

from rotatedspectrum 587 days ago #
Votes: 0 | Vote:
+ -

You don't even know how cool this is.  I'm in the midst of writing (yes, this is off topic) an SEO friendly CMS and was wondering how to dynamically rewrite URLs using info from the database. 

I had know idea this could be done.  Well, I had some idea because I'm a vbseo user, but didn't realize it was this 'easy.'...or obvious i mean.

from caravan 585 days ago #
Votes: 0 | Vote:
+ -

Interesting article there. It has given me plenty to think about and consider. Cheers.

from UtahSEOpro 581 days ago #
Votes: 0 | Vote:
+ -

Rough around the edges isn't what makes him unique. How many other SEO guys have a thorough database/development background and include such strategies and tutorials in their postings? Not many. He has carved his identity as such.


Log in to comment or register here.

Sphinn Sponsors

Be a Sphinn Sponsor - Click Here

Search Marketing Expo

Save the date for:
SMX Singapore - July 2-3, 2009
SMX São Paulo - August 4-5
SMX East - October 5-7, 2009
SMX Stockholm - 12-13 October, 2009
SMX Mexico - November 11, 2009

Search Marketing Now

Learn more about search marketing through free online webcasts and webinars from our sister site Search Marketing Now.

Upcoming Webcasts: