Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Sphinn Home » SEM
What is Your robots.txt File Telling Your Competitors About You?
SEM News Source: http://www.jenniferslegg.com
Published: Feb 13, 2008 - 01:55 pm
Story Found By: MattMcGee 1563 Days ago
Category: SEM
Jen Slegg: "One client was able to launch an entire section to their site because they noticed their competitor was doing something very similar when the directory popped up in the competitor’s robots.txt file. And like any good businessperson, my client beat them to the punch and launched it first."
 Comments5 Comments  

Comments

Avatar
from BartGibby 1562 Days ago #
Votes: 0

lol dude this is crazy man. I have always thought that the robots.txt file should actually be located in any location much like the xml sitemap. Just add it to the webmaster account.They always seem so revealing. Your client’s webmaster should know to use sub domains for testing and never to link from a "live" source to the testing area. Bots only know about pages because of the links. Some do some fancy search queries and so forth.

Avatar
from IncrediBILL 1561 Days ago #
Votes: 1

I’ve been warning people about the exposure of robots.txt for years purely from a website security point of view and solving this is not rocket science.The solution is called a DYNAMIC ROBOTS.TXT file and you serve up the REAL robots.txt file only to valid whitelisted bots from major search engines verified by IP range and/or full trip DNS validation.Everyone else sees the following:User-agent: *Disallow: /Which will confuse the hell out of them, which is a good thing if they’re snooping!

Avatar
from amusedbystander 1561 Days ago #
Votes: 0

Thanks Bill. That sure confused me once when I saw it on someone’s website - snooping as I was. Nice to know the reason! Now can we game it ;-)

Avatar
from willneu 1561 Days ago #
Votes: 0

Its quite interesting how major websites cloak this, and some don’t seem to care. A simple, allow the major’s leave the small engines out seems to be a good tactic to me. This is cloaking at its simplest, but one of the few times it won’t be frowned upon by the search engines ;)

Avatar
from g1smd 1560 Days ago #
Votes: 0

@incredibill  If someone links to the robots.txt file from another site, then the robots.txt file will appear in the SERPs, and the Google cache of the file will show you what Google really saw... unless some extra steps are taken by the site owner (most don’t).

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

Latest News From Marketing Land
Upcoming Webcasts

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


Latest News From Search Engine Land