Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Barry reports on a new study of Robot.txt files from Penn St. University.
Comments2 Comments  

Comments

Avatar
from garyprice 1549 Days ago #
Votes: 0

Along with the study, the team at  Penn State have a new released a new database named BotSeer and made it available to the public. "BotSeer is a search engine for robots.txt. Its goal is to provide  information about and access to robots.txt files throughout the web  by crawling and indexing web robots.txt files and related  documents. In addition, statistics about favored robots, comments  and robot behavior is analyzed and presented."

Avatar
from asnider 1548 Days ago #
Votes: 0

In regard to the quote at the end: why would people go out of their way to specifically block MSNBot? MSN Search is hardly the best engine around, but people do use it, and I can’t see a reason you wouldn’t want MSN to index you.

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.



Join us at an upcoming SMX event:

Upcoming Webcasts

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include: