Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Sphinn Home » Vertical Search
Robots.txt Study Shows Webmasters Favor Google
Avatar Posted By: garyprice 607 Days ago
Source: http://searchengineland.com
Category: Vertical Search
Barry reports on a new study of Robot.txt files from Penn St. University.
 2 Comments  

Comments

from garyprice 607 Days ago #
Votes: 0

Along with the study, the team at  Penn State have a new released a new database named BotSeer and made it available to the public. "BotSeer is a search engine for robots.txt. Its goal is to provide  information about and access to robots.txt files throughout the web  by crawling and indexing web robots.txt files and related  documents. In addition, statistics about favored robots, comments  and robot behavior is analyzed and presented."

from asnider 606 Days ago #
Votes: 0

In regard to the quote at the end: why would people go out of their way to specifically block MSNBot? MSN Search is hardly the best engine around, but people do use it, and I can’t see a reason you wouldn’t want MSN to index you.

Sphinn Sponsors
Be a Sphinn Sponsor - Click Here
Top 10 Most Sphunn In What's New
  1. Page segmentation and link building (27)
  2. The Google Operating System Is… (25)
  3. Keywords, Probability &… (19)
  4. Interview with Rae Hoffman AKA… (19)
  5. Jordan K’s Definitive Mobile… (19)
  6. 30 Blog Improvement Tips to… (19)
  7. Happy Birthday Sphinn (18)
  8. What do SEOs need to know about… (18)
  9. SEO Company Caught By Client… (17)
  10. Building Traction with Search and… (16)
Latest Posts In Sphinn Blog
Latest Comments
RSS Feeds
Upcoming Conferences

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

Upcoming Webcasts

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »