Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Sphinn Home » SEO
MSNbot 2.0b is ignoring robots.txt and No Index meta tags
Avatar Went Hot: Apr 16, 2009 - 09:42 am
Posted By: janecopland 298 Days ago
Source: http://www.chewie.co.uk
Category: SEO
Microsoft appears to be having a spot of bother with robots.txt files. That is, it’s messing them up completely. Dean Chew writes about how MSNbot 2.0b’s been in ur server, crawling ur files.
 Comments4 Comments  

Comments

Avatar
from Chewie 298 Days ago #
Votes: 0

Thanks for the submit Jane, this could cause some very really problems for websites who are obviously not wanting area’s of their site to appear in the SERPS.

Avatar
from sza 298 Days ago #
Votes: 0

If it’s just a case of wrong DNS lookup, how does it explain the example where MSNbot specifically requested a directory (/forbidden) that wasn’t linked to at all? Another domain’s robots.txt told them about it?

Avatar
from jaamit 298 Days ago #
Votes: 0

wow. msn really need to give up the pretence of being in the search engine game - i’m all for google having some competition but lets face it with their poor results, lack of true innovation and slow lumbery microsoftiness, MSN/Live ain’t gonna do anything but continue dwindling as a search engine...

Avatar
from lorenbaker 298 Days ago #
Votes: 0

I’ve been experiencing problems with MSN and Robots.txt for over a year. Especially with them indexing and serving affiliate or ad tracking URLs in the SERPS. Usually if you contact them, they can work on it with you, but this is an issue which really needs to be taken care of.

Sphinn Sponsors
Be a Sphinn Sponsor - Click Here
Private Forum Header
Private Forums
Become A Member!
Top 10 Most Sphunn Latest Submissions
  1. The Ultimate Guide to Keyword… (20)
  2. How much HTML does an SEO need to… (20)
  3. Abort Your Blog -- 5 Signs It's… (20)
  4. 5 Powerful Ways to Use Google… (17)
  5. 7 Awesome Ideas to Use Gmail… (17)
  6. How to Get Banned on Social Media… (15)
  7. SEO standards, accreditation,… (14)
  8. Wordpress SEO: The Only Guide You… (14)
  9. Steal this SEO FAQ: 30+ SEO… (14)
  10. Should SEOs care about Real Time… (12)
Latest Comments
Latest Posts In Sphinn Blog
RSS Feeds
Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

Upcoming Webcasts

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


See more webcast topics »