Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

Sphinn Home » SEO
MSNbot 2.0b is ignoring robots.txt and No Index meta tags
SEO News Source: http://www.chewie.co.uk
Published: Apr 16, 2009 - 09:42 am
Story Found By: janecopland 1131 Days ago
Category: SEO
Microsoft appears to be having a spot of bother with robots.txt files. That is, it’s messing them up completely. Dean Chew writes about how MSNbot 2.0b’s been in ur server, crawling ur files.
 Comments4 Comments  

Comments

Avatar
from Chewie 1131 Days ago #
Votes: 0

Thanks for the submit Jane, this could cause some very really problems for websites who are obviously not wanting area’s of their site to appear in the SERPS.

Avatar
from sza 1131 Days ago #
Votes: 0

If it’s just a case of wrong DNS lookup, how does it explain the example where MSNbot specifically requested a directory (/forbidden) that wasn’t linked to at all? Another domain’s robots.txt told them about it?

Avatar
from jaamit 1131 Days ago #
Votes: 0

wow. msn really need to give up the pretence of being in the search engine game - i’m all for google having some competition but lets face it with their poor results, lack of true innovation and slow lumbery microsoftiness, MSN/Live ain’t gonna do anything but continue dwindling as a search engine...

Avatar
from lorenbaker 1131 Days ago #
Votes: 0

I’ve been experiencing problems with MSN and Robots.txt for over a year. Especially with them indexing and serving affiliate or ad tracking URLs in the SERPS. Usually if you contact them, they can work on it with you, but this is an issue which really needs to be taken care of.

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.


SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

Latest News From Marketing Land
Upcoming Webcasts

Search Marketing Now Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:


Latest News From Search Engine Land