News Source: http://sebastians-pamphlets.com
Published: Nov 20, 2007 - 07:17 pm
Story Found By: Sebastian 1543 Days ago
Category: SEO
I don’t know for sure which experimental crawler directives Google has implemented yet, but for example a line like
Noindex: /
in your robots.txt will now deindex your complete Web site.
Better check your robots.txt and make sure it doesn’t contain crawler directives belonging to robots meta tags respectively X-Robots-Tags.
8 Comments
Comments
You have great, thought out posts and i’m a fan of the swearing and brutal honesty.
Thanks for the compliment, Ron.
Great article. For this entry:Noindex: /repstuff/noindex.phpDid this URL get indexed: example.com/repstuff/ at all?I would have picked a non-index-page URL for the test just in case Google indexed the bare URL without filename anyway.
Thanks. :) Google can’t index example.com/repstuff/ because there’s no default document and direcory browsing is forbidden.
OK. I follow that case.
Hey Seb - great post. Did you see it got blogged by Dave at WPN? Here http://www.webpronews.com/insiderreports/2007/11/21/unvalidated-robots-txt-risks-google-banishment
Thanks :)
Here are the first <a href="http://sebastians-pamphlets.com/validate-your-robots-txt-or-google-might-deindex-your-site/#robots-txt-test-results-2007-11-28">test results</a>.It seems Google indeed treats Noindex: in robots.txt as Disallow:, if that is so that’s a bad move. I hope they’ll do the right thing eventually. Noindex: shouldn’t block crawling, because it implies Follow: