Sphinn Home » Google SEO
Forgotten –and, until recently, ignored– not standardized statements in your robots.txt might change Googlebot's behavior all of a sudden, without notice.

I don't know for sure which experimental crawler directives Google has implemented yet, but for example a line like
Noindex: /
in your robots.txt will now deindex your complete Web site.

Better check your robots.txt and make sure it doesn't contain crawler directives belonging to robots meta tags respectively X-Robots-Tags.
8 Comments     

Comments

from ronwicker 326 days ago #
Votes: 2 | Vote:
+ -

You have great, thought out posts and i'm a fan of the swearing and brutal honesty.


from Sebastian 326 days ago #
Votes: 0 | Vote:
+ -

Thanks for the compliment, Ron.

from g1smd 326 days ago #
Votes: 0 | Vote:
+ -

Great article.  For this entry:

Noindex: /repstuff/noindex.php

Did this URL get indexed:   example.com/repstuff/ at all?

I would have picked a non-index-page URL for the test just in case Google indexed the bare URL without  filename anyway.

from Sebastian 326 days ago #
Votes: 0 | Vote:
+ -

Thanks. :)
Google can't index example.com/repstuff/ because there's no default document and direcory browsing is forbidden.

from g1smd 325 days ago #
Votes: 0 | Vote:
+ -

OK. I follow that case.

from Kalena 323 days ago #
Votes: 0 | Vote:
+ -

Hey Seb - great post. Did you see it got blogged by Dave at WPN? Here http://www.webpronews.com/insiderreports/2007/11/21/unvalidated-robots-txt-risks-google-banishment

from Sebastian 322 days ago #
Votes: 0 | Vote:
+ -

Thanks :)

from Sebastian 318 days ago #
Votes: 0 | Vote:
+ -

Here are the first test results.

It seems Google indeed treats Noindex: in robots.txt as Disallow:, if that is so that's a bad move. I hope they'll do the right thing eventually. Noindex: shouldn't block crawling, because it implies Follow:


Log in to comment or register here.
Search Marketing Expo

Save the date for:
SMX London - Nov. 4-5: Pre-agenda rate now available. Click here.
SMX West - Feb. 10-12

Search Marketing Now

Learn more about search marketing through free online webcasts and webinars from our sister site Search Marketing Now.

Upcoming Webcasts: