Sphinn - Validate your robots.txt - Googlebot becomes smarter

42
Sphinn It!

Validate your robots.txt - Googlebot becomes smarter

Went Hot: November 20, 2007 - 7:17 pm
Posted By: Sebastian 592 days ago
Topic Type: News Story (Jump to http://sebastians-pamphlets.com)
Category: Google SEO

Forgotten –and, until recently, ignored– not standardized statements in your robots.txt might change Googlebot's behavior all of a sudden, without notice.

I don't know for sure which experimental crawler directives Google has implemented yet, but for example a line like
Noindex: /
in your robots.txt will now deindex your complete Web site.

Better check your robots.txt and make sure it doesn't contain crawler directives belonging to robots meta tags respectively X-Robots-Tags.

8 Comments

Who Desphunn This (0)

Comments

from ronwicker 592 days ago #

Votes: 2 | Vote:

+ -

You have great, thought out posts and i'm a fan of the swearing and brutal honesty.

from Sebastian 592 days ago #

Votes: 0 | Vote:

+ -

Thanks for the compliment, Ron.

from g1smd 591 days ago #

Votes: 0 | Vote:

+ -

Great article. For this entry:

Noindex: /repstuff/noindex.php

Did this URL get indexed: example.com/repstuff/ at all?

I would have picked a non-index-page URL for the test just in case Google indexed the bare URL without filename anyway.

from Sebastian 591 days ago #

Votes: 0 | Vote:

+ -

Thanks. :)
Google can't index example.com/repstuff/ because there's no default document and direcory browsing is forbidden.

from g1smd 591 days ago #

Votes: 0 | Vote:

+ -

OK. I follow that case.

from Kalena 588 days ago #

Votes: 0 | Vote:

+ -

Hey Seb - great post. Did you see it got blogged by Dave at WPN? Here http://www.webpronews.com/insiderreports/2007/11/21/unvalidated-robots-txt-risks-google-banishment

from Sebastian 588 days ago #

Votes: 0 | Vote:

+ -

Thanks :)

from Sebastian 584 days ago #

Votes: 0 | Vote:

+ -

Here are the first test results.

It seems Google indeed treats Noindex: in robots.txt as Disallow:, if that is so that's a bad move. I hope they'll do the right thing eventually. Noindex: shouldn't block crawling, because it implies Follow: