Sphinn - Q&A: An undocumented robots.txt crawler directive from Google

35
Sphinn It!

Q&A: An undocumented robots.txt crawler directive from Google

Went Hot: November 14, 2007 - 7:05 pm
Posted By: Sebastian 599 days ago
Topic Type: News Story (Jump to http://sebastians-pamphlets.com)
Category: Google SEO

Update: Googlebot does understand the undocumented Noindex: directive in robots.txt, but Google said it's not (yet) set in stone so be careful when you play with it.

***

If you ever need a hot tip on how to waste an hour or two with quite funny but completely useless SEO research, read my divulgement of a new and somewhat mysterious robots.txt statement invented by Google:
Noindex: /

3 Comments

Who Desphunn This (0)

Comments

from igorthetroll 599 days ago #

Votes: 0 | Vote:

+ -

Bring anti sea-sickness pills, gets dizzy when you read it.

Fun stuff.

from g1smd 594 days ago #

Votes: 0 | Vote:

+ -

Is it just a part of the Robustness Principle in operation at Google?

Maybe they are supporting syntax that they see a lot of people using in error?

from Sebastian 593 days ago #

Votes: 1 | Vote:

+ -

Google has confirmed that their crawler understands the Noindex: directive in robots.txt, but it's an experiment so be careful when you play with it:
http://sebastians-pamphlets.com/about-noindex-crawler-directives-in-robots-txt/#comment-710