Sphinn - The Top 10 Most Needed Robots.txt Statements or How Google Messes With The REP

55
Sphinn It!

The Top 10 Most Needed Robots.txt Statements or How Google Messes With The REP

Went Hot: January 4, 2008 - 4:59 am
Posted By: Sebastian 547 days ago
Topic Type: News Story (Jump to http://sebastians-pamphlets.com)
Category: SEO

"Dear Googlers, I believe that a dog company like Google needs to take real-world Webmasters into the boat when playing with standards like the Robots Exclusions Protocol (REP), for the sake of the SEO cats."

Seriously, that's kinda RFC for robots.txt enhancements and dull as the dirt you're used to read when bothering with Web standards.

6 Comments

Who Desphunn This (0)

Comments

from Sebastian 547 days ago #

Votes: 0 | Vote:

+ -

Table of contents:

Rel-Nofollow or how Google abused standardization of crawler directives for selfish goals

Google's "Noindex: in robots.txt" experiment

Recap: Existing robots.txt directives

URIs, patterns, ...

User-agent: [name]

Disallow: /path

Allow: /path

Sitemap: [absolute URL]

Recap: Existing REP tags

Problems with REP tags in robots.txt

Priority settings

Noindex: /path

Norank: /path

Nofollow: /path

Noarchive: /path

Nosnippet: /path

Nopreview: /path

Noodp: /path

Noydir: /path

Unavailable_after [date]: /path

Truncate-variable [string|pattern]: /path

Truncate-value [string|pattern]: /path

Order-arguments [charset]: /path

Will all this come true?

from baiduyou 547 days ago #

Votes: 0 | Vote:

+ -

Another quality post. How Google have not yet seen fit to employ you I do not know.

from dannysullivan 547 days ago #

Votes: 0 | Vote:

+ -

To be fair on nocomment, Google did that in particular after Dave Winer ragged on them for years that they needed to do something to solve the blog spam problem that blog platforms themselves created by having open commenting systems. So Google finally got sick of it, rolled out nofollow, and PR problem solved. Spam didn't go away, as we all knew it wouldn't, but at least they stopped getting yelled at over it.

from g1smd 547 days ago #

Votes: 0 | Vote:

+ -

I don't agree with every last detail, but this seriously needs some wider coverage to get a discussion going towards some better standards.

You ought to find a sponsor that could take this through as a draft RFC. I don't believe that individual netizens can just submit an RFC on their own.

from Sebastian 547 days ago #

Votes: 0 | Vote:

+ -

Baiduyou, thanks. :) Maybe I lost their email in an ocean of spammy job offers from far east companies and alike that I filter right to /dev/null. ;) Not.

Danny, that's true, and flies as sole reason in the PR dept., but Matt is a very smart guy who wished for ages to have such an instrument. Of course back in 2004/5 they thought of paid links (too) as another usage for rel-nofollow. However, it's a plausible theory, isn't it? If I'm dead wrong, I'll apologize.

G1smd, I guess that even when I find an RFC sponsor today, the whole process takes years and eventually gets buried. I believe that the major engines --and even Google alone-- can implement that better, and way faster, *if* they're interested. Actually, they owe that to the Webmaster community, because their ever changing policies burned billions of man hours for edits and updates forced by their guidelines.