News Source: http://sebastians-pamphlets.com
Published: Jan 04, 2008 - 04:59 am
Story Found By: Sebastian 1602 Days ago
Category: SEO
Seriously, that’s kinda RFC for robots.txt enhancements and dull as the dirt you’re used to read when bothering with Web standards.
6 Comments
Comments
Table of contents:<ul><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#rel-nofollow-rep-abuse">Rel-Nofollow or how Google abused standardization of crawler directives for selfish goals</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#google-noindex-robots-txt-experiment">Google’s "<em>Noindex:</em> in robots.txt" experiment</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#existing-robots-txt-statements">Recap: Existing robots.txt directives</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-patterns">URIs, patterns, ...</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-user-agent">User-agent: [name]</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-disallow">Disallow: /path</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-allow">Allow: /path</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-sitemap">Sitemap: [absolute URL]</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#existing-rep-tags">Recap: Existing REP tags</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#probs-with-rep-tags-in-robots-txt">Problems with REP tags in robots.txt</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#rep-command-priority">Priority settings</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noindex">Noindex: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-norank">Norank: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nofollow">Nofollow: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noarchive">Noarchive: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nosnippet">Nosnippet: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nopreview">Nopreview: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noodp">Noodp: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noydir">Noydir: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-unavail-after">Unavailable_after [date]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-truncate-variable">Truncate-variable [string|pattern]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-truncate-value">Truncate-value [string|pattern]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-order-arguments">Order-arguments [charset]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#will-my-dreams-come-true">Will all this come true?</a></li></ul>
Another quality post. How Google have not yet seen fit to employ you I do not know.
To be fair on nocomment, Google did that in particular after Dave Winer ragged on them for years that they needed to do something to solve the blog spam problem that blog platforms themselves created by having open commenting systems. So Google finally got sick of it, rolled out nofollow, and PR problem solved. Spam didn’t go away, as we all knew it wouldn’t, but at least they stopped getting yelled at over it.
I don’t agree with every last detail, but this seriously needs some wider coverage to get a discussion going towards some better standards.You ought to find a sponsor that could take this through as a draft RFC. I don’t believe that individual netizens can just submit an RFC on their own.
Baiduyou, thanks. :) Maybe I lost their email in an ocean of spammy job offers from far east companies and alike that I filter right to /dev/null. ;) Not.Danny, that’s true, and flies as sole reason in the PR dept., but Matt is a very smart guy who wished for ages to have such an instrument. Of course back in 2004/5 they thought of paid links (too) as another usage for rel-nofollow. However, it’s a plausible theory, isn’t it? If I’m dead wrong, I’ll apologize.G1smd, I guess that even when I find an RFC sponsor today, the whole process takes years and eventually gets buried. I believe that the major engines --and even Google alone-- can implement that better, and way faster, *if* they’re interested. Actually, they owe that to the Webmaster community, because their ever changing policies burned billions of man hours for edits and updates forced by their guidelines.
I like the idea for "Noodp: path". Rest of the article seems useless for me.