Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

"Dear Googlers, I believe that a dog company like Google needs to take real-world Webmasters into the boat when playing with standards like the Robots Exclusions Protocol (REP), for the sake of the SEO cats."

Seriously, that’s kinda RFC for robots.txt enhancements and dull as the dirt you’re used to read when bothering with Web standards.
Comments6 Comments  

Comments

Avatar Moderator
from Sebastian 3632 Days ago #
Votes: 0

Table of contents:<ul><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#rel-nofollow-rep-abuse">Rel-Nofollow or how Google abused standardization of crawler directives for selfish goals</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#google-noindex-robots-txt-experiment">Google’s "<em>Noindex:</em> in robots.txt" experiment</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#existing-robots-txt-statements">Recap: Existing robots.txt directives</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-patterns">URIs, patterns, ...</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-user-agent">User-agent: [name]</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-disallow">Disallow: /path</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-allow">Allow: /path</a></li><li style="padding-left:10px;"><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-sitemap">Sitemap: [absolute URL]</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#existing-rep-tags">Recap: Existing REP tags</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#probs-with-rep-tags-in-robots-txt">Problems with REP tags in robots.txt</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#rep-command-priority">Priority settings</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noindex">Noindex: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-norank">Norank: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nofollow">Nofollow: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noarchive">Noarchive: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nosnippet">Nosnippet: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-nopreview">Nopreview: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noodp">Noodp: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-noydir">Noydir: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-unavail-after">Unavailable_after [date]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-truncate-variable">Truncate-variable [string|pattern]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-truncate-value">Truncate-value [string|pattern]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#robots-txt-order-arguments">Order-arguments [charset]: /path</a></li><li><a href="http://sebastians-pamphlets.com/standardization-of-rep-tags-as-robots-txt-directives/#will-my-dreams-come-true">Will all this come true?</a></li></ul>

Avatar
from baiduyou 3631 Days ago #
Votes: 0

Another quality post. How Google have not yet seen fit to employ you I do not know.

Avatar Administrator
from dannysullivan 3631 Days ago #
Votes: 0

To be fair on nocomment, Google did that in particular after Dave Winer ragged on them for years that they needed to do something to solve the blog spam problem that blog platforms themselves created by having open commenting systems. So Google finally got sick of it, rolled out nofollow, and PR problem solved. Spam didn’t go away, as we all knew it wouldn’t, but at least they stopped getting yelled at over it.

Avatar
from g1smd 3631 Days ago #
Votes: 0

I don’t agree with every last detail, but this seriously needs some wider coverage to get a discussion going towards some better standards.You ought to find a sponsor that could take this through as a draft RFC. I don’t believe that individual netizens can just submit an RFC on their own.

Avatar Moderator
from Sebastian 3631 Days ago #
Votes: 0

Baiduyou, thanks. :)  Maybe I lost their email in an ocean of spammy job offers from far east companies and alike that I filter right to /dev/null. ;) Not.Danny, that’s true, and flies as sole reason in the PR dept., but Matt is a very smart guy who wished for ages to have such an instrument. Of course back in 2004/5 they thought of paid links (too) as another usage for rel-nofollow. However, it’s a plausible theory, isn’t it? If I’m dead wrong, I’ll apologize.G1smd, I guess that even when I find an RFC sponsor today, the whole process takes years and eventually gets buried. I believe that the major engines --and even Google alone-- can implement that better, and way faster, *if* they’re interested. Actually, they owe that to the Webmaster community, because their ever changing policies burned billions of man hours for edits and updates forced by their guidelines.

Avatar
from dzver 3630 Days ago #
Votes: 0

I like the idea for "Noodp: path". Rest of the article seems useless for me.

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.



Join us at an upcoming SMX event: