Sphinn
What's Sphinn?  |  Join us!  |  Login  | Help | Tools

  • Hot Topics
  • What's New
  • Greatest Hits
  • Comments
  • Submit
  • Calendar
  • Network
  • Feeds
  • Search Engine Land
  • All Categories
  • Google
  • Yahoo
  • Microsoft
  • Search Marketing
  • Social Media
  • Online Marketing
  • Searching
  • Other
  •      :: Sphinn Live ::

Sphinn Home » SEO
  • 54
  • Sphinn It!
The Top 10 Most Needed Robots.txt Statements or How Google Messes With The REP
Avatar Went Hot: January 4, 2008 - 4:59 am
Posted By: Sebastian 139 days ago
Topic Type: News Story (Jump to http://sebastians-pamphlets.com) my network
Category: SEO
"Dear Googlers, I believe that a dog company like Google needs to take real-world Webmasters into the boat when playing with standards like the Robots Exclusions Protocol (REP), for the sake of the SEO cats."

Seriously, that's kinda RFC for robots.txt enhancements and dull as the dirt you're used to read when bothering with Web standards.
6 Comments     

  • Comments
  • Who Sphunn This (54)
  • Who Desphunn This (0)

Comments

from Sebastian 139 days ago #
Votes: 0 | Vote:
+ -

Table of contents:


  • Rel-Nofollow or how Google abused standardization of crawler directives for selfish goals

  • Google's "Noindex: in robots.txt" experiment

  • Recap: Existing robots.txt directives

  • URIs, patterns, ...

  • User-agent: [name]

  • Disallow: /path

  • Allow: /path

  • Sitemap: [absolute URL]

  • Recap: Existing REP tags

  • Problems with REP tags in robots.txt

  • Priority settings

  • Noindex: /path

  • Norank: /path

  • Nofollow: /path

  • Noarchive: /path

  • Nosnippet: /path

  • Nopreview: /path

  • Noodp: /path

  • Noydir: /path

  • Unavailable_after [date]: /path

  • Truncate-variable [string|pattern]: /path

  • Truncate-value [string|pattern]: /path

  • Order-arguments [charset]: /path

  • Will all this come true?



from baiduyou 139 days ago #
Votes: 0 | Vote:
+ -

Another quality post. How Google have not yet seen fit to employ you I do not know.

from dannysullivan 139 days ago #
Votes: 0 | Vote:
+ -

To be fair on nocomment, Google did that in particular after Dave Winer ragged on them for years that they needed to do something to solve the blog spam problem that blog platforms themselves created by having open commenting systems. So Google finally got sick of it, rolled out nofollow, and PR problem solved. Spam didn't go away, as we all knew it wouldn't, but at least they stopped getting yelled at over it.

from g1smd 139 days ago #
Votes: 0 | Vote:
+ -

I don't agree with every last detail, but this seriously needs some wider coverage to get a discussion going towards some better standards.

You ought to find a sponsor that could take this through as a draft RFC. I don't believe that individual netizens can just submit an RFC on their own.

from Sebastian 139 days ago #
Votes: 0 | Vote:
+ -

Baiduyou, thanks. :)  Maybe I lost their email in an ocean of spammy job offers from far east companies and alike that I filter right to /dev/null. ;) Not.

Danny, that's true, and flies as sole reason in the PR dept., but Matt is a very smart guy who wished for ages to have such an instrument. Of course back in 2004/5 they thought of paid links (too) as another usage for rel-nofollow. However, it's a plausible theory, isn't it? If I'm dead wrong, I'll apologize.

G1smd, I guess that even when I find an RFC sponsor today, the whole process takes years and eventually gets buried. I believe that the major engines --and even Google alone-- can implement that better, and way faster, *if* they're interested. Actually, they owe that to the Webmaster community, because their ever changing policies burned billions of man hours for edits and updates forced by their guidelines.

from dzver 137 days ago #
Votes: 0 | Vote:
+ -

I like the idea for "Noodp: path". Rest of the article seems useless for me.


Log in to comment or register here.

Who Sphunn This Topic?

  1. Avatar dannysullivan
  2. Avatar billslawski
  3. Avatar JohnWeb
  4. Avatar imnotadoctor
  5. Avatar mpilatow
  6. Avatar Sebastian
  7. Avatar pratt
  8. Avatar beussery
  9. Avatar oggin
  10. Avatar northrock
  11. Avatar Sergi
  12. Avatar Harith
  13. Avatar Eavesy
  14. Avatar TannerC
  15. Avatar vangogh
  16. Avatar pittfall
  17. Avatar SearchBuzz
  18. Avatar AndyBeard
  19. Avatar SEMSpot
  20. Avatar mbeharry
  21. Avatar g1smd
  22. Avatar clickfire
  23. Avatar Jill
  24. Avatar Webnauts
  25. Avatar msaleem
  26. Avatar mickb68
  27. Avatar aglobalwarming
  28. Avatar SexySEO
  29. Avatar TheNanny612
  30. Avatar SEPTom
  31. Avatar NickWilsdon
  32. Avatar Menchov
  33. Avatar Kalena
  34. Avatar jennosborne
  35. Avatar SeoDesignSolutions
  36. Avatar OsmanP
  37. Avatar theGypsy
  38. Avatar swags2804
  39. Avatar Gab
  40. Avatar JMorris
  41. Avatar deezin
  42. Avatar KevinNewcomb
  43. Avatar baiduyou
  44. Avatar EmilyHanesNYC
  45. Avatar gunjanpandya
  46. Avatar yeepage
  47. Avatar mban23
  48. Avatar sphinndr
  49. Avatar dzver
  50. Avatar Gasmoney
  51. Avatar benrabicoff
  52. Avatar nolimitdomains
  53. Avatar Musa
  54. Avatar samirb

Who Desphunn This Topic what is desphinn?

Related Links

There are no related links for this story.

Top 10 Most Sphunn
In What's New

  1. How Papa John's averted a WOM disas... (20)
  2. Not Even a Little Evil (20)
  3. BunSpace is my space! (19)
  4. Social proof: The key to web 2.0 ma... (19)
  5. Are Broadband Internet Providers Sp... (18)
  6. Beating Google ? Can It Be Done Usi... (17)
  7. Representing Your Business on the S... (16)
  8. How to Get in Google News (And Rece... (16)
  9. GHOST BLOGGER REVEALS ALL! – The Se... (15)
  10. High Gas Prices Are Driving Relatio... (15)

Latest Comments

  • This is just another example of Google exerting its …
  • I Sphunn this because it is an interesting debate, but not likely sustainab…
  • $15 a gallon!$%^)(*&^%$…
  • Yeah, it provides a lot more insight into the issues they are facing and th…
  • It just depends on the client.  Enterprise level clients don't usually giv…
  • excellent article, very in-depth about the Twitter "melt-downs" we have bee…
  • -Real and human right from the start-Knowledge of the blog's content/…
  • i just reviewed the comment i made that was down voted so many times. i fel…
  • - read more -
Search Marketing Expo

Save the date for:
• SMX Advanced - Seattle, WA (June 3-4): All Access Passes are now sold out, but there are still 3 ways to attend. Click here for details.
• SMX Local & Mobile - San Francisco, CA (July 24-25) See the agenda, and register now!
• SMX East - NYC - (Oct. 6-8) Registration is now open!
• SMX London - November 4 & 5, 2008

©2008 Third Door Media, Inc. Privacy Policy & Terms. All rights reserved.

Contact Us  |  Sphinn Staff     powered by Pligg