Sphinn Home » SEO
Avatar Went Hot: October 1, 2008 - 8:52 am
Posted By: annie7 61 days ago
Topic Type: News Story (Jump to http://seogadget.co.uk)
Category: SEO
An extremely useful example of how to use wildcard (*) in Robots.txt for sites that use dynamic query parameters.
3 Comments     

Comments

from morgret 61 days ago #
Votes: 0 | Vote:
+ -

I've found Google takes a long, long time to get rid of pages with parameters. Unfortunately, they have no way to remove URLs with specific parameters.

Yahoo Site Explorer has a way to specify what are your parameters, and how you would like Yahoo to deal with them. They also don't have a way to force removal of specific parameters, but I've found they do a better job at dealing with them, since you can specify your dynamic parameters.

From their page:

Specify up to 10 dynamic parameters that you want us to treat specially whenever these are seen in URLs belonging to [url] We will automatically rewrite the URLs containing these parameters as specified below. You can choose to:

  1. Remove these parameters from the URLs, such as in case of session ids, you could ask to remove 'sid' from URLs.

  2. Use a default value for the parameter, for example you could set the 'src' parameter to be 'yhoo_srch'

More help at http://help.yahoo.com/l/us/yahoo/search/siteexplorer/dynamic/

from robots 60 days ago #
Votes: 1 | Vote:
+ -

Nice guide with specific examples on one of the lesser known features of robots.txt protocol - thanks!

from wilreynolds 60 days ago #
Votes: 0 | Vote:
+ -

Great guide, wildcards have always been a bit of a questionmark for me.  Thanks!


Log in to comment or register here.

Top 10 Most Sphunn
In What's New

  1. Google Launches a Sweet Competitive... (18)
  2. Could you Buy or Sell Social Media ... (16)
  3. 30 Tips on How to Get, Convince, Ke... (15)
  4. fRank Takes on PageRank (12)
  5. How To Actually Get Links From Your... (12)
  6. Top 10 Reasons for PPC & SEO Synerg... (11)
  7. Rumors: Microsoft in talks to buy Y... (11)
  8. 16 Business Lessons for Freelancers (9)
  9. Backlinks index size comparison bet... (8)
  10. Google local search has directory e... (8)

Latest Comments

Search Marketing Expo

Save the date for:
• SMX West - Feb. 10-12, 2009
• SMX Munich - April 22-23, 2009
• SMX Social Media Marketing - April 29-30
• SMX Advanced - June 2-3, 2009

Search Marketing Now

Learn more about search marketing through free online webcasts and webinars from our sister site Search Marketing Now.

Upcoming Webcasts: