Published: Feb 18, 2008 - 07:40 am
Story Found By: ColinCochrane 3169 Days ago
This isnt a simple robots.txt exclusion either, but rather a 404 response that is now being served based on the requesting User-Agent.
This raises some questions as to the intentions of del.icio.us, and perhaps Yahoo! With Yahoo! recently integrating del.icio.us bookmarks into its search results this could an attempt to enhance the effectiveness of that new feature by preventing competing search engines from indexing content from del.icio.us.