Sphinn Home » Other Searching
Avatar Went Hot: March 3, 2008 - 10:39 am
Posted By: SpostareDuro 172 days ago
Topic Type: News Story (Jump to http://www.semclubhouse.com)
Category: Other Searching
There are 3 major parts to what a search engine does. What are some of the problems facing crawling programs?
5 Comments     

Comments

from Ruud 172 days ago #
Votes: 0 | Vote:
+ -

Good write-up by Bill Slawski. Nice nuggets in the post.

from iamlost 172 days ago #
Votes: 0 | Vote:
+ -

D*mn it, Bill, I'm several recommended papers (plus related research) behind already. Never mind developing associated business and domain plans sparked by your references.

I have rewritten 1.3 Politeness: 'webmasters become easily annoyed when web crawlers slow down their servers, consume too much bandwidth, or simply visit pages with “too much” frequency.'
as:
"web developers suffer recurring headaches when Bill Slawski overloads their minds and diverts their attention by simply uploading valuable information digests and links with “too much” frequency."

from billslawski 172 days ago #
Votes: 2 | Vote:
+ -

Thanks for the sphinn, Kimberly.

And thank you, Ruud. 

I've been told that I need to work on my politeness, iamlost.  I hate to say that there's more in the queue.   

from IncrediBILL 169 days ago #
Votes: 0 | Vote:
+ -

Most of us have IRLBot blocked because we already knew it was a complete waste of bandwidth and now it's totally confirmed, thanks!

from crazycat 162 days ago #
Votes: 0 | Vote:
+ -

Nice article. Crawling the web in the right way.


Log in to comment or register here.

Top 10 Most Sphunn
In What's New

  1. How Do You Measure Your SEO Campaig... (21)
  2. 43 Killer Viral Tips From Top Marke... (20)
  3. Yahoo Grouping Search Results (19)
  4. Domain Name Gaming: Tricking Engine... (16)
  5. Top 5 Legitimate SEO Techniques to ... (14)
  6. 24 Ways for SEO Detractors to Prove... (14)
  7. 5 FireFox Extensions to Enhance Lin... (14)
  8. Blumenthal & Mihm Jump Over Buildin... (14)
  9. Google Reader Adds Friends List, Sh... (13)
  10. How many businesses really ‘get’ th... (13)

Latest Comments

Search Marketing Expo

Save the date for:
• SMX China (Nanjing) - Sept. 23-24
• SMX Stockholm - Sept. 23-24: See who's speaking or register now.
• SMX East (New York City) - Oct. 6-8: See the agenda or register today and save!
• SMX London - Nov. 4-5: Pre-agenda rate now available. Click here.