News Source: http://www.searchenginepeople.com
Published: Mar 08, 2008 - 02:27 pm
Story Found By: vangogh 1541 Days ago
Category: SEO
Rudd’s How Search Really Works Series.
2 Comments
Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.
News Source: http://www.searchenginepeople.com
Published: Mar 08, 2008 - 02:27 pm
Story Found By: vangogh 1541 Days ago
Category: SEO
Rudd’s How Search Really Works Series.
2 Comments
Upcoming Conferences
Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.
Join us at an upcoming SMX event:
- SMX Advanced Seattle - June 5-6, 2012
- SMX Paris - June 7-8 2012
- SMX Melbourne - Sept. 5-6, 2012
- SMX East - Oct. 2-4, 2012
- SMX Stockholm - Oct. 15-16, 2012
- SMX Social Media Marketing - Dec. 5-6, 2012
- SMX Munich - Apr. 9-10, 2013
- SMX West - March 2013
- SMX Toronto - April 2013
- SMX Sydney - May 2013
Latest News From Marketing Land
- Yahoo Shutters Livestand iPad “Living Magazine”
- Marketing Day: May 25, 2012
- Marketing Biz: Tracking Google’s Dune Strategy
- How Performance Marketing Is Influencing The Next Generation Of Internet Startups
- New Google+ Android App Adds Vivid Imagery & Additional Functionality
- - See more news at Marketing Land -
Upcoming Webcasts
Learn more about search marketing with our free online webcasts and webinars from our sister site, Search Marketing Now. Upcoming online events include:
- Search, Social and Mobile: What Travel Marketers Must Know - June 26, 2012
Latest News From Search Engine Land
- Google Releases Penguin Update 1.1
- Microsoft: Yes, We Do Send Takedown Requests To Bing, Too
- SearchCap: The Day In Search, May 25, 2012
- Search In Pics: SEO LEGO, Facebook Cooler & A Happy SEO
- Google Disapproved 134M Ads And Disabled More Than 800K AdWords Accounts In 2011
- - See more news at Search Engine Land -
Comments
"New stuff you find on the web goes into another, more temporary index."Not sure where you got this Ruud. Freshness is a factor that helps pages remain in the main index, not the other way around. Also, many pages, from my experience, start off in the main index, before turning supplemental."In such an index you can’t randomly insert new or updated documents or remove deleted ones. You would have to re-sort on every update."An interesting argument but the conclusion at the very least assumes Google is avoiding to re-sort on every update.
An interesting argument but the conclusion at the very least assumes Google is avoiding to re-sort on every update.Yup. And for the model described it’s not an assumption but a factual conclusion :)In some things that Google does, or did, you can recognize the fingerprints of setups common to information retrieval.To make a sorted positional inverted index takes huge amounts of memory. The common way to deal with that is to sort it in parts and merge those parts back.The common problem of that setup is that once you have sorted the index it becomes harder to insert fresh documents and dictionary terms in it.And the common way to handle that situation is to work with two indexes, two dictionaries: one main one and one supplemental one with fresh, new stuff that hasn’t yet been folded back into the main index.These are all basic facts and this is how Google has worked for most of its life. What they got much, much better at was folding updates back into the main index up to the point where they didn’t need a fresh crawl anymore.This is just the basic technical setup; any search engine engineer can tweak or improve the way this works. For example, I can take less popular (= less useful) information out of the main index, into a supplemental . It becomes a form of lossy compression; take the top 1000 postings for a query and discard the ones under it, keeping them ready to search only when the top 1000 doesn’t answer that.In that version of the model freshness or PageRank or indeed any other factor can be used to determine whether it’s worth the resources to keep working with a specific document.Freshness is a factor that helps pages remain in the main index, not the other way around. Also, many pages, from my experience, start off in the main index, before turning supplemental.I think you’re refering to supplemental results, not a or the supplemental index.Besides which, although these days the basic framework still is pretty much as described in my post, something very essential has changed in a big (daddy) way; the way Google store the index. The way they (now) generate and store the index and the dictionary has made main and supplemental as obsolete as ignoring stop words.Of course more about that in next week’s post :)