Sphinn - Crawling vs. Indexing

61
Sphinn It!

Crawling vs. Indexing

Posted By: martinbowling 258 days ago
Topic Type: News Story (Jump to http://sebastians-pamphlets.com)
Category: Water Cooler

sebastian weighs in on what is Crawling vs what is Indexing. Very apropos discussion at the moment :)

9 Comments

Who Desphunn This (0)

Comments

from NickWilsdon 258 days ago #

Votes: 1 | Vote:

+ -

The first post since June 30th! This gets my vote just to encourage Sebastian to post more often ;)

from Sebastian 258 days ago #

Votes: 3 | Vote:

+ -

Thanks guys :)

from ColinCochrane 257 days ago #

Votes: 0 | Vote:

+ -

Spot on as always Sebastian.

from ChelseaBlacker 257 days ago #

Votes: 0 | Vote:

+ -

Sebastian, you should ask Aaron Wall if he's interseted in adding these to his SEO dicitonary (expand the current definitions)! And give you a link for credit.

from JohnHGohde 257 days ago # - show/hide this comment

Votes: -4 | Vote:

+ -

No! I would say that sebastian spoke out on the SEOmoz controversey.

SEOmoz is NOT that important people. If I see, let along read, another SEOmoz spinoff I am going to scream!!!

New articles are not necessary. Comments are just fine, Sebastia.

from annie7 257 days ago #

Votes: 2 | Vote:

+ -

I am happy Sebastian is back at last!

from mike 256 days ago #

Votes: 1 | Vote:

+ -

Not to be a pedant, but there are three steps:

Crawl.
Parser - this takes something and looks for meaning, e.g. title, links etc.
Indexing - this puts the new, meaningful data somewhere.

One can do any of these steps without tother, and even some stages multiple times (think - multiple parsers).

from IncrediBILL 256 days ago #

Votes: 0 | Vote:

+ -

per Sebastian's comment:
"The meta tag method is nice to have, but not practically, IOW more or less propaganda."

The meta tag method is the only way people using hosted CMS and blogs can actually control their sites because most services don't provide an efficient method such as robots.txt.

I think they should provide both.

I'm pretty sure they already do since I'm pretty sure I blocked their crawler and know it respects robots.txt, but I'm not 100% sure it's the only thing in use.

from Sebastian 256 days ago #

Votes: 1 | Vote:

+ -

> "The meta tag method is nice to have, but not practically, IOW more or less propaganda."
That addresses meta tags for each and every SEO tool out there. In the same comment I said I do use meta tags to steer the indexing of search engines, but can't be bothered to implement on-the-page indexer directives for everybody.
Of course I agree, they should provice both.