Sphinn - In Google index

Sphinn Home » Google SEO

3
Sphinn It!

In Google index - in Days

Posted By: DLPerry 502 days ago
Topic Type: News Story (Jump to http://willmaster.com)
Category: Google SEO

...with potential 'Duplicate Content' issues to boot. :)

3 Comments

Who Desphunn This (0)

Comments

from MattMcGee 502 days ago #

Votes: 0 | Vote:

+ -

That's a very good overview of appropriate steps to take in such a situation. Well written and easy to understand. Nice job.

BTW, robots.txt is THE first thing the spider looks for when reaching a domain. :-)

from kimber 502 days ago #

Votes: 0 | Vote:

+ -

hey so even if i have a small site and don't need to block anything are you saying it is best to include the robots.txt file indicating all bots can crawl all pages? will that give the search engines a little nudge to crawl if they haven't in a while?

from DLPerry 501 days ago #

Votes: 0 | Vote:

+ -

Thanks MattMcGee - I appreciate the kind words, and the confirmation about spiders and robots.txt. :)

kimber - I do feel it's best to include a robots.txt file - but I don't believe it will kick start any crawls.

I have noted that adding a Sitemap where one did not exist before 'can' spur a negligent crawler into activity, but since a robots.txt file doesn't actually list urls, per se - I doubt it has that effect.

But - as MattMcGee confirmed - a robots.txt is THE first thing a spider looks for when arrives at a domain. If there is the possibility that it's presence can speed or improve the efficiency of an already active crawl - I say go for it.

Of course - as Vanessa Fox mentioned in her blog on 'The First Rule of Indexing' - http://www.vanessafoxnude.com/2007/07/18/the-first-rule-of-indexing-make-sure-youre-letting-the-site-be-indexed/

don't forget to check Metas as well. :)

--dlp

Top 10 Most Sphunn
In What's New

5 Critical PPC Budgeting Problems (18)
Could you Buy or Sell Social Media ... (18)
fRank Takes on PageRank (15)
Top 10 Reasons for PPC & SEO Synerg... (13)
Conversion Immersion - Making the M... (13)
How To Actually Get Links From Your... (13)
Rumors: Microsoft in talks to buy Y... (11)
How to Write SEO Content (10)
How to Build Strong Relationships o... (10)
The Evolutionary Role of Social Med... (9)

Latest Comments

Save the date for:
• SMX West - Feb. 10-12, 2009
• SMX Munich - April 22-23, 2009
• SMX Social Media Marketing - April 29-30
• SMX Advanced - June 2-3, 2009