Sphinn - Managing Robot's Access To Your Website

50
Sphinn It!

Went Hot: June 10, 2008 - 4:08 pm
Posted By: Rhea 390 days ago
Topic Type: News Story (Jump to http://janeandrobot.com)
Category: SEO

Everything you ever wanted to know about Robots from Vanessa over at Jane and Robot. Huge and awesome.

5 Comments

Who Desphunn This (0)

Comments

from NeO 390 days ago #

Votes: 0 | Vote:

+ -

YOU RAWK Vanessa! Thanks... instant classic + EPIC WIN!

from Dez 390 days ago #

Votes: 0 | Vote:

+ -

Hooray for Vanessa! I'm sending this straight to my development team.

from GeorgeH 389 days ago #

Votes: 0 | Vote:

+ -

This is great! I just sent the link over to several people I know who were working with this.

Thanks!

from oldschool 389 days ago #

Votes: 0 | Vote:

+ -

One of the best robots posts I have seen. Will definately link to it when the topic comes up. Incidentally I did a post yesterday which could have used it, but rather links to robotstxt.org.

from g1smd 388 days ago #

Votes: 0 | Vote:

+ -

Use Case - Robots.txt - META/ X-Robots-Tag - Other - Supported By

Allow access to your content - Allow - FOLLOW - INDEX - Google/Yahoo/Microsoft

Disallow access to your content - Disallow - NOINDEX - NOFOLLOW - Google/Yahoo/Microsoft

There is a lot of over-simplification here. It might be wise to explain that links on a page can be followed even if the page is not indexed.

Be aware that noindex (meta tag) does not mean "do not crawl the page", it means "do not show in SERPs". Google quite happily takes a copy of the page (they have to, in order to see the meta tag) and will follow links out from it unless you expressly disallow that.