Sphinn Home » SEO
Everything you ever wanted to know about Robots from Vanessa over at Jane and Robot. Huge and awesome.
5 Comments     

Comments

from NeO 122 days ago #
Votes: 0 | Vote:
+ -

YOU RAWK Vanessa! Thanks... instant classic + EPIC WIN!

from Dez 122 days ago #
Votes: 0 | Vote:
+ -

Hooray for Vanessa! I'm sending this straight to my development team.

from GeorgeH 122 days ago #
Votes: 0 | Vote:
+ -

This is great!  I just sent the link over to several people I know who were working with this.

Thanks!

from oldschool 121 days ago #
Votes: 0 | Vote:
+ -

One of the best robots posts I have seen. Will definately link to it when the topic comes up. Incidentally I did a post yesterday which could have used it, but rather links to robotstxt.org.

from g1smd 120 days ago #
Votes: 0 | Vote:
+ -


Use Case  - Robots.txt - META/ X-Robots-Tag - Other - Supported By

Allow access to your content - Allow - FOLLOW - INDEX Google/Yahoo/Microsoft

Disallow access to your content - Disallow - NOINDEX - NOFOLLOW - Google/Yahoo/Microsoft


There is a lot of over-simplification here.   It might be wise to explain that links on a page can be followed even if the page is not indexed. 

Be aware that noindex (meta tag) does not mean "do not crawl the page", it means "do not show in SERPs". Google quite happily takes a copy of the page (they have to, in order to see the meta tag) and will follow links out from it unless you expressly disallow that.



Log in to comment or register here.
Search Marketing Expo

Save the date for:
SMX London - Nov. 4-5: Pre-agenda rate now available. Click here.
SMX West - Feb. 10-12

Search Marketing Now

Learn more about search marketing through free online webcasts and webinars from our sister site Search Marketing Now.

Upcoming Webcasts: