Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

The most recent instructions from the SEOMOZE people suggest that the way to get removed from their Linkscape tool (to prevent your data from being sold to your competitors, for example) is to put an SEOMOZE-specific meta tag onto your pages.

CAUTION! Any time a company asks you to customize your web publishing for them, especially if that process includes adding their BRAND to your web pages, you should think carefully before acting.

By branding your pages with SEOMOZE, you demonstrate their INFLUENCE over you, and trust me, that influence will be leveraged further in the future. If Randy needs to demonstrate the reach and importance of his LinkScrape project, for example, he could simply point to how many web pages in Google specifically mention his "crawler". Your web pages will be counted as evidence of SEOMOZ’s success.

Never, ever add someone’s brand to your web pages unless you want to actively support them.

Please discuss if you agree/disagree. I can’t see any reason why SEOMOZ would ask for meta tags, except as a way to further benefit from the activity.

For reference, Archive.org publishes bot-blocking instructions *and* checks the robots.txt directive before displaying archived web data, as a means of providing privacy for those who want it. No meta tags.
Comments26 Comments  

Comments

Avatar
from merrick 2253 Days ago #
Votes: 2

When I read about the seomoz meta tag, I remembered the footprint the digital point link coop left behind. I don’t want my sites targetted via footprints.What’s so hard with following robots.txt?

Avatar
from merrick 2253 Days ago #
Votes: 0

nickwilsdon answered my question on why it’s so hard to follow the robots.txt:http://sphinn.com/story/80142#c56044in short, they don’t control the crawling.

Avatar
from randfish 2253 Days ago #
Votes: 0

@merrick - that’s not entirely accurate. We control the vast majority of crawling, but if/when we backfill with data from other providers, we need to make sure that those sources don’t contain pages people would want removed from our list of links. Thus, a meta tag on the page level is the best way to go.

Avatar
from patrickaltoft 2253 Days ago #
Votes: 4

Ok so assuming I did block my own websites surely all my competitors could still see my link information because I am unable to block crawling of all the sites that link to me? Correct?<div></div><div>If so then there seems little point in blocking it.</div>

Avatar
from NickWilsdon 2253 Days ago #
Votes: 0

@MerrickAs Donna said, you can block DotBot with robots.txt. This spider seems to be the main robot providing data for Linkscape (and the one Moz "controls") User-agent: dotbot Disallow: / However your site information can still be included from the other sources Linkscape buys data from (or free API data from major search engines). As Rand suggests, the META is the only way they can ensure you are removed.

Avatar
from randfish 2253 Days ago #
Votes: -3

@patrickaltoft - yes, that’s right. You’d need to get all of your link partners to block in order for us not show links to your sites/pages/@nick - yes, meta is the only way you’ll ensure removal from the results right now, though we may build in blocking for sites that verify in the future.

Avatar
from merrick 2253 Days ago #
Votes: 0

@nickwilsdon got it@randfish thanks for clarifying, I understand why a meta tag is needed - it’s just not ideal.

Avatar
from IncrediBILL 2253 Days ago #
Votes: 0

Do I hear the x-files theme playing?

Avatar
from axemedia 2253 Days ago #
Votes: 1

Now if you did have a certain network of sites you didnt want competitors to know were linking to a certain site then you would block it on all those sites, regardless if other sites you had no control over were linking to the certain site. Linkscape would only show a portion of what was linking to what.And some IP cloaking could prevent meta tag footprints, perhaps.Just thinking out loud.

Avatar
from liamvictor 2253 Days ago #
Votes: -1

<div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10px; background-color: #ffffff">Even if SEOmoz doesn’t control the crawling of all the pages, surely it’s fairly trivial to get the robots.txt once a week or so for each domain discovered and drop the data from those that want to be excluded.</div>

Avatar
from Skitzzo 2253 Days ago #
Votes: 1

@liamvictor, maybe if you controlled a single site and didn’t have anything better to do with your time. Unfortunately I don’t care to spend any amount of time making up for someone else’s attemt to make money off my data, and certainly not once a week for every single site I own.

Avatar
from liamvictor 2253 Days ago #
Votes: 1

@Skitzzo Perhaps I dind’t make myself clear, I’m not suggesting publishers do anything apart from add a rule to robots.txt, just like we do with any other spider we want to exclude.<div></div><div>I’m suggesting that SEOmoz get the robots.txt themselves and promptly exclude the sites that don’t wish to be in their index. They can easily get this regardless of if they actually spider the pages themselves and can therefore exclude pages much quicker than the 60 or so days Rand spoke of.</div>

Avatar
from g1smd 2253 Days ago #
Votes: 2

<div style="font-family: Verdana, Arial, Helvetica, sans-serif; font-size: 10px; background-color: #ffffff">Err, the "once per week" reference wasn’t for you to do anything at all.<div></div><div></div><div>It was in reference to bots maybe checking it once per week.</div><div></div></div>

Avatar
from liamvictor 2253 Days ago #
Votes: 1

@g1smd Exactly. For *us* it would be a set it and forget it change to the robots.txt

Avatar
from neyne 2253 Days ago #
Votes: 3

OK, can somebody explain what am I seeing here ? :http://ws.arin.net/whois/?queryinput=seomoz.orgOrgName: seomoz.orgOrgID: SEOMOAddress: dotnetdotcom.orgAddress: 93 S. Jackson Street 10070City: SeattleStateProv: WAPostalCode: 98104-2818Country: USComment: RegDate: 2008-07-07Updated: 2008-07-07AdminHandle: NGE11-ARINAdminName: Gerner, Nick AdminPhone: +1-206-299-9628AdminEmail: admin@dotnetdotcom.orgTechHandle: NGE11-ARINTechName: Gerner, Nick TechPhone: +1-206-299-9628TechEmail: admin@dotnetdotcom.org

Avatar
from Skitzzo 2253 Days ago #
Votes: 0

@liamvictor ahh, my bad. I thought you meant we would have to once a week update our files to keep up with any new data sources they release. Sorry about that!

Avatar
from SamFreedom 2253 Days ago #
Votes: -1

I feel bad when even all you MENSA brainiacs can’t protect yourselves from yourselves...

Avatar
from johnandrews 2253 Days ago #
Votes: 3

Since Linkscrape is making the profits, have Linkscrape check the robots.txt for permission *before* delivering reports. Just like archive.org.... it’s not a problem with the so-called "crawl", it canbe blocked at the service delivery point.@neyne yeah, there’s apparently some funding of IPs going on.

Avatar
from WillSEOForFood 2253 Days ago #
Votes: 4

This might be a tad paranoid, but going with the footprint theory I certainly wouldnt add anything SEOMOZ to any of my clients site. Calling attention to yourself or to your clients that they are under "SEO supervision" especially after the past issues with SEOMOZ having no problem playing "tattle tale" on other sites like a certain directory that shall go unnamed... doesnt sound like such a great idea.It seems like they make money off of your data one way and if you try to opt out (which it seems like you really cant) in the future they or the companies they are getting their data from (you know small companies like Yahoo, Google, etc...) could use those "lists" against thousands of sites. Think about what happened with link exchanges and the big daddy update in 05...It’s sad when SEO’s appear to be trying to capitalize and cannibalize on not only other SEO’s but it appears the industry as a whole.

Avatar
from SEMSEO 2253 Days ago #
Votes: 0

I have more than a hundred site hosted on a server. Will Linkscrape take up my server resources the way major search engines and spambots did?

Avatar
from JohnHGohde 2252 Days ago #
Votes: -7

Adding a meta tag definitely suggests that SEOMOZ is crawling the Web.  So what was all that endless argument about SEOMOZ using somebodies elses data all about?Not only is SEOMOZ not important enough to add a special meta tag, but my concern is that any use of NOINDEX might cause pages not to be indexed by the major search engines.  If SEOMOZE wanted to use a meta tag then they should have used something like: noSEOMOZ.If SEOMOZ has their own bot then the robots.txt would obviously do the job.My next line of defense are WordPress plugins that are supposed to protect my blog against evil bots.

Avatar
from claye 2252 Days ago #
Votes: 2

This conversation, period (along with the half-dozen Linkscape-related stories on the front page of Sphinn), gives the SEOMOZ brand as much recognition than a few invisible meta tags ever will - and isn’t brand recognition the only thing at stake here?"Their INFLUENCE over you" = brand recognitionThat’s all. Don’t these comments represent the SEOMOZ brand’s influence over all of us? Does that make everyone supportive?

Avatar
from johnandrews 2252 Days ago #
Votes: 0

@claye: There’s no question Randy and SEOMOz are influential in search marketing - you can’t take that away from them. They deserve their brand, no question. But this is a bigger picture. If the fix for a negative problem is putting SEOMOZ into your headers, then the fix is making the problem bigger for the long term. I think we need to resist supporting such negative actions with endorsement.Talk about SEOMOZ and Linkscrape as much as the project deserves. I’m still amazed Randy hasn’t turned this into a net positive for his company and Linkscrape, but that’s his business.

Avatar
from SEMSEO 2252 Days ago #
Votes: 0

But if we don’t put SEOMOZ in our robots.txt, then how are we going to stop Linkscape from crawling our server?

Avatar Moderator
from Jill 2252 Days ago #
Votes: 1

@johnandrews curious why you call Rand "Randy."

Avatar
from johnandrews 2251 Days ago #
Votes: 0

@SEMSEO robots.txt is not the same as meta tags... an exclusion in robots.txt is clearly a negative vote, while a meta tag is a supportive vote. That’s all this post was about... being careful not to accidentally endorse a service when you actually wanted to block it.@jill that’s the second time I’ve been asked that... is that not his name? I called him that when I met him in Portland and he didn’t seem to mind.I sense that now that things are clarified, some might be looking for more in these discussions. Sorry... it’s all about clarity, not "personal agendas". I know that’s not as colorful as some would like things to be...

Upcoming Conferences

Search Marketing ExpoSearch Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.



Join us at an upcoming SMX event: