How To Block The Bots SEOmoz *Isn’t* Telling You About

Hot News Archive

Sorry this site requires JavaScript to be enabled in your browser. See the following guide on How to enable JavaScript in Internet Explorer, Netscape, Firefox and Safari. Alternatively you may be blocking JavaScript with an advert-related or developer plugin. Please check your browser plugins.

News Source: http://smackdown.blogsblogsblogs.com
Published: Oct 17, 2008 - 02:35 pm
Story Found By: mvandemar 2296 Days ago
Category: SEM

Some insights into the data that SEOmoz revealed today about what their sources for the Linkscape tool are, and what those sources reveal about things behind the scenes.

61 Comments

Comments

from seanmag 2295 Days ago #

Votes: 15

First, let me say that I’m surprised to be the first person commenting on this thread, considering there are 42 Sphinns. Clearly, if Mr. Vandemar’s harsh accusations are true, (he basically calls Rand and others in his company liars), this is a SEVERE BREACH OF TRUST of the entire SEO community by a self proclaimed honest and forthright guy - Rand Fishkin.In the post, Mr. Vandemar makes no bones about his accusation that Rand Fishkin and others within SEOmoz, propagated a series of lies and mistruths, all in an effort to dupe the SEO community out of more money. In other words he implies that they are selling the proverbial Snake Oil.Just one week ago, despite the fact that I didn’t suggest Rand was being dishonest, he had this to say to a general comment I left at Marketing Pilgrim at the following url http://zi.ma/270511<i>"Ouch Sean. You really think that I would write nice things about a session dishonestly due to the corrupting influence of our sponsorship of the event or my friendship with Danny? That hurts."</i>Rand - let me say this - Based on all of the pre-release promotion you did with regard to LinkScape, and after reading every piece of commentary I could find post release, I’ve come to the conclusion that you would say something dishonest if you felt it was in your best interests. I don’t know how, after reading everything I have in relation to LinkScape, I could conclude otherwise.I also feel you violated a trademark of LBI-Netrank in using the the name LinkScape, as pointed out by SEOmoz community member Richard Manely. http://zi.ma/d68beaInterestingly, this accusation was only responded to publicly by your employee Scott, who stated:<i>"We did check it at the outset and during planning (we’re not that careless) and saw nothing that would be an impediment."</i>I find this comment shocking and hard to swallow since a simple Google search of "linkscape" would have returned the LBI-Netrank trademark.When I consider everything I’ve read I can only conclude that you did mislead the SEO community and only when it was apparent that the truth would be found out, did you begin to "come clean". While your approach may not have any bearing on the value of the tool, it does demonstrate a conscious effort to misrepresent your product.Further, I find your response, to Mr. Vandermar’s following statement on this post, to be severly lacking.Mr. Vandemar:<i>I have to admit, Rand, it’s pretty bold to basically admit this late in the game that you guys lied through your teeth and grossly misrepresented the facts, just so you could appear to have accomplished a much bigger task than you actually did, all in the name of getting more money from webmasters. That’s a much bigger admission than saying you cloaked your bot, if you ask me. Gratz on coming clean.</i>Rand:<i>Sorry about that - obviously, it was supposed to be Gigablast (lousy typos). Gagablast sounds like some sort of infant vomit <img class="wp-smiley" src="</i>Personally, I feel you owe the SEO community and your own community of thousands, a much better response than that. If I were to have purchased your service under the impression that you had created your own bot/s, and had effectively crawled and indexed the web to the tune of 30 billion plus pages, and if I was led to believe that SEOmoz would be able to continually update and thus maintain the relevancy of this data, only to find out that it is actually an aggregated mashup of other company’s data - I would be pretty upset.

from seanmag 2295 Days ago #

Votes: 7

For some reason, the tail end of my comment was left off, but I think it’s worth re-stating:If I had purchased this service under the pretense of SEOmoz having created their own bot/s, and having effectively crawled and indexed the internet to the tune of 30 billion plus pages, leaving me with the impression that SEOmoz could, at will, continue to update and keep the information relevant, only to eventually find out after the fact that the information is an aggregate mashup of data from a wide range of providers - I’d be pretty disappointed.

from neyne 2295 Days ago #

Votes: 5

It is funny how this keeps happening to Rand. Somehow, there is always something that needs post-launch fixing and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare. This was the case with Marty’s post and with the directory outingNot to talk about launching a tool promising to be one of the most revolutionary and useful tools for SEOs on the web and then leaving for SMX, without anyone (except for ppl responding in the blog comments) to respond,like immediately, to any issue that arises. It kinda hints that no one has even thought that something about the tool could be problematic.@seanmag. I guess that a tool should be judged by the quality of the information provided, not by the method the data was achieved. I mean, if Linkscape provides me with more incoming link data and analysis that I could have gotten elsewhere, then there is value in their "mashup". I do see a problem with presenting the tool as a based on web crawling as its selling point, only to turn out that the crawling was done mainly on offline databases

from Skitzzo 2295 Days ago #

Votes: 7

@neyne, exactly. If something like this happened once or twice, I’d give them the benefit of the doubt but it’s become a pattern of behavior with SEOmoz.1) Controversy2) Benefit from attention3) Apologize but due to time can’t fix it just yet.4) Continue to gain attention.5) Finally fix it and explain how aw shucks guys, I didn’t mean it like THAT. 6) Ask people to talk to you about it in person rather than writing about it in public.I don’t think Rand is a bad guy and he may not even see that this is a pattern but it certainly has become one.

from nicknick 2295 Days ago #

Votes: 3

I was certainly under the impression that they had their own crawler....

from neyne 2295 Days ago #

Votes: 1

@skitzzo I would like to personally thank you for using the word "shucks". It is an excellent word that i have completely forgotten about :)

from IncrediBILL 2295 Days ago #

Votes: 7

It’s possible it’s all just a coldly calculated marketing ploy designed to keep people talking about SEOMoz and Linkscape, you still are, and it’s way cheaper than advertising.If in fact it’s deliberate marketing tactics, then he’s working way above the heads of most of the marketers in this industry and you should all be taking notes.So either it’s an inadvertent trend that just happened as some people pointed out or strategically planned marketing designed to be socially viral, either way it’s working.

from randfish 2295 Days ago #

Votes: 4

Sigh... I wish I could comment, but the decisions made internally are firm. The information that’s been released so far is the extent of what we’re offering right now.I will say that we do have our own index of ~30 billion pages. We’re updating every month, and should almost double that come November. Our index does bias towards domain diversity (hence, while Microsoft says they keep about 72 million domains in their index, we have ~150 million), but we dont have the same depth as the engines, especially on very large, deep sites (Amazon, Wikipedia, etc.). We’re certainly hoping that in the months to come (and as the product moves out of beta), it will be a more close approximation of Google/Yahoo!/etc. but even now, if you do a lot of link research in there, you’ll find plenty of cases where we report very similar or even a few more links than Yahoo!’s Site Explorer.As far as this - "I’ve come to the conclusion that you would say something dishonest if you felt it was in your best interests" - I don’t know. Maybe I should just stop trying to fight this perception. So many people tell me I should just ignore it. Maybe I should listen.SEOmoz is, in my opinion a good company, with lots of great, hardworking, talented people. They’re honest, committed and, at least in my opinion, dedicated to a set of impressive ethical ideals. Linkscape itself was built because we wanted more disclosure on what the engines did, and felt that as a community, we deserved to have a resource from which to obtain it. You might argue that, since we’re not a non-profit, and took investment, the free data we provide isn’t enough and the paid data is too expensive, but I see so many companies in this field and others who never try to give back to the extent that we do, and I guess is just saddens me to see this devolve into accusations of dishonesty and greed.So - sorry we can’t give full disclosure about everything. Anyone who knows me knows that in an ideal world, that would be my desire. I do hope, however, that the level of discourse can rise above petty vitriol and negative personal attacks to something more aspirational and more worthwhile.

from mvandemar 2295 Days ago #

Votes: 6

Rand, seriously, what school did you go to that teaches that playing the wounded bird and professing honesty will beat out cut and dried facts? You very clearly introduced your tool with the impression that SEOmoz was a company capable of crawling the web on their own and building an index of 30 billion webpages. Obviously you did so because you didn’t feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant. The fact of the matter is, that’s the path you decided to take. Once you got in a bind over the fact that if you had done that then you did it with cloaked bots, you decided to release where you actually got the information... and it wasn’t from the way that you presented it. There is no gray area here. There is no secret that you can claim you can’t tell us that would make the lie you told not be a lie. C’mon now.

from johnandrews 2295 Days ago #

Votes: 9

@neyne said: "It is funny how this keeps happening to Rand."Not funny. I suggest it’s a reflection of Rand’s judgement; how he makes his decisions, in other words how he behaves. And it’s good to see, because you can pretty much bank on people behaving in the future the way they have behaved in the past. Another great clue useful for predicting future behavior is the quality and character of the people someone chooses to be around, hire, or otherwise engage in business with.... which is one reason I pay attention to such things. Hands down, with virtually no exceptions, organizational behavior is a tail wagged by the dogs at the top. The most interesting for me throughout all of this year’s SEOMOZ reputation nightmare is that bit Rand added at the end of his comments above (and other SEOMOZ fanboys have added to other discussions elsewhere) -- that this conversation is "petty vitriol and negative personal attacks". As before, consistent behavior -- if you don’t like something Rand does, no matter what the facts, you’ll be accused of making "personal attacks". Given that is apparently how Rand and his fans behave when questioned, I know what will be said of this.. my unsolicited comment on this thread. Must be sour grapes, right? Something personal between me and Rand? Sorry.. there is nothing personal between Rand and I.

from linkmoses 2295 Days ago #

Votes: 10

Good grief everyone, if you consider the entire body of work Rand has contributed freely to the SEM community over the years, his genuine and generous contributions are easily among the most useful ever made. Whether or not you like the way LS was rolled out, or whether or not you like Rand period, rather than piling on why not give the man the opportunity to do what he’s alway’s done. Sure things could have been done differently, but he will make it right because my gut tells me he wont be happy inside until he does. He’s a guy who wants us to like what he does, and that’s a rare and good thing nowadays. When I started link building I think Rand was 14 years old. To watch what he’s done and what he’s freely shared is amazing. I want guys like him nurtured and funded, because I want to see what he does next. I’ll take a few bumps along the way because the ride is worth it.

from randfish 2295 Days ago #

Votes: -2

There have been a few times when I’ve seen people call Danny Sullivan out, usually unfairly (but, then again, I’m pretty biased when it comes to him), in discussions like these, and I always liked the way he responded to all the criticisms one by one, so I’m going to give that format a try.On the accusation that we don’t have our own web crawl - patently false. Go use the tool, note how the information is different than any other publicly available source. Linkscape runs on a completely unique index we built from scratch. It would be completely impossible to construct the link graph metrics without it. In fact, it would be impossible to extract many of the data points we’ve shared about size, diversity, coverage, use of links, etc.On the accusation that the construction of the tool is unethical or unwarranted - I believe very strongly that the information the search engines have at their disposal, in particular, the link graph, should be available to everyone who wants it.On the charge that it’s unfair or unreasonable to charge for some of this information - I disagree. I think that without a capitalistic model, innovations like these wouldn’t be possible. Would it be nice if we gave all the data away for free? Yes. But it costs an incredible amount to build something like this, and takes tremendous time and energy from a lot of smart, talented people who are not independently wealthy. They deserve to reap the rewards of their labor.On the Linkscape copyright - we have an in-house attorney who is extremely careful about all this stuff. I don’t know nearly enough about it, but I know they’ve been in touch, and the discussions have been positive. I think it’s unfair to criticize when you’re not the offended party and don’t know what the situation is.About dishonesty to help myself or my company - I guess it’s something I hope I wouldn’t do, but the bigger the business has grown, the more I’ve found that honesty (and especially transparency) does come into conflict with promises that have been made and the needs for stealth and protection. I will always try to do my best on this issue, but I think it’s a fair critique to level. I wish there were more I could do, but so far, it’s been a tough path to navigate, and I doubt that as we get bigger it will get easier, unfortunately.Regarding the "duped" into buying Linkscape - Sean, your arguments seem to center around the idea that we won’t live up to the promises of growing the index and keeping things fresh. If that’s ever the case, you can always get a refund, first off, and second, I reject the notion that we won’t live up to this promise - we will. We’re already on track, as I said, to nearly double the index size in November, and to continue to get bigger and fresher thereafter.On the "pattern of behavior" - I think it’s a very accurate assessment of me and how I operate. I’m frequently very busy, nearly to the point of being overwhelmed, and most of the other folks at SEOmoz are in a similar situation. Thus, you’ll see that responses aren’t always extremely quick. We also haven’t been and apparently are not good at predicting what will raise the ire of certain folks in the community - sometimes when we do think something will cause a stir, nothing and then when we think people will react positively, it turns ugly. Again, it’s something we’ll try to work on, but I don’t know how far we’ll get. Our focus and our energy are always going to be primarily centered around building our tools, content and resources. I’m not sure that we’ll ever find that there’s ROI in being hyper-sensitive or hyper-responsive (but we will try harder).Coldly calculated marketing ploy - if it was, it would be a pretty bad one. Negative attention is something that really messes with me and with all of us. We’re a surprisingly (or maybe not so) sensitive group, so keeping everyone happy with what we do is a big goal, even if it’s not something we’ve always achieved. I know that many people give us the advice that we can’t please everyone unless we’re mediocre and boring, and that the anger and attacks just show that we’re on the right path. I’ve never been able to fully buy into that argument, and so I keep trying to have it both ways. Maybe that’s foolish, but we’ll probably keep trying.On Michael’s accusation that people would only buy Linkscape because they felt bad for all the work we put in - that seems very incongruous and illogical. While I’d certainly appreciate the gesture, I’d seriously hope that it wouldn’t matter whether it took us ten minutes or ten months, the judgement would come based on the value of the tool and how they could use it to make themselves better marketers and SEOs. To buy for any other reason seems tremendously unwise.Regarding John’s accusation that my judgement is poor - it could be. I obviously try hard to make the right decisions, but as a CEO, I make dozens every day and hundreds every week. Whether all of those are the best they could be and always right isn’t up for discussion - with so many to consider, it’s impossible that I’d always be perfect. I don’t think anyone is. My job is to make enough "right" decisions to help grow the company, keep our investors, our employees and our customers happy. To date, I think I’ve had some degree of success, but I know that I haven’t generally (or often) made John happy. However, I do think you’re being a bit untruthful when you say it’s not personal. Go and re-read the many things you’ve written about me and about SEOmoz - it’s personal. That’s OK - I think I might have been the one to make it personal when I got angry about an attack you made on Danny in the past. Let’s just not try to pretend that it’s not, though.I think that’s just about everything. Let’s see if this style actually works... :-|

from Gab 2295 Days ago #

Votes: 2

@Mike Vandemar: You statement assumes that other people are as technically capable as you are or that they are interested in replicating what SEOmoz have done: "Obviously you did so because you didn’t feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant."Personally, I could care less about assembling the team and hardware required for what Linkscape originally appeared to be OR for what it has turned out to be based on. Of course, it was quite misleading as you highlight to talk about crawling so much, since it gives the impression that the data was first-party data. My point is just that at the end of the data, the practical implications are not that significant. For your average search shop, you can just go in a la Dan Thies, run some reports and compare the data to Yahoo’s. If it’s better, use it. If not, drop it. @Neyne - "Somehow, there is always something that needs post-launch fixing and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare. " --> Launch and iterate is hardly an SEOmoz-only practice. Any business startup / product launch advice columnist/blogger etc will suggest that."This was the case with Marty’s post and with the directory outing." --> Two screwups I’m sure they regret. On a related note, I personally have said things online that were interpreted incorrectly and by the time I found out about the problem, my original comments were deleted having been deemed offensive. You can imagine the difficulty I had making it up to the offended parties.Sean’s highlighting the trademark issue is perhaps the most legitimate, and cutting criticism of SEOmoz’s behaviour here. If LBI’s product was around first, I don’t see how SEOmoz can maintain their product’s name, frankly. The potential for confusion, especially with both products being in the same online marketing industry, seems pretty significant. Of course, this is just my humble 2 cents... not a legal opinion.Finally, If you meet Rand et co. personally, you’ll see that they’re really nice, friendly people. While much of the criticism (e.g. stealing bandwidth and the TM issue) is legitimate, there is an undercurrent of ad hominem attacks or else an overt current to it ("quality and character of the people someone chooses to be around, hire, or otherwise engage in business with"). I find that childish and something Sphinn can do without.@John Andrews - since I singled out your comment, let me say that you know from my past comments on your blog that I have tremendous respect and appreciation for your expertise. You’re one of the brightest guys in online marketing, no doubt. But when you get into the stuff about people’s personalities, those are personal attacks - even if you deny it or call me a fanboy - and they detract from your personal brand. SEOmoz’s boss and staff are human, and they’re entitled to make mistakes too. (As to SEOmoz not learning from their past mistakes, well that’s obviously something they need to get an outside hand on.)

from Gab 2295 Days ago #

Votes: 1

@Sean - Scott, as you know, isn’t their inhouse counsel. Perhaps Sarah Bird didn’t check it herself, or didn’t check thoroughly.

from randfish 2295 Days ago #

Votes: 0

@Gab - Sarah generally advises me not to talk publicly about legal situations, but I promise that she did her homework very carefully. One major difference is the country of registration for the trademarks - UK vs. US - but there very likely are other things in the equation as well.

from mvandemar 2295 Days ago #

Votes: 4

Rand, I made exactly one accusation, and tossing in a bunch of bs that I never said isn’t helping your case.A crawler is a web spider that goes out and collects webpages. You don’t have one, you said you did. You obtained the data from publicly available sources, some free, others you bought. That’s what I blogged about. I didn’t come to that conclusion combing through secret govt. records or by hacking your email, I did it by reading your blog. It’s right there in black and white for anyone to see. You said one thing at one point, and another later on, and the two things conflict. End of story.Onward to your new bs... "On Michael’s accusation that people would only buy Linkscape because they felt bad for all the work we put in" - please show where I ever said that. I know you love to obfuscate things, but I never even said anything even close to that, nor have I made any of the other accusations you mention.

from randfish 2295 Days ago #

Votes: -1

Michael - up above, you just said:"You very clearly introduced your tool with the impression that SEOmoz was a company capable of crawling the web on their own and building an index of 30 billion webpages. Obviously you did so because you didn’t feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant."I believe the exact intuition there is that we hoped to "impress" not with the value of the tool, but with the quantity of effort. Maybe that wasn’t your intent and I misread it - if so, my apologies, but if it’s not a point I needed to refute, then let me know if there is a valid one I should be addressing.As for the crawler - the decision has been made not to reveal any more information than we already have on that front (http://www.seomoz.org/linkscape/help/sources), but regardless, the crawl itself (aka index) is unique and built, 100% by us. No other organization has access to it and no one but SEOmoz is responsible for its creation. I said we built a crawl of the web, and we did, and anyone can see it by using the tool. I recognize that I’m not being completely transparent (and that it’s a conscious decision and one that, perhaps, will receive warranted criticism), but I fail to see how it’s a "lie."

from IncrediBILL 2295 Days ago #

Votes: 6

@Rand "unfair or unreasonable to charge for some of this information"It’s unfair and unreasonable to charge a site owner for his own information.Even Majestic12/MajesticSEO got that right, you give something back to the community so the community has a vested interest in helping your project instead of being a one-way leech.If you’re going to ask to crawl someone’s site you need to give something in return.Otherwise people will lock your ass out and your investment will turn into a steaming pile.

from mvandemar 2295 Days ago #

Votes: 2

You did do it to impress, I stand by that. I never said you did it so people would "feel bad for all the work" you put in. A peacock fluffs out it’s feathers to make itself look more important... not so others will "feel bad" about the effort they made.<div>And, nice try, but a crawl is not synonymous with index, especially when you start tossing in phrases like "Our process for crawling the web...", or giving the bullshit impression that you prioritize which sites/pages you will crawl next (or more often) based on how important you think those pages are.</div>

from randfish 2295 Days ago #

Votes: -1

@Incredibill - a fair critique. We will definitely talk about providing site owners the ability to verify their own domain and see data related to their sites. I think that’s a very reasonable request and would even be a good move for the product on the marketing side.@Michael - If you stand by your critique of us trying to impress, then I’m going to stand by my refutation that this would be a very bizarre criteria on which to base a marketing plan and an even worse reason to make a purchase. Judge Linkscape (or any product) on its merits, not on what you perceive the construction effort to be.On the second point - I believe a "crawl" would be, semantically, very much the same as an "index" though technically, an "index" would take the raw data from the crawling process and use it to build structured information (and we have both of these). I think your issue is with the perceived deception that there was a single Linkscape bot collecting the initial data - and again, all I can say is that all the stuff we’re willing to say on that subject publicly is out there now. I think you’re attempting to accuse based on semantics, and it feels, as previous attacks have, very personal and very biased. I do think you have a legitimate case to complain about plenty of things about Linkscape and SEOmoz and my reactions - I’m trying the best I can, but I know I make mistakes and I can even be an ass at times (and if I have, I apologize), but I don’t think this singular issue is worthy of the magnitude you profess.

from mvandemar 2295 Days ago #

Votes: 1

Rand, whatever. "Our process for crawling the web" and "we’re crawling everything we can" are not ambiguous statements, no matter how badly you wished they were at this moment.You are right on one point though... it is all publicly out there now. You’re banking now on the fact that most people won’t go sift through the data to see where you are bullshitting, and that by Monday no one will care. But the bottom line is that for those who want to read them, your words do in fact speak for themselves, in all their self contradicting glory.As to the "personal attack" defense, I think pops of SEORefugee said it best... it’s hard to accuse someone of lying and not have it sound personal.

from NickWilsdon 2295 Days ago #

Votes: 10

@Rand To be fair, you did actually say you built a crawler not a crawl. There used to be a section in the LinkScape help section entitled "Linkscape’s Crawler" which has now been removed but remains in the Google cache. Michael and Sean’s point there seems valid.

from randfish 2295 Days ago #

Votes: -1

Fair enough - I think the style of your blog post and the history make it even harder not to be taken as personal, but that’s a fair point about how difficult it can be to make the accusation without it being personal.As far as crawling goes, I’ll just say that those statements are not dishonest or misleading. Our crawl comes from sources we control and therefore we have both processes and biases for what information goes into the index Linkscape produces and how. We bias towards domain diversity, we use mozRank as a metric to determine how deep the information from a particular site is and we make determinations about what to include next and what needs re-visiting sooner vs. later.And one more thing - SNL just now airing on the west coast and OMG Tina Fey/Sarah Palin thing is hilarious. Macgruber!!

from mvandemar 2295 Days ago #

Votes: 2

Rand, you honestly think this is a factual statement?Over the course of several weeks, Linkscape’s spiders reach many billions of unique URLs on many millions of domains, which is similar to various search engines’ crawl rate... Linkscape crawls the web just like any other spider, discovering links to new pages and following them.As to the "history" making things seem more personal... I can honestly say that I have never resorted to making things up about you in order to make you look bad, while that is indeed something you have done to me in the past.

from randfish 2295 Days ago #

Votes: 0

It is a factual statement. The crawl we’re showing was collected between August and September for the October release, so I think that fits very fairly in with the "several weeks" assertion. And Linkscape’s data is not substantively different from that of any other crawl/index - just play around with the tool and you’ll see the link data matching up really solidly to what’s listed in Yahoo! Site Explorer or Google’s Webmaster Tools.On the history account - I think that’s a dead end. We’re just not going to agree there.

from mvandemar 2295 Days ago #

Votes: 1

Dude, Linkscape’s spiders don’t reach many billions of pages because Linkscape’s spiders don’t exist. This isn’t semantics, it’s fantasy. Even if you try and make the claim that you do have additional spiders of your own that you are disguising the user agent on, those spiders aren’t reaching "billions of unique URLs". Also, I would think very, very carefully if your planned defense to this is going to rely on some as of yet undisclosed spider running around that you refuse to let identify itself. Seriously.

from randfish 2295 Days ago #

Votes: -1

@NickWilsdon - A very fair point - we did say we built a crawler, and that’s something I stand by, because it’s true. More detail I can’t provide, but it is an accurate statement. Sorry I missed your comment earlier - I think I was composing while it was posting.@Michael - We’ve revealed all the sources we use - they have spiders, they reach billions of pages - those billions of pages are in the index. It sounds like your complaint is that there’s no spider called "Linkscape" but I think that would be immediately apparent to anyone who’s seen their sites/pages in the tool, since no bot I’m aware of crawls the web with that user agent. We also disclosed when we first wrote about the service that there was no bot named Linkscape or SEOmoz, which is what aroused ire that it couldn’t be blocked.I’m not suggesting we have any bots that disguise the user agent name - let’s not start any more inaccurate rumors. If you were describing the situation, wouldn’t you say that "Linkscape’s spiders reach billions of pages" since it is entirely true that, well, Linkscape’s spiders reach billions of pages?

from mvandemar 2295 Days ago #

Votes: 3

No, it’s not even close to true, and no, we’re not talking semantics, and no, you are not going to pull it off that this was some sort of honest mistake."Linkscape crawls the web just like any other spider", ergo a) Linkscape has a spider, and b) since it is "just like any other spider" it must be distinct from them.You went out of your way to give the impression that this was data you guys collected yourselves, not data that other companies made available. And seriously, as someone trying to bill themselves as a master marketer, you honestly think anyone is going to believe that it is your standpoint that hyping up a company’s capabilities is "a very bizarre criteria on which to base a marketing plan"? That’s your assertion?

from randfish 2295 Days ago #

Votes: -5

My assertions are:1) That Linkscape’s crawl is exactly like that of other indices - built by crawling the web’s link structure.2) Linkscape has crawl sources generated by spiders - there’s no other way to get the data, so how is this a point of contention or argument? How else could we possibly even have the information?3) Linkscape’s index is unique - it is not a copy of any other data source. It was built by us for use in the tool and for our members’ competitive and self intelligence.I don’t even see why you’re calling it an "honest mistake" - what was the mistake? The original messaging was that we wouldn’t disclose our sources. Now we do disclose them. I can’t understand your insistence that we don’t have a unique index or crawl sources for our data. It’s right there. The quotes you copied are entirely accurate - "Linkscape crawls the web just like any other spider." It does! That’s how the index was compiled - with spider crawled data.

from mvandemar 2295 Days ago #

Votes: 1

Again, whatever. I know I have voiced my opinion before on you not being the brightest crayon in the box, but right now they way that you are trying to maintain that you are actually that dumb is, frankly, very embarrassing.You do not own the spiders that gathered the data that Linkscape uses, and you initially made a big play to try and get everyone to believe that you did. "They have spiders" is substantially different from "our spider", and you know it. That’s all there is to it.

from randfish 2295 Days ago #

Votes: -2

I don’t know how you can claim, after writing comments like this one above, that your attacks are not personally motivated and therefore at least a little biased and lacking in equitable treatment of my responses.I also take issue with your assertion about our sources and their ownership - we own the data in the index, we own the technology that built it, we’re responsible for the collection of the pages and metrics that go into the product. Your entire argument, despite prior protests, centers entirely around semantics and the nuances of how you perceived language. And, what’s stranger, I don’t even think there’s a way you’ve presented in which that perception was inaccurate - our data is our data and no one else’s. The spiders that collect data for us crawl the billions of pages that are in our index. The crawl is for Linkscape, with our design, our preferences, our depth and diversity priority.Accusing me of a lack of intellect or ability pushes this conversation into arenas that aren’t worth anyone’s time. If you’d like to maintain a low opinion of my mental faculties or skills, so be it. I certainly won’t suggest the same of you - you’re intelligent and from the many friends we have in common, I’ve even heard that you’re a nice guy, who just likes to stick it to people/companies on occasion (which is why, I presume, you run a blog dedicated to "smacking down"). I just don’t think you’ve got anything but your claim of mis-stated semantics to argue in this case, which is not particularly strong, doesn’t demean the value or capabilities of the product, and fails, at least in my mind, to prove your initial assertion that I lied in order to benefit myself or my company.It’s getting late (2am here in Seattle), so maybe we can call it a night.

from mvandemar 2295 Days ago #

Votes: 2

"we’re responsible for the collection of the pages"No, you’re not. You are acquiring data that other people collected. Continuing to assert otherwise does not make it true, and pretending not to know the difference isn’t helping your case.I do suggest that you get some rest, and come back and re-assess at a later time what you said here, because your arguments are not getting stronger as the night progresses... quite the opposite.Sleep well.

from randfish 2295 Days ago #

Votes: -5

Yes, we are. We are responsible for the collection of pages. You might not like the fact, but it’s a fact. I guess I can’t convince you more than I’ve already tried, so this is quickly devolving into a "yes we did," "no you didn’t," argument which can’t go anywhere positive. If you refuse to believe it, that’s fine, but I don’t think you have any proof, where as Linkscape has engineers who designed it, code they’ve written and an accessible index that anyone can see and test against.

from randfish 2295 Days ago #

Votes: -2

To help offer up some proof, let me put forward mozTrust - a metric we calculate based on a seed set of trusted domains and pages we designed. If the crawl was not our design and our responsibility, it would be impossible to construct or calculate this metric. The same goes for the domain preferencing and the crawl priorities, size, speed and depth. Maybe you’ve built a large scale web index in the past and thus know from experience that this is the case, but I suspect not.If you’d like, I can post a CSV with all the link data we’ve got on Smackdown.BlogsBlogsBlogs.com and you can compare against data from other indices. We’re showing:2,038 links from 194 unique domains. Some of the more important links (at least, according to our metrics) come from sites like:adseok.combad-neighborhood.combadijones.comekstreme.comDigg.comSearchEngineWatch.comMetafilter.comProBlogger.netMattCutts.com (most of these are nofollowed, but there’s one we see as followed)If there’s other data that would help, I can certainly share.

from mvandemar 2295 Days ago #

Votes: 1

Again, there is a distinct difference from having the data to analyze, and being the ones that went out and collected that data. I am not saying that you do not now currently have a large set of data in your possession, and there is no way whatsoever that my words could have been misinterpreted to that effect.

from g1smd 2294 Days ago #

Votes: 5

I finally get what Rand is saying: "Linkscape’s spiders reach billions of pages" as in "Linkscape’s data comes from a system that indexes billions of pages". So "Linkscape’s spiders", not "SEOmoz’s spiders". That doesn’t say who runs the spider/crawler, it just says what type of system the data comes from. <div></div><div></div><div>Yes, it doesn’t say "SEOmoz has a crawler that...". However, that was the assumption everyone had made. That’s a clever usage of words; and Rand blames the listener for their misunderstanding. I think when you say it out load, and change the intonation slightly, suddenly it becomes clear.</div><div></div><div></div><div>Whether it was a calculated effort to be "fuzzy", I have no idea. I asked fairly early on for the UA, as I wanted to make sure I was not blocking it, and i wanted to see how many pages of a site had been analysed before seeing what data had been extrapolated from that.</div>

from mvandemar 2294 Days ago #

Votes: 3

g1smd, there was nothing "fuzzy" about it. The help page had an image with these words on it:"Linkscapes bot crawls sites and pages on the web by following the links it discovers."It had a little picture of a robot, crawling from one link to another. It was very clearly labled "Linkscapes bot", not "the bots that the others guys use that we grab via their API’s and whatnot".Besides, I have a very sneaking suspicion that Rand is going to try another tactic now.

from DazzlinDonna 2294 Days ago #

Votes: 29

Rand, there’s no way you and Michael are ever going to be able to see eye to eye. There’s just too much history there, so it’s pointless for either of you to try. However, just because it may or may not be personal, it doesn’t necessarily mean Michael is wrong. If you didn’t lie, you have to at least take the blinders off and realize that many, many, many people "feel like you lied". It sure "seems like" a lie. It sure "feels like" we were deceived and duped. Even if your inside information is enough to convince you that you didn’t lie, you have to at least acknowledge that a whole bunch of people "feel like" it was a lie. It sure "felt like" you told us that you had a crawler. (And please, let’s not call a crawler an index). It sure "felt like" you were expounding upon how great it was that you had a crawler. Regardless of the inside information that your company won’t reveal, you most definitely did give impressions that now feel like lies.Some quotes from a page that has now been removed from seomoz’s site:"Linkscape’s spiders""Linkscape crawls the web just like any other spider, discovering links to new pages and following them.""Linkscapes bot crawls sites and pages on the web by following the links it discovers".It was precisely because of quotes like this that everyone asked for the spider’s name. It was precisely because of quotes like this that we believed you had your very own spider(s). No matter how the actual functionality now works that makes you "feel like" you have your very own spider, you don’t. Or if you do, then you are now making us even more confused, and more likely to wonder which of the many versions are lies.Had you originally said something like, "we have an index" but hadn’t said, "we have a spider", that would have been completely different. But you told us you had a spider(s). You made us wonder what the name of that spider was, so that we could block it if we chose. You then turned around and told us you wouldn’t give us a name, which made us a little antsy. Then you said you didn’t have a spider, you used a bunch of data from other places who had spiders of their own. Imagine how that would make people feel? What? You said you had one. You said we wouldn’t be given the name. Then you said you don’t have one. Now you say you do - sorta - cuz you use other people’s. C’mon. Put yourself in everyone else’s shoes. Does all of that not sound like there’s a lie somewhere in there? And if it "sounds like a lie", maybe you should at least acknowledge that there’s good reason for everyone to call it a lie? If it hadn’t been Michael to call it a lie, would that have made a difference?Just like in blunders of the past, if you’d only started out with reality, it would have gone down very differently. Why not just say, "hey ya’ll, we’ve got a huge index that we’ve compiled from various sources, and we have this really cool tool that pulls great info out of that index! Sweet, huh?" Everybody would have said, yeah, very cool, sign me up. (or something similar). And the thing, it’s obvious that you understand the mistake you made, because you changed the wording and/or deleted wording from posts/pages that now make "index" much more prominent than "spider".Look, I’d be willing to bet the tool is really great (though I haven’t tried it). But it sure started out on the wrong foot, and no matter what you want to believe, it all boils down to the fact that it "feels like" you lied. Whether or not you did really makes little difference at this point. It’s going to be very difficult to undo what you said, then unsaid, then said, then unsaid, etc. I, like many others, know what we read. We know what you said, and then later said. We know how that "feels", and for many of us, that feels like a lie. So, just thought you should hear that from someone other than Michael. BECAUSE I like you, in fact. Because I like you, you should hear that it’s even people who like you that "feel like" you lied. Thing is, I like Michael too. And say what you will about him, he only said what many others were thinking but didn’t have the balls to say out loud. Like it or not, you need to realize that.

from mvandemar 2294 Days ago #

Votes: -1

I just want to clear one thing up:Rand, there’s no way you and Michael are ever going to be able to see eye to eye. There’s just too much history there, so it’s pointless for either of you to try.For my part, that’s simply just not true. I’ve said it before in the past, and I’ll repeat it again... if Rand wants me to not call him on his bs, then don’t give me anything to call him on. If he stops lying then I’ll stop pointing it out. It’s a very simple concept really.The majority of the time that I visit moz is when someone else points me to something that was said doesn’t sound right to them. Sometimes I look at it and say, sure, that might be, and sometimes I think it’s bs but don’t really care that much. On a very few occasions I felt compelled to speak up because it was above average bs that was happening, such as with this case.I am not one of those people who believes that other people cannot mend their ways. Rand came on to this scene 4 years ago, and wound up getting banned from what was back then a large portion of the community for scamming people. I have seen nothing whatsoever to indicate to me that he actually has become an honest person since then, but if he does, great. At that point I’ll stop pointing out that he’s not. :)

from seanmag 2294 Days ago #

Votes: 9

@Rand - to your comment:I do hope, however, that the level of discourse can rise above petty vitriol and negative personal attacks to something more aspirational and more worthwhile.I think it’s fair to say that I’ve been a great advocate and supporter of SEOmoz. What’s disheartening to me is that you would make such dismissive comments like that simply because I held you to task. It was your complete dismissiveness of Mr. Vandemar’s post with a silly comment that led me to write to begin with. I really felt that you owed the search community at large an explanation for what many clearly feel was a breach of trust.Many companies withhold a great deal of information with regard to their strategies, tactics, sources, etc. I take no issue with that. I take issue, not for what you are holding back but rather what you said - what you "sold" to the community. I think there’s enough evidence to validate that you expounded far beyond a semantic error. Essentially, you’ve oversold your product. The fact is, most SEO tools pull from or leverage other sources. From my perspective, it was unnecessary and unwise to mislead people as to how the data was gathered and in this case, knowing the sources you are drawing from could be considered important for someone to determine whether they felt investing in your product was worthwhile. As for being personal - you’re the voice of your company, and with a small company especially, anything said - whether positive or negative, is going to come across as personal. I agree that you have a great team of people. However, I feel that just because that is the case doesn’t mean that I should be required to turn a blind eye when I see something that breaches the trust and respect you’ve earned.Forty two people voiced their opinion with a vote for this post prior to my comment. Clearly they saw some validity to the content. That is something that should be neither ignored nor dismissed.

from g1smd 2294 Days ago #

Votes: 0

<div>*** Linkscapes bot ***</div><div></div><div></div><div>Yes, "Linkscape’s bot" - the bot that collected the data to build the Linkscape product.</div><div></div><div></div><div>The Linkscape product is the data and the analysis.</div><div></div><div></div><div>"A" bot must have collected it, it didn’t just appear out of thin air.</div><div></div><div></div><div>Note that "Linkscape’s bot" is not interchangeable with "SEOmoz’s bot".</div><div></div><div></div><div>"SEOmoz’s bot" would imply a computer sat in the MOZplex, connected to the web, and pulling pages from sites all over the world to analyse.</div><div></div><div></div><div>"Linkscapes’s bot" only implies "the bot that collected the data" without saying who owns or where that bot is physically located. It could be owned by a third party and be "hired" by the Moz to collect specific data. In that case it is "Linkscape’s bot" the bot what made Linkscape, but it still isn’t "SEOmoz’s bot".</div><div></div><div></div><div>Yeah, it’s all semantics.</div>

from neyne 2294 Days ago #

Votes: 0

@gab "Launch and iterate is hardly an SEOmoz-only practice. Any business startup / product launch advice columnist/blogger etc will suggest that."Of course. I was not pointing out that. I was pointing out the second part of the sentence:and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare.This is the recurring motif i was talking about. Not the fact that they launched an iterated...

from neyne 2294 Days ago #

Votes: 1

g1smd i think that that is going into the words beyond reasonable expectation. Linkscape belongs to SEOMoz, it was built by people in SEOMoz, it is advertized by SEOMoz and the money that it earns goes into SEOMoz’s pockets. So when you say "Linkscape’s bot", people understand it (and rightly so) as SEOMoz’s bot. You don’t put that apostrophe before an "s" to mark that it doesn’t belong to Linkscape. Had the intention been to describe the bot as it really is, i guess a good way would be to say "bots that had gathered the data for Linkscape to analyze". Or something like that. Another thing that could have been done is, after people started asking for the bot identity, to say something like "hey guys, sorry if i got you to believe that it was our bot (or bot of our making) that has gathered the data, cause it wasn’t. the data was compiled from other places with their own bots." That is a way to put things in a non-vague way.There is no better explanation to this whole thing than what Donna had written. It is a pity that we are discussing the circumstances around the release of the tool and not the service that the tool provides. Rand, some of your products are great and valuable, i sometimes just wish you would make it easier for people to feel that way about them. Instead, and partly with your fault, we are discussing other issues around the tool.

from mvandemar 2294 Days ago #

Votes: 2

Yes, "Linkscape’s bot" - the bot that collected the data to build the Linkscape product.No. You cannot collectively refer to an assortment of bots owned individually by a variety of different organizations out there as a singular bot owned by a tool that is owned by exclusively by seomoz. That’s not semantics. That’s borrowing your dad’s Ferrari, and not only referring to it all night long to your date as "My Ferrari", but also boasting about how you paid cash for it.

from g1smd 2294 Days ago #

Votes: 0

Linkscape is a product. SEOmoz is a company. The possessive takes a subtle and different meaning.

from randfish 2294 Days ago #

Votes: 0

Sorry - had to get eventually get some sleep, but I’ll try to re-address all the new issues above.@Michael - on being the ones to collect the data. You’re suggesting that because the user agent name wasn’t "seomoz" or "linkscape," it’s not data collected by us. I reject that - we collected the data.@g1msd - we’re now revealing all the potential UA’s that power Linkscape’s data, which I believe was the request. It’s our intent to continue to list any source that might be part of that.@Michael - Again, I think it’s really picky to say that because the spiders that crawl don’t have the name "Linkscape," I lied, but if that’s the accusation and if it was misleading, you have my apologies. Now the UAs that crawl are all listed.@donna - We have our very own spiders and sources - they crawl just for us, on our design. They don’t have the name "Linkscape," but I think the disconnect ends there. We said we wouldn’t give out the names, but the community had a llot of impact on changing our minds and we decided to release them - they’re all listed, and if new ones come up, we’ll list them as well. If we had originally said "we don’t have spiders" that would have been an outright lie - and it would have suggested that there was no way to block us, which isn’t true. I guess if we had come out with the list of UAs right at the beginning, that might have helped, and I do apologize that it took us a week to get that in shape. We initially made the decision to be quiet about it, because we felt it would harm the product’s potential, but changed our minds when it became clear that the community (and a lot of people we really respect) felt that it was unacceptable to keep it hidden.And yes, I agree that the due to the conflicts between Michael and I in the past, it’s tough to address his concerns or view them as reasonable and fair, but that’s what I’ve tried to do in this thread. Reading over my responses, they probably come across as too defensive, but I feel a bit lost by both the level of antagonism and the parsing of words to make it seem as though there’s some direct, outright lie, which I don’t believe is the case. To be honest, I don’t even know why the guys pulled down the crawler page - it’s completely accurate - I think they just meant to replace it with the new sources page and didn’t consider that it would look different. I’ll ask that it gets put back up. There is, in my mind, no error in what it says:"Linkscape’s spiders" - referrs accurately to the sources listed."They crawl the web to build an index" - absolutely true"The bots crawl the web through the links they discover" - also absolutely trueThanks for the valuable input and for the level-headedness, and let me know if there’s more I can clear up.@Michael - I think your focus on historical issues doesn’t help to make this process any better. It only makes you appear more biased and antagonistic. We’ve been through all the previous disputes and every time there’s a new one, you bring up all the old ones to help make me look worse and to remind anyone reading that I’ve made mistakes in the past and that to you, that makes me untrustworthy in the present or future. "I am not one of those people who believes that other people cannot mend their ways." If that were true, this wouldn’t be your style or approach.@Sean - I’m sorry that you didn’t like my response to Michael’s post. I felt it was a ridiculous and inaccurate one - and I think I’ve done a solid job of refuting and explaining that in this thread many, many times.And you’re totally right - you’ve been a great supporter of SEOmoz and someone I always considered a friend until I saw that comment last week. I hope that we can mend fences and return to a place of mutual respect and support - you certainly have mine.On the charge of overselling the product - I strongly disagree. There is no claim - absolutely none - that the product does anything more than what it does. Reading through the help and edcuation center, including the crawler page, the blog post that launched it, the presentation I gave at SMX East on it, the radio interviews and the videos, there is simply no exaggeration. Linkscape’s crawl has 30 billion pages. The crawl is our design, executed to create this link graph that can be researched through the tool. We’ve never made any claims that the product does something or is something that it’s not. Give me any specifics here and I’ll be happy to compare them against what’s in the tool with data to back it up.On being dismissive - I think that’s an unfair charge. I came into this thread and responded for hours - many, many hours - up until 3:30am last night, and now again starting at 9:30am this morning. I did this because I don’t want to be dismissive. I want to address the concerns and the accusations and any misunderstandings.@neyne - I agree. It would be nice to talk more about what the tool does and how it does it. If there are any questions around that, I’m happy to address them as well.Michael - that analogy is completely inaccurate. The bots may not have the name Linkscape, but this doesn’t mean they don’t crawl for us and it doesn’t mean we didn’t pay for them (we most certainly did - our investors and bank accounts can attest to that).I’ll try to be in here more this morning to address more.

from mvandemar 2294 Days ago #

Votes: 1

@Michael - on being the ones to collect the data. You’re suggesting that because the user agent name wasn’t "seomoz" or "linkscape," it’s not data collected by us. I reject that - we collected the data.No I’m not. I suggesting that because they turned out to have names like Googlebot, Yahoo! Slurp, msnbot, Exalead, and Gigabot that you didn’t collect the data.You did NOT say "The bots crawl the web through the links they discover". What you said was, "Linkscapes bot crawls sites and pages on the web by following the links it discovers". Singular. One bot, owned by one company, and going back and trying to edit what you actually said does not make it true.

from randfish 2294 Days ago #

Votes: -4

I think that once again, you’re really playing at semantics. We did not initially reveal UAs - and we do now. Parsing a line attempting to explain the crawl process and accusing the singularity of a noun rather than a plural as being responsible for lies of the level you’re claiming is exactly what I’ve been suggesting is the problem the entire time - semantics.I think it just blows things way out of proportion to say that because we illustrated the crawl process with one bot, which we wouldn’t disclose, but actually have multiple sources, which we do now disclose is this huge lie that distorts the abilities or value of the product.

from mvandemar 2294 Days ago #

Votes: 4

No, Rand, and despite what you think your attempts at playing innocent aren’t helping you. Your entire initial sales pitch hyped the fact that your company went out and spidered 30 billion web pages. You weren’t "attempting to explain the crawl process", you were boasting of your feat.

from DazzlinDonna 2294 Days ago #

Votes: 11

Rand, it may be semantics, but sometimes semantics matter. If I said "Donna’s dog ate Rand’s cat", you’d assume the dog was mine and the cat was yours. You wouldn’t assume that the dog was one I borrowed from someone else (or rented) nor would you assume the cat was one you kidnapped from a kid walking across the street. In reality, both of those things might be true, so if someone later said I lied because the dog wasn’t mine, and the cat wasn’t yours, I could theoretically claim that I didn’t lie, because by renting the dog, it was "mine", and by kidnapping the cat, it was yours. Semantics might play in my favor, true, but it still wouldn’t sit right with a lot of people. So even if by paying for the use of other people’s bots, they can be claimed as "yours", it still "feels like" we were duped from the get-go. Again, the tool is probably awesome, and we should all probably let this go (cuz everyone will in time anyway), but sometimes, semantics do matter. (I never inhaled). :)

from eKstreme 2294 Days ago #

Votes: 6

Rand, you did NOT disclose a UA. A user agent is defined in the HTTP protocol (e.g. version HTTP 1.1 at http://cli.gs/QqayjG ). You disclosed a meta tag that only an HTML parser can read. These are very different things and it’s not a semantic difference.As for a crawler, I agree with the comments above that the wording suggested there is a spider that SEOmoz controls, but you know that I didn’t think that’s true (you saw my blog post). The way I understand LS to work is that SEOmoz gets hold of downloaded pages from around the web, say an index of cached pages commercially or freely available. Those are then parsed on SEOmoz computers and the link graph calculated on SEOmoz computers. Right or wrong? If wrong, what’s the exact method of operation? Again, a parser and a crawler are very different things that are not just semantically different.As for any suggestions that SEOmoz, or you personally as Rand, were driven by any "malicious" motives (for the lack of a better word), I think that’s just plain wrong. I think as a marketer you could have handled this better but I don’t see any evidence of malice.Cheers,Pierre

from Jill 2294 Days ago #

Votes: 9

I’ve read this entire thread, and the one thing that stands out for me (besides Donna’s EXCELLENT post) is Rand having said this:"...the more I’ve found that honesty (and especially transparency) does come into conflict with promises that have been made and the needs for stealth and protection." Transparency and honesty are not synonymous. Of course you, your company or any business does not need to be transparent in everything you do. That would be silly, imo.But honesty, in my opinion, should be the foundation for every business and for every person of integrity. Anything less will get found out and will create more problems then simply being honest from the outset.While I can’t say whether or not Rand has been completely honest on this crawler issue or others (and quite frankly, I don’t really care), his statement here that honesty comes into conflict with other things is one I would not have expected to hear.

from randfish 2294 Days ago #

Votes: -2

@Michael - If we boasted, I think it was a fair boast, since all of the work we claimed has been accomplished by the people in SEOmoz’s offices and on our payroll. That index was built by us for use in the Linkscape tool - your continual attempt to deny it doesn’t make it untrue. It’s a unique data source. We designed it, and we are responsible for the how, when, what, where and why it was built.@Donna - The semantics can matter, but I think in this case, Michael’s just desperate to try to show some massive fraud that was perpetrated, when that’s simply not true. If I say "Donna’s dog knocked over the trash" and it turns out Donna has more than one dog involved in the trash knocking over process, that might mean that I wasn’t telling the whole truth, but it doesn’t mean I was lying. And I admitted, as soon as we revealed our sources, that we hadn’t been telling the whole truth. In fact, we admitted when we launched that we weren’t revealing the whole truth - that we were holding back on disclosing the source. Now that we’ve come out with it, and the source is against expectations (in that we may pull from one or more of the datasets listed), it’s being taken to mean there was a perpetration of fraud.I reject that notion. I accept that we were not fully truthful, and in fact, we still have lots of information about the index and the data that we’re keeping under our hats, but I maintain that a lack of disclosure is not a lie and that no claims were made about the products capabilities or value or functionality that were false.

from seanmag 2294 Days ago #

Votes: 3

@dazzlindonna - Actually, after reading this thread, if you had said to me "Donna’s dog ate Rand’s cat", I would think that you had a pack of undisclosed dogs, one of which may or may not have been yours and that that undisclosed dog may or may not have been involved in eating Rand’s cat. Further, that Rand’s cat may in fact have been a litter of harmless kittens and as such, you should be found guilty of cruelty to animals and spend the remainder of your life in prison.No harm intended. You understand, yes? :)(Ok, that was my effort to bring some levity and hopefully closure, to a thread that’s gone way beyond the call of duty.)and for your reading pleasure: http://www.slate.com/id/1000162/

from randfish 2294 Days ago #

Votes: -3

@Pierre - We disclosed all the sources, and each of them have UAs that can be blocked. Linkscape doesn’t use data that’s freely available (although from a financial perspective, I wish that were the case), and the index is completely unique, designed by us and built with Linkscape alone in mind. Spiders with UAs that we’ve disclosed collected the URLs to our specifications (as I pointed out above, it could not have been otherwise or we wouldn’t have been able to build the metrics or design the crawl structure, refresh rate, diversity, etc.).On the malicious motives - thanks. I guess that it is true that our motives for not revealing the UAs was based on a desire not to be blocked or restricted, so some might view that as malicious. I think we’re doing the right thing now, both by revealing sources and by allowing anyone to remove a page exclusively from SEOmoz with the meta seomoz noindex tag.@Jill - I guess I should have been more clear "full honesty" or "complete honesty" which would require complete disclosure and transparency are what comes into conflict with running a business and growing to a larger scale. I think that as this thread has illustrated, we were not fully transparent (and still are not in some areas), but I believe we haev been completely honest - certainly in spirit, even if a few semantics were off.

from Jill 2294 Days ago #

Votes: 1

Rand, might I suggest that you step away from the computer today? (If only to not make Sarah’s job even harder than it is already going to be on Monday morning!)

from DazzlinDonna 2294 Days ago #

Votes: 2

@seanmag - LOL! good one!@Rand - I guess we’ll just have to agree to disagree on some things. Your perception is different than others. That’s what makes the world go round, and leads to a lot of disagreements.And on that note, I’m gonna go play spider solitaire.

from richardbaxterseo 2294 Days ago #

Votes: 2

My 2c as follows. Anyone can write a crawler. Ok, it’s NOT that easy but fundamentally - a piece of software that runs on a server that fetches data and stores it somewhere. It’s the storage, analysis and recall of the data into meaningful reporting that is impressive. All of that number crunching requires some serious equipment to run on, serious development to create and perfect and serious money to make happen. I personally don’t care if Linkscape uses lots of external data sources with or without their own crawl becuase I appreciate the additional tool at my disposal and I appreciate the level of work involved to get those reports to make any real world sense. There are some interesting similarities between Linkscapes metric results and Google serp positioning. It’s a nice algo!

from randfish 2294 Days ago #

Votes: -1

Jill - I was accused of not caring about this problem and not addressing the concerns or charges leveled, and I think that if I’m going to put in this much effort, I should probably continue to make sure there’s no other questions left unanswered. Admittedly, with my 6 hours of sleep, I might be a bit more sensitive than usual (although, I’m pretty sensitive in general, and let me highly recommend to any other startup CEOs out there not having that trait or limiting it as much as possible), but hopefully I can still honestly and forthrightly present our case.Regarding Sarah’s job - I assume you’re referring to our in-house counsel/COO? Have I said something that violates the law or suggested that we’re doing something that’s illegal?

from evilgreenmonkey 2294 Days ago #

Votes: 2

I think that the comments on this story are getting out of hand and going around in circles. I’ll be closing comments as neither party are likely to back down.

from greatscott 2028 Days ago #

Votes: -3

Our bots, our crawl, our index...hand-built from the ground up. Mr. Vandemar’s accusations are simply and completely untrue.

Upcoming Conferences

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.

SMX Web Site » | SMX Difference » | SMX News »

Join us at an upcoming SMX event:

SMX West - March 3-5, 2015
SMX Munich - March 17-18, 2015
SMX London - May 2015
SMX Sydney - May 2015
SMX Advanced - June 3-4, 2015
SMX Paris - June 2015
SMX East - Sept. 29 - Oct. 1, 2015
SMX Israel - November 2015
SMX Beijing - December 2015