
Published: Oct 17, 2008 - 02:35 pm
Story Found By: mvandemar 4478 Days ago
Category: SEM

Search Engine Land produces SMX, the Search Marketing Expo conference series. SMX events deliver the most comprehensive educational and networking experiences - whether you're just starting in search marketing or you're a seasoned expert.
Join us at an upcoming SMX event:
Learn more about search marketing with our free online webcasts and webinars from our sister site, Digital Marketing Depot. Upcoming online events include:
Comments
First, let me say that Im surprised to be the first person commenting on this thread, considering there are 42 Sphinns. Clearly, if Mr. Vandemars harsh accusations are true, (he basically calls Rand and others in his company liars), this is a SEVERE BREACH OF TRUST of the entire SEO community by a self proclaimed honest and forthright guy - Rand Fishkin.In the post, Mr. Vandemar makes no bones about his accusation that Rand Fishkin and others within SEOmoz, propagated a series of lies and mistruths, all in an effort to dupe the SEO community out of more money. In other words he implies that they are selling the proverbial Snake Oil.Just one week ago, despite the fact that I didnt suggest Rand was being dishonest, he had this to say to a general comment I left at Marketing Pilgrim at the following url http://zi.ma/270511<i>"Ouch Sean. You really think that I would write nice things about a session dishonestly due to the corrupting influence of our sponsorship of the event or my friendship with Danny? That hurts."</i>Rand - let me say this - Based on all of the pre-release promotion you did with regard to LinkScape, and after reading every piece of commentary I could find post release, Ive come to the conclusion that you would say something dishonest if you felt it was in your best interests. I dont know how, after reading everything I have in relation to LinkScape, I could conclude otherwise.I also feel you violated a trademark of LBI-Netrank in using the the name LinkScape, as pointed out by SEOmoz community member Richard Manely. http://zi.ma/d68beaInterestingly, this accusation was only responded to publicly by your employee Scott, who stated:<i>"We did check it at the outset and during planning (were not that careless) and saw nothing that would be an impediment."</i>I find this comment shocking and hard to swallow since a simple Google search of "linkscape" would have returned the LBI-Netrank trademark.When I consider everything Ive read I can only conclude that you did mislead the SEO community and only when it was apparent that the truth would be found out, did you begin to "come clean". While your approach may not have any bearing on the value of the tool, it does demonstrate a conscious effort to misrepresent your product.Further, I find your response, to Mr. Vandermars following statement on this post, to be severly lacking.Mr. Vandemar:<i>I have to admit, Rand, it’s pretty bold to basically admit this late in the game that you guys lied through your teeth and grossly misrepresented the facts, just so you could appear to have accomplished a much bigger task than you actually did, all in the name of getting more money from webmasters. That’s a much bigger admission than saying you cloaked your bot, if you ask me. Gratz on coming clean.</i>Rand:<i>Sorry about that - obviously, it was supposed to be Gigablast (lousy typos). Gagablast sounds like some sort of infant vomit <img class="wp-smiley" src="</i>Personally, I feel you owe the SEO community and your own community of thousands, a much better response than that. If I were to have purchased your service under the impression that you had created your own bot/s, and had effectively crawled and indexed the web to the tune of 30 billion plus pages, and if I was led to believe that SEOmoz would be able to continually update and thus maintain the relevancy of this data, only to find out that it is actually an aggregated mashup of other companys data - I would be pretty upset.
For some reason, the tail end of my comment was left off, but I think its worth re-stating:If I had purchased this service under the pretense of SEOmoz having created their own bot/s, and having effectively crawled and indexed the internet to the tune of 30 billion plus pages, leaving me with the impression that SEOmoz could, at will, continue to update and keep the information relevant, only to eventually find out after the fact that the information is an aggregate mashup of data from a wide range of providers - Id be pretty disappointed.
It is funny how this keeps happening to Rand. Somehow, there is always something that needs post-launch fixing and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare. This was the case with Martys post and with the directory outingNot to talk about launching a tool promising to be one of the most revolutionary and useful tools for SEOs on the web and then leaving for SMX, without anyone (except for ppl responding in the blog comments) to respond,like immediately, to any issue that arises. It kinda hints that no one has even thought that something about the tool could be problematic.@seanmag. I guess that a tool should be judged by the quality of the information provided, not by the method the data was achieved. I mean, if Linkscape provides me with more incoming link data and analysis that I could have gotten elsewhere, then there is value in their "mashup". I do see a problem with presenting the tool as a based on web crawling as its selling point, only to turn out that the crawling was done mainly on offline databases
@neyne, exactly. If something like this happened once or twice, Id give them the benefit of the doubt but its become a pattern of behavior with SEOmoz.1) Controversy2) Benefit from attention3) Apologize but due to time cant fix it just yet.4) Continue to gain attention.5) Finally fix it and explain how aw shucks guys, I didnt mean it like THAT. 6) Ask people to talk to you about it in person rather than writing about it in public.I dont think Rand is a bad guy and he may not even see that this is a pattern but it certainly has become one.
I was certainly under the impression that they had their own crawler....
@skitzzo I would like to personally thank you for using the word "shucks". It is an excellent word that i have completely forgotten about :)
Its possible its all just a coldly calculated marketing ploy designed to keep people talking about SEOMoz and Linkscape, you still are, and its way cheaper than advertising.If in fact its deliberate marketing tactics, then hes working way above the heads of most of the marketers in this industry and you should all be taking notes.So either its an inadvertent trend that just happened as some people pointed out or strategically planned marketing designed to be socially viral, either way its working.
Sigh... I wish I could comment, but the decisions made internally are firm. The information thats been released so far is the extent of what were offering right now.I will say that we do have our own index of ~30 billion pages. Were updating every month, and should almost double that come November. Our index does bias towards domain diversity (hence, while Microsoft says they keep about 72 million domains in their index, we have ~150 million), but we dont have the same depth as the engines, especially on very large, deep sites (Amazon, Wikipedia, etc.). Were certainly hoping that in the months to come (and as the product moves out of beta), it will be a more close approximation of Google/Yahoo!/etc. but even now, if you do a lot of link research in there, youll find plenty of cases where we report very similar or even a few more links than Yahoo!s Site Explorer.As far as this - "Ive come to the conclusion that you would say something dishonest if you felt it was in your best interests" - I dont know. Maybe I should just stop trying to fight this perception. So many people tell me I should just ignore it. Maybe I should listen.SEOmoz is, in my opinion a good company, with lots of great, hardworking, talented people. Theyre honest, committed and, at least in my opinion, dedicated to a set of impressive ethical ideals. Linkscape itself was built because we wanted more disclosure on what the engines did, and felt that as a community, we deserved to have a resource from which to obtain it. You might argue that, since were not a non-profit, and took investment, the free data we provide isnt enough and the paid data is too expensive, but I see so many companies in this field and others who never try to give back to the extent that we do, and I guess is just saddens me to see this devolve into accusations of dishonesty and greed.So - sorry we cant give full disclosure about everything. Anyone who knows me knows that in an ideal world, that would be my desire. I do hope, however, that the level of discourse can rise above petty vitriol and negative personal attacks to something more aspirational and more worthwhile.
Rand, seriously, what school did you go to that teaches that playing the wounded bird and professing honesty will beat out cut and dried facts? You very clearly introduced your tool with the impression that SEOmoz was a company capable of crawling the web on their own and building an index of 30 billion webpages. Obviously you did so because you didnt feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant. The fact of the matter is, thats the path you decided to take. Once you got in a bind over the fact that if you had done that then you did it with cloaked bots, you decided to release where you actually got the information... and it wasnt from the way that you presented it. There is no gray area here. There is no secret that you can claim you cant tell us that would make the lie you told not be a lie. Cmon now.
@neyne said: "It is funny how this keeps happening to Rand."Not funny. I suggest its a reflection of Rands judgement; how he makes his decisions, in other words how he behaves. And its good to see, because you can pretty much bank on people behaving in the future the way they have behaved in the past. Another great clue useful for predicting future behavior is the quality and character of the people someone chooses to be around, hire, or otherwise engage in business with.... which is one reason I pay attention to such things. Hands down, with virtually no exceptions, organizational behavior is a tail wagged by the dogs at the top. The most interesting for me throughout all of this years SEOMOZ reputation nightmare is that bit Rand added at the end of his comments above (and other SEOMOZ fanboys have added to other discussions elsewhere) -- that this conversation is "petty vitriol and negative personal attacks". As before, consistent behavior -- if you dont like something Rand does, no matter what the facts, youll be accused of making "personal attacks". Given that is apparently how Rand and his fans behave when questioned, I know what will be said of this.. my unsolicited comment on this thread. Must be sour grapes, right? Something personal between me and Rand? Sorry.. there is nothing personal between Rand and I.
Good grief everyone, if you consider the entire body of work Rand has contributed freely to the SEM community over the years, his genuine and generous contributions are easily among the most useful ever made. Whether or not you like the way LS was rolled out, or whether or not you like Rand period, rather than piling on why not give the man the opportunity to do what hes always done. Sure things could have been done differently, but he will make it right because my gut tells me he wont be happy inside until he does. Hes a guy who wants us to like what he does, and thats a rare and good thing nowadays. When I started link building I think Rand was 14 years old. To watch what hes done and what hes freely shared is amazing. I want guys like him nurtured and funded, because I want to see what he does next. Ill take a few bumps along the way because the ride is worth it.
There have been a few times when Ive seen people call Danny Sullivan out, usually unfairly (but, then again, Im pretty biased when it comes to him), in discussions like these, and I always liked the way he responded to all the criticisms one by one, so Im going to give that format a try.On the accusation that we dont have our own web crawl - patently false. Go use the tool, note how the information is different than any other publicly available source. Linkscape runs on a completely unique index we built from scratch. It would be completely impossible to construct the link graph metrics without it. In fact, it would be impossible to extract many of the data points weve shared about size, diversity, coverage, use of links, etc.On the accusation that the construction of the tool is unethical or unwarranted - I believe very strongly that the information the search engines have at their disposal, in particular, the link graph, should be available to everyone who wants it.On the charge that its unfair or unreasonable to charge for some of this information - I disagree. I think that without a capitalistic model, innovations like these wouldnt be possible. Would it be nice if we gave all the data away for free? Yes. But it costs an incredible amount to build something like this, and takes tremendous time and energy from a lot of smart, talented people who are not independently wealthy. They deserve to reap the rewards of their labor.On the Linkscape copyright - we have an in-house attorney who is extremely careful about all this stuff. I dont know nearly enough about it, but I know theyve been in touch, and the discussions have been positive. I think its unfair to criticize when youre not the offended party and dont know what the situation is.About dishonesty to help myself or my company - I guess its something I hope I wouldnt do, but the bigger the business has grown, the more Ive found that honesty (and especially transparency) does come into conflict with promises that have been made and the needs for stealth and protection. I will always try to do my best on this issue, but I think its a fair critique to level. I wish there were more I could do, but so far, its been a tough path to navigate, and I doubt that as we get bigger it will get easier, unfortunately.Regarding the "duped" into buying Linkscape - Sean, your arguments seem to center around the idea that we wont live up to the promises of growing the index and keeping things fresh. If thats ever the case, you can always get a refund, first off, and second, I reject the notion that we wont live up to this promise - we will. Were already on track, as I said, to nearly double the index size in November, and to continue to get bigger and fresher thereafter.On the "pattern of behavior" - I think its a very accurate assessment of me and how I operate. Im frequently very busy, nearly to the point of being overwhelmed, and most of the other folks at SEOmoz are in a similar situation. Thus, youll see that responses arent always extremely quick. We also havent been and apparently are not good at predicting what will raise the ire of certain folks in the community - sometimes when we do think something will cause a stir, nothing and then when we think people will react positively, it turns ugly. Again, its something well try to work on, but I dont know how far well get. Our focus and our energy are always going to be primarily centered around building our tools, content and resources. Im not sure that well ever find that theres ROI in being hyper-sensitive or hyper-responsive (but we will try harder).Coldly calculated marketing ploy - if it was, it would be a pretty bad one. Negative attention is something that really messes with me and with all of us. Were a surprisingly (or maybe not so) sensitive group, so keeping everyone happy with what we do is a big goal, even if its not something weve always achieved. I know that many people give us the advice that we cant please everyone unless were mediocre and boring, and that the anger and attacks just show that were on the right path. Ive never been able to fully buy into that argument, and so I keep trying to have it both ways. Maybe thats foolish, but well probably keep trying.On Michaels accusation that people would only buy Linkscape because they felt bad for all the work we put in - that seems very incongruous and illogical. While Id certainly appreciate the gesture, Id seriously hope that it wouldnt matter whether it took us ten minutes or ten months, the judgement would come based on the value of the tool and how they could use it to make themselves better marketers and SEOs. To buy for any other reason seems tremendously unwise.Regarding Johns accusation that my judgement is poor - it could be. I obviously try hard to make the right decisions, but as a CEO, I make dozens every day and hundreds every week. Whether all of those are the best they could be and always right isnt up for discussion - with so many to consider, its impossible that Id always be perfect. I dont think anyone is. My job is to make enough "right" decisions to help grow the company, keep our investors, our employees and our customers happy. To date, I think Ive had some degree of success, but I know that I havent generally (or often) made John happy. However, I do think youre being a bit untruthful when you say its not personal. Go and re-read the many things youve written about me and about SEOmoz - its personal. Thats OK - I think I might have been the one to make it personal when I got angry about an attack you made on Danny in the past. Lets just not try to pretend that its not, though.I think thats just about everything. Lets see if this style actually works... :-|
@Mike Vandemar: You statement assumes that other people are as technically capable as you are or that they are interested in replicating what SEOmoz have done: "Obviously you did so because you didnt feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant."Personally, I could care less about assembling the team and hardware required for what Linkscape originally appeared to be OR for what it has turned out to be based on. Of course, it was quite misleading as you highlight to talk about crawling so much, since it gives the impression that the data was first-party data. My point is just that at the end of the data, the practical implications are not that significant. For your average search shop, you can just go in a la Dan Thies, run some reports and compare the data to Yahoos. If its better, use it. If not, drop it. @Neyne - "Somehow, there is always something that needs post-launch fixing and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare. " --> Launch and iterate is hardly an SEOmoz-only practice. Any business startup / product launch advice columnist/blogger etc will suggest that."This was the case with Martys post and with the directory outing." --> Two screwups Im sure they regret. On a related note, I personally have said things online that were interpreted incorrectly and by the time I found out about the problem, my original comments were deleted having been deemed offensive. You can imagine the difficulty I had making it up to the offended parties.Seans highlighting the trademark issue is perhaps the most legitimate, and cutting criticism of SEOmozs behaviour here. If LBIs product was around first, I dont see how SEOmoz can maintain their products name, frankly. The potential for confusion, especially with both products being in the same online marketing industry, seems pretty significant. Of course, this is just my humble 2 cents... not a legal opinion.Finally, If you meet Rand et co. personally, youll see that theyre really nice, friendly people. While much of the criticism (e.g. stealing bandwidth and the TM issue) is legitimate, there is an undercurrent of ad hominem attacks or else an overt current to it ("quality and character of the people someone chooses to be around, hire, or otherwise engage in business with"). I find that childish and something Sphinn can do without.@John Andrews - since I singled out your comment, let me say that you know from my past comments on your blog that I have tremendous respect and appreciation for your expertise. Youre one of the brightest guys in online marketing, no doubt. But when you get into the stuff about peoples personalities, those are personal attacks - even if you deny it or call me a fanboy - and they detract from your personal brand. SEOmozs boss and staff are human, and theyre entitled to make mistakes too. (As to SEOmoz not learning from their past mistakes, well thats obviously something they need to get an outside hand on.)
@Sean - Scott, as you know, isnt their inhouse counsel. Perhaps Sarah Bird didnt check it herself, or didnt check thoroughly.
@Gab - Sarah generally advises me not to talk publicly about legal situations, but I promise that she did her homework very carefully. One major difference is the country of registration for the trademarks - UK vs. US - but there very likely are other things in the equation as well.
Rand, I made exactly one accusation, and tossing in a bunch of bs that I never said isnt helping your case.A crawler is a web spider that goes out and collects webpages. You dont have one, you said you did. You obtained the data from publicly available sources, some free, others you bought. Thats what I blogged about. I didnt come to that conclusion combing through secret govt. records or by hacking your email, I did it by reading your blog. Its right there in black and white for anyone to see. You said one thing at one point, and another later on, and the two things conflict. End of story.Onward to your new bs... "On Michaels accusation that people would only buy Linkscape because they felt bad for all the work we put in" - please show where I ever said that. I know you love to obfuscate things, but I never even said anything even close to that, nor have I made any of the other accusations you mention.
Michael - up above, you just said:"You very clearly introduced your tool with the impression that SEOmoz was a company capable of crawling the web on their own and building an index of 30 billion webpages. Obviously you did so because you didnt feel that people would pay for something that could be built without being able to gather all of the data themselves, but honestly your reasons for doing it are irrelevant."I believe the exact intuition there is that we hoped to "impress" not with the value of the tool, but with the quantity of effort. Maybe that wasnt your intent and I misread it - if so, my apologies, but if its not a point I needed to refute, then let me know if there is a valid one I should be addressing.As for the crawler - the decision has been made not to reveal any more information than we already have on that front (http://www.seomoz.org/linkscape/help/sources), but regardless, the crawl itself (aka index) is unique and built, 100% by us. No other organization has access to it and no one but SEOmoz is responsible for its creation. I said we built a crawl of the web, and we did, and anyone can see it by using the tool. I recognize that Im not being completely transparent (and that its a conscious decision and one that, perhaps, will receive warranted criticism), but I fail to see how its a "lie."
@Rand "unfair or unreasonable to charge for some of this information"Its unfair and unreasonable to charge a site owner for his own information.Even Majestic12/MajesticSEO got that right, you give something back to the community so the community has a vested interest in helping your project instead of being a one-way leech.If youre going to ask to crawl someones site you need to give something in return.Otherwise people will lock your ass out and your investment will turn into a steaming pile.
You did do it to impress, I stand by that. I never said you did it so people would "feel bad for all the work" you put in. A peacock fluffs out its feathers to make itself look more important... not so others will "feel bad" about the effort they made.<div>And, nice try, but a crawl is not synonymous with index, especially when you start tossing in phrases like "Our process for crawling the web...", or giving the bullshit impression that you prioritize which sites/pages you will crawl next (or more often) based on how important you think those pages are.</div>
@Incredibill - a fair critique. We will definitely talk about providing site owners the ability to verify their own domain and see data related to their sites. I think thats a very reasonable request and would even be a good move for the product on the marketing side.@Michael - If you stand by your critique of us trying to impress, then Im going to stand by my refutation that this would be a very bizarre criteria on which to base a marketing plan and an even worse reason to make a purchase. Judge Linkscape (or any product) on its merits, not on what you perceive the construction effort to be.On the second point - I believe a "crawl" would be, semantically, very much the same as an "index" though technically, an "index" would take the raw data from the crawling process and use it to build structured information (and we have both of these). I think your issue is with the perceived deception that there was a single Linkscape bot collecting the initial data - and again, all I can say is that all the stuff were willing to say on that subject publicly is out there now. I think youre attempting to accuse based on semantics, and it feels, as previous attacks have, very personal and very biased. I do think you have a legitimate case to complain about plenty of things about Linkscape and SEOmoz and my reactions - Im trying the best I can, but I know I make mistakes and I can even be an ass at times (and if I have, I apologize), but I dont think this singular issue is worthy of the magnitude you profess.
Rand, whatever. "Our process for crawling the web" and "were crawling everything we can" are not ambiguous statements, no matter how badly you wished they were at this moment.You are right on one point though... it is all publicly out there now. Youre banking now on the fact that most people wont go sift through the data to see where you are bullshitting, and that by Monday no one will care. But the bottom line is that for those who want to read them, your words do in fact speak for themselves, in all their self contradicting glory.As to the "personal attack" defense, I think pops of SEORefugee said it best... its hard to accuse someone of lying and not have it sound personal.
@Rand To be fair, you did actually say you built a crawler not a crawl. There used to be a section in the LinkScape help section entitled "Linkscape’s Crawler" which has now been removed but remains in the Google cache. Michael and Seans point there seems valid.
Fair enough - I think the style of your blog post and the history make it even harder not to be taken as personal, but thats a fair point about how difficult it can be to make the accusation without it being personal.As far as crawling goes, Ill just say that those statements are not dishonest or misleading. Our crawl comes from sources we control and therefore we have both processes and biases for what information goes into the index Linkscape produces and how. We bias towards domain diversity, we use mozRank as a metric to determine how deep the information from a particular site is and we make determinations about what to include next and what needs re-visiting sooner vs. later.And one more thing - SNL just now airing on the west coast and OMG Tina Fey/Sarah Palin thing is hilarious. Macgruber!!
Rand, you honestly think this is a factual statement?Over the course of several weeks, Linkscapes spiders reach many billions of unique URLs on many millions of domains, which is similar to various search engines crawl rate... Linkscape crawls the web just like any other spider, discovering links to new pages and following them.As to the "history" making things seem more personal... I can honestly say that I have never resorted to making things up about you in order to make you look bad, while that is indeed something you have done to me in the past.
It is a factual statement. The crawl were showing was collected between August and September for the October release, so I think that fits very fairly in with the "several weeks" assertion. And Linkscapes data is not substantively different from that of any other crawl/index - just play around with the tool and youll see the link data matching up really solidly to whats listed in Yahoo! Site Explorer or Googles Webmaster Tools.On the history account - I think thats a dead end. Were just not going to agree there.
Dude, Linkscapes spiders dont reach many billions of pages because Linkscapes spiders dont exist. This isnt semantics, its fantasy. Even if you try and make the claim that you do have additional spiders of your own that you are disguising the user agent on, those spiders arent reaching "billions of unique URLs". Also, I would think very, very carefully if your planned defense to this is going to rely on some as of yet undisclosed spider running around that you refuse to let identify itself. Seriously.
@NickWilsdon - A very fair point - we did say we built a crawler, and thats something I stand by, because its true. More detail I cant provide, but it is an accurate statement. Sorry I missed your comment earlier - I think I was composing while it was posting.@Michael - Weve revealed all the sources we use - they have spiders, they reach billions of pages - those billions of pages are in the index. It sounds like your complaint is that theres no spider called "Linkscape" but I think that would be immediately apparent to anyone whos seen their sites/pages in the tool, since no bot Im aware of crawls the web with that user agent. We also disclosed when we first wrote about the service that there was no bot named Linkscape or SEOmoz, which is what aroused ire that it couldnt be blocked.Im not suggesting we have any bots that disguise the user agent name - lets not start any more inaccurate rumors. If you were describing the situation, wouldnt you say that "Linkscapes spiders reach billions of pages" since it is entirely true that, well, Linkscapes spiders reach billions of pages?
No, its not even close to true, and no, were not talking semantics, and no, you are not going to pull it off that this was some sort of honest mistake."Linkscape crawls the web just like any other spider", ergo a) Linkscape has a spider, and b) since it is "just like any other spider" it must be distinct from them.You went out of your way to give the impression that this was data you guys collected yourselves, not data that other companies made available. And seriously, as someone trying to bill themselves as a master marketer, you honestly think anyone is going to believe that it is your standpoint that hyping up a companys capabilities is "a very bizarre criteria on which to base a marketing plan"? Thats your assertion?
My assertions are:1) That Linkscapes crawl is exactly like that of other indices - built by crawling the webs link structure.2) Linkscape has crawl sources generated by spiders - theres no other way to get the data, so how is this a point of contention or argument? How else could we possibly even have the information?3) Linkscapes index is unique - it is not a copy of any other data source. It was built by us for use in the tool and for our members competitive and self intelligence.I dont even see why youre calling it an "honest mistake" - what was the mistake? The original messaging was that we wouldnt disclose our sources. Now we do disclose them. I cant understand your insistence that we dont have a unique index or crawl sources for our data. Its right there. The quotes you copied are entirely accurate - "Linkscape crawls the web just like any other spider." It does! Thats how the index was compiled - with spider crawled data.
Again, whatever. I know I have voiced my opinion before on you not being the brightest crayon in the box, but right now they way that you are trying to maintain that you are actually that dumb is, frankly, very embarrassing.You do not own the spiders that gathered the data that Linkscape uses, and you initially made a big play to try and get everyone to believe that you did. "They have spiders" is substantially different from "our spider", and you know it. Thats all there is to it.
I dont know how you can claim, after writing comments like this one above, that your attacks are not personally motivated and therefore at least a little biased and lacking in equitable treatment of my responses.I also take issue with your assertion about our sources and their ownership - we own the data in the index, we own the technology that built it, were responsible for the collection of the pages and metrics that go into the product. Your entire argument, despite prior protests, centers entirely around semantics and the nuances of how you perceived language. And, whats stranger, I dont even think theres a way youve presented in which that perception was inaccurate - our data is our data and no one elses. The spiders that collect data for us crawl the billions of pages that are in our index. The crawl is for Linkscape, with our design, our preferences, our depth and diversity priority.Accusing me of a lack of intellect or ability pushes this conversation into arenas that arent worth anyones time. If youd like to maintain a low opinion of my mental faculties or skills, so be it. I certainly wont suggest the same of you - youre intelligent and from the many friends we have in common, Ive even heard that youre a nice guy, who just likes to stick it to people/companies on occasion (which is why, I presume, you run a blog dedicated to "smacking down"). I just dont think youve got anything but your claim of mis-stated semantics to argue in this case, which is not particularly strong, doesnt demean the value or capabilities of the product, and fails, at least in my mind, to prove your initial assertion that I lied in order to benefit myself or my company.Its getting late (2am here in Seattle), so maybe we can call it a night.
"were responsible for the collection of the pages"No, youre not. You are acquiring data that other people collected. Continuing to assert otherwise does not make it true, and pretending not to know the difference isnt helping your case.I do suggest that you get some rest, and come back and re-assess at a later time what you said here, because your arguments are not getting stronger as the night progresses... quite the opposite.Sleep well.
Yes, we are. We are responsible for the collection of pages. You might not like the fact, but its a fact. I guess I cant convince you more than Ive already tried, so this is quickly devolving into a "yes we did," "no you didnt," argument which cant go anywhere positive. If you refuse to believe it, thats fine, but I dont think you have any proof, where as Linkscape has engineers who designed it, code theyve written and an accessible index that anyone can see and test against.
To help offer up some proof, let me put forward mozTrust - a metric we calculate based on a seed set of trusted domains and pages we designed. If the crawl was not our design and our responsibility, it would be impossible to construct or calculate this metric. The same goes for the domain preferencing and the crawl priorities, size, speed and depth. Maybe youve built a large scale web index in the past and thus know from experience that this is the case, but I suspect not.If youd like, I can post a CSV with all the link data weve got on Smackdown.BlogsBlogsBlogs.com and you can compare against data from other indices. Were showing:2,038 links from 194 unique domains. Some of the more important links (at least, according to our metrics) come from sites like:adseok.combad-neighborhood.combadijones.comekstreme.comDigg.comSearchEngineWatch.comMetafilter.comProBlogger.netMattCutts.com (most of these are nofollowed, but theres one we see as followed)If theres other data that would help, I can certainly share.
Again, there is a distinct difference from having the data to analyze, and being the ones that went out and collected that data. I am not saying that you do not now currently have a large set of data in your possession, and there is no way whatsoever that my words could have been misinterpreted to that effect.
I finally get what Rand is saying: "Linkscapes spiders reach billions of pages" as in "Linkscapes data comes from a system that indexes billions of pages". So "Linkscapes spiders", not "SEOmozs spiders". That doesnt say who runs the spider/crawler, it just says what type of system the data comes from. <div></div><div></div><div>Yes, it doesnt say "SEOmoz has a crawler that...". However, that was the assumption everyone had made. Thats a clever usage of words; and Rand blames the listener for their misunderstanding. I think when you say it out load, and change the intonation slightly, suddenly it becomes clear.</div><div></div><div></div><div>Whether it was a calculated effort to be "fuzzy", I have no idea. I asked fairly early on for the UA, as I wanted to make sure I was not blocking it, and i wanted to see how many pages of a site had been analysed before seeing what data had been extrapolated from that.</div>
g1smd, there was nothing "fuzzy" about it. The help page had an image with these words on it:"Linkscapes bot crawls sites and pages on the web by following the links it discovers."It had a little picture of a robot, crawling from one link to another. It was very clearly labled "Linkscapes bot", not "the bots that the others guys use that we grab via their APIs and whatnot".Besides, I have a very sneaking suspicion that Rand is going to try another tactic now.
Rand, theres no way you and Michael are ever going to be able to see eye to eye. Theres just too much history there, so its pointless for either of you to try. However, just because it may or may not be personal, it doesnt necessarily mean Michael is wrong. If you didnt lie, you have to at least take the blinders off and realize that many, many, many people "feel like you lied". It sure "seems like" a lie. It sure "feels like" we were deceived and duped. Even if your inside information is enough to convince you that you didnt lie, you have to at least acknowledge that a whole bunch of people "feel like" it was a lie. It sure "felt like" you told us that you had a crawler. (And please, lets not call a crawler an index). It sure "felt like" you were expounding upon how great it was that you had a crawler. Regardless of the inside information that your company wont reveal, you most definitely did give impressions that now feel like lies.Some quotes from a page that has now been removed from seomozs site:"Linkscapes spiders""Linkscape crawls the web just like any other spider, discovering links to new pages and following them.""Linkscapes bot crawls sites and pages on the web by following the links it discovers".It was precisely because of quotes like this that everyone asked for the spiders name. It was precisely because of quotes like this that we believed you had your very own spider(s). No matter how the actual functionality now works that makes you "feel like" you have your very own spider, you dont. Or if you do, then you are now making us even more confused, and more likely to wonder which of the many versions are lies.Had you originally said something like, "we have an index" but hadnt said, "we have a spider", that would have been completely different. But you told us you had a spider(s). You made us wonder what the name of that spider was, so that we could block it if we chose. You then turned around and told us you wouldnt give us a name, which made us a little antsy. Then you said you didnt have a spider, you used a bunch of data from other places who had spiders of their own. Imagine how that would make people feel? What? You said you had one. You said we wouldnt be given the name. Then you said you dont have one. Now you say you do - sorta - cuz you use other peoples. Cmon. Put yourself in everyone elses shoes. Does all of that not sound like theres a lie somewhere in there? And if it "sounds like a lie", maybe you should at least acknowledge that theres good reason for everyone to call it a lie? If it hadnt been Michael to call it a lie, would that have made a difference?Just like in blunders of the past, if youd only started out with reality, it would have gone down very differently. Why not just say, "hey yall, weve got a huge index that weve compiled from various sources, and we have this really cool tool that pulls great info out of that index! Sweet, huh?" Everybody would have said, yeah, very cool, sign me up. (or something similar). And the thing, its obvious that you understand the mistake you made, because you changed the wording and/or deleted wording from posts/pages that now make "index" much more prominent than "spider".Look, Id be willing to bet the tool is really great (though I havent tried it). But it sure started out on the wrong foot, and no matter what you want to believe, it all boils down to the fact that it "feels like" you lied. Whether or not you did really makes little difference at this point. Its going to be very difficult to undo what you said, then unsaid, then said, then unsaid, etc. I, like many others, know what we read. We know what you said, and then later said. We know how that "feels", and for many of us, that feels like a lie. So, just thought you should hear that from someone other than Michael. BECAUSE I like you, in fact. Because I like you, you should hear that its even people who like you that "feel like" you lied. Thing is, I like Michael too. And say what you will about him, he only said what many others were thinking but didnt have the balls to say out loud. Like it or not, you need to realize that.
I just want to clear one thing up:Rand, theres no way you and Michael are ever going to be able to see eye to eye. Theres just too much history there, so its pointless for either of you to try.For my part, thats simply just not true. Ive said it before in the past, and Ill repeat it again... if Rand wants me to not call him on his bs, then dont give me anything to call him on. If he stops lying then Ill stop pointing it out. Its a very simple concept really.The majority of the time that I visit moz is when someone else points me to something that was said doesnt sound right to them. Sometimes I look at it and say, sure, that might be, and sometimes I think its bs but dont really care that much. On a very few occasions I felt compelled to speak up because it was above average bs that was happening, such as with this case.I am not one of those people who believes that other people cannot mend their ways. Rand came on to this scene 4 years ago, and wound up getting banned from what was back then a large portion of the community for scamming people. I have seen nothing whatsoever to indicate to me that he actually has become an honest person since then, but if he does, great. At that point Ill stop pointing out that hes not. :)
@Rand - to your comment:I do hope, however, that the level of discourse can rise above petty vitriol and negative personal attacks to something more aspirational and more worthwhile.I think its fair to say that Ive been a great advocate and supporter of SEOmoz. Whats disheartening to me is that you would make such dismissive comments like that simply because I held you to task. It was your complete dismissiveness of Mr. Vandemars post with a silly comment that led me to write to begin with. I really felt that you owed the search community at large an explanation for what many clearly feel was a breach of trust.Many companies withhold a great deal of information with regard to their strategies, tactics, sources, etc. I take no issue with that. I take issue, not for what you are holding back but rather what you said - what you "sold" to the community. I think theres enough evidence to validate that you expounded far beyond a semantic error. Essentially, youve oversold your product. The fact is, most SEO tools pull from or leverage other sources. From my perspective, it was unnecessary and unwise to mislead people as to how the data was gathered and in this case, knowing the sources you are drawing from could be considered important for someone to determine whether they felt investing in your product was worthwhile. As for being personal - youre the voice of your company, and with a small company especially, anything said - whether positive or negative, is going to come across as personal. I agree that you have a great team of people. However, I feel that just because that is the case doesnt mean that I should be required to turn a blind eye when I see something that breaches the trust and respect youve earned.Forty two people voiced their opinion with a vote for this post prior to my comment. Clearly they saw some validity to the content. That is something that should be neither ignored nor dismissed.
<div>*** Linkscapes bot ***</div><div></div><div></div><div>Yes, "Linkscapes bot" - the bot that collected the data to build the Linkscape product.</div><div></div><div></div><div>The Linkscape product is the data and the analysis.</div><div></div><div></div><div>"A" bot must have collected it, it didnt just appear out of thin air.</div><div></div><div></div><div>Note that "Linkscapes bot" is not interchangeable with "SEOmozs bot".</div><div></div><div></div><div>"SEOmozs bot" would imply a computer sat in the MOZplex, connected to the web, and pulling pages from sites all over the world to analyse.</div><div></div><div></div><div>"Linkscapess bot" only implies "the bot that collected the data" without saying who owns or where that bot is physically located. It could be owned by a third party and be "hired" by the Moz to collect specific data. In that case it is "Linkscapes bot" the bot what made Linkscape, but it still isnt "SEOmozs bot".</div><div></div><div></div><div>Yeah, its all semantics.</div>
@gab "Launch and iterate is hardly an SEOmoz-only practice. Any business startup / product launch advice columnist/blogger etc will suggest that."Of course. I was not pointing out that. I was pointing out the second part of the sentence:and somehow there is always an issue with timely response and somehow it is always a matter of good intentions turning into a PR nightmare.This is the recurring motif i was talking about. Not the fact that they launched an iterated...
g1smd i think that that is going into the words beyond reasonable expectation. Linkscape belongs to SEOMoz, it was built by people in SEOMoz, it is advertized by SEOMoz and the money that it earns goes into SEOMozs pockets. So when you say "Linkscapes bot", people understand it (and rightly so) as SEOMozs bot. You dont put that apostrophe before an "s" to mark that it doesnt belong to Linkscape. Had the intention been to describe the bot as it really is, i guess a good way would be to say "bots that had gathered the data for Linkscape to analyze". Or something like that. Another thing that could have been done is, after people started asking for the bot identity, to say something like "hey guys, sorry if i got you to believe that it was our bot (or bot of our making) that has gathered the data, cause it wasnt. the data was compiled from other places with their own bots." That is a way to put things in a non-vague way.There is no better explanation to this whole thing than what Donna had written. It is a pity that we are discussing the circumstances around the release of the tool and not the service that the tool provides. Rand, some of your products are great and valuable, i sometimes just wish you would make it easier for people to feel that way about them. Instead, and partly with your fault, we are discussing other issues around the tool.
Yes, "Linkscapes bot" - the bot that collected the data to build the Linkscape product.No. You cannot collectively refer to an assortment of bots owned individually by a variety of different organizations out there as a singular bot owned by a tool that is owned by exclusively by seomoz. Thats not semantics. Thats borrowing your dads Ferrari, and not only referring to it all night long to your date as "My Ferrari", but also boasting about how you paid cash for it.
Linkscape is a product. SEOmoz is a company. The possessive takes a subtle and different meaning.
Sorry - had to get eventually get some sleep, but Ill try to re-address all the new issues above.@Michael - on being the ones to collect the data. Youre suggesting that because the user agent name wasnt "seomoz" or "linkscape," its not data collected by us. I reject that - we collected the data.@g1msd - were now revealing all the potential UAs that power Linkscapes data, which I believe was the request. Its our intent to continue to list any source that might be part of that.@Michael - Again, I think its really picky to say that because the spiders that crawl dont have the name "Linkscape," I lied, but if thats the accusation and if it was misleading, you have my apologies. Now the UAs that crawl are all listed.@donna - We have our very own spiders and sources - they crawl just for us, on our design. They dont have the name "Linkscape," but I think the disconnect ends there. We said we wouldnt give out the names, but the community had a llot of impact on changing our minds and we decided to release them - theyre all listed, and if new ones come up, well list them as well. If we had originally said "we dont have spiders" that would have been an outright lie - and it would have suggested that there was no way to block us, which isnt true. I guess if we had come out with the list of UAs right at the beginning, that might have helped, and I do apologize that it took us a week to get that in shape. We initially made the decision to be quiet about it, because we felt it would harm the products potential, but changed our minds when it became clear that the community (and a lot of people we really respect) felt that it was unacceptable to keep it hidden.And yes, I agree that the due to the conflicts between Michael and I in the past, its tough to address his concerns or view them as reasonable and fair, but thats what Ive tried to do in this thread. Reading over my responses, they probably come across as too defensive, but I feel a bit lost by both the level of antagonism and the parsing of words to make it seem as though theres some direct, outright lie, which I dont believe is the case. To be honest, I dont even know why the guys pulled down the crawler page - its completely accurate - I think they just meant to replace it with the new sources page and didnt consider that it would look different. Ill ask that it gets put back up. There is, in my mind, no error in what it says:"Linkscapes spiders" - referrs accurately to the sources listed."They crawl the web to build an index" - absolutely true"The bots crawl the web through the links they discover" - also absolutely trueThanks for the valuable input and for the level-headedness, and let me know if theres more I can clear up.@Michael - I think your focus on historical issues doesnt help to make this process any better. It only makes you appear more biased and antagonistic. Weve been through all the previous disputes and every time theres a new one, you bring up all the old ones to help make me look worse and to remind anyone reading that Ive made mistakes in the past and that to you, that makes me untrustworthy in the present or future. "I am not one of those people who believes that other people cannot mend their ways." If that were true, this wouldnt be your style or approach.@Sean - Im sorry that you didnt like my response to Michaels post. I felt it was a ridiculous and inaccurate one - and I think Ive done a solid job of refuting and explaining that in this thread many, many times.And youre totally right - youve been a great supporter of SEOmoz and someone I always considered a friend until I saw that comment last week. I hope that we can mend fences and return to a place of mutual respect and support - you certainly have mine.On the charge of overselling the product - I strongly disagree. There is no claim - absolutely none - that the product does anything more than what it does. Reading through the help and edcuation center, including the crawler page, the blog post that launched it, the presentation I gave at SMX East on it, the radio interviews and the videos, there is simply no exaggeration. Linkscapes crawl has 30 billion pages. The crawl is our design, executed to create this link graph that can be researched through the tool. Weve never made any claims that the product does something or is something that its not. Give me any specifics here and Ill be happy to compare them against whats in the tool with data to back it up.On being dismissive - I think thats an unfair charge. I came into this thread and responded for hours - many, many hours - up until 3:30am last night, and now again starting at 9:30am this morning. I did this because I dont want to be dismissive. I want to address the concerns and the accusations and any misunderstandings.@neyne - I agree. It would be nice to talk more about what the tool does and how it does it. If there are any questions around that, Im happy to address them as well.Michael - that analogy is completely inaccurate. The bots may not have the name Linkscape, but this doesnt mean they dont crawl for us and it doesnt mean we didnt pay for them (we most certainly did - our investors and bank accounts can attest to that).Ill try to be in here more this morning to address more.
@Michael - on being the ones to collect the data. Youre suggesting that because the user agent name wasnt "seomoz" or "linkscape," its not data collected by us. I reject that - we collected the data.No Im not. I suggesting that because they turned out to have names like Googlebot, Yahoo! Slurp, msnbot, Exalead, and Gigabot that you didnt collect the data.You did NOT say "The bots crawl the web through the links they discover". What you said was, "Linkscapes bot crawls sites and pages on the web by following the links it discovers". Singular. One bot, owned by one company, and going back and trying to edit what you actually said does not make it true.
I think that once again, youre really playing at semantics. We did not initially reveal UAs - and we do now. Parsing a line attempting to explain the crawl process and accusing the singularity of a noun rather than a plural as being responsible for lies of the level youre claiming is exactly what Ive been suggesting is the problem the entire time - semantics.I think it just blows things way out of proportion to say that because we illustrated the crawl process with one bot, which we wouldnt disclose, but actually have multiple sources, which we do now disclose is this huge lie that distorts the abilities or value of the product.
No, Rand, and despite what you think your attempts at playing innocent arent helping you. Your entire initial sales pitch hyped the fact that your company went out and spidered 30 billion web pages. You werent "attempting to explain the crawl process", you were boasting of your feat.
Rand, it may be semantics, but sometimes semantics matter. If I said "Donnas dog ate Rands cat", youd assume the dog was mine and the cat was yours. You wouldnt assume that the dog was one I borrowed from someone else (or rented) nor would you assume the cat was one you kidnapped from a kid walking across the street. In reality, both of those things might be true, so if someone later said I lied because the dog wasnt mine, and the cat wasnt yours, I could theoretically claim that I didnt lie, because by renting the dog, it was "mine", and by kidnapping the cat, it was yours. Semantics might play in my favor, true, but it still wouldnt sit right with a lot of people. So even if by paying for the use of other peoples bots, they can be claimed as "yours", it still "feels like" we were duped from the get-go. Again, the tool is probably awesome, and we should all probably let this go (cuz everyone will in time anyway), but sometimes, semantics do matter. (I never inhaled). :)
Rand, you did NOT disclose a UA. A user agent is defined in the HTTP protocol (e.g. version HTTP 1.1 at http://cli.gs/QqayjG ). You disclosed a meta tag that only an HTML parser can read. These are very different things and its not a semantic difference.As for a crawler, I agree with the comments above that the wording suggested there is a spider that SEOmoz controls, but you know that I didnt think thats true (you saw my blog post). The way I understand LS to work is that SEOmoz gets hold of downloaded pages from around the web, say an index of cached pages commercially or freely available. Those are then parsed on SEOmoz computers and the link graph calculated on SEOmoz computers. Right or wrong? If wrong, whats the exact method of operation? Again, a parser and a crawler are very different things that are not just semantically different.As for any suggestions that SEOmoz, or you personally as Rand, were driven by any "malicious" motives (for the lack of a better word), I think thats just plain wrong. I think as a marketer you could have handled this better but I dont see any evidence of malice.Cheers,Pierre
Ive read this entire thread, and the one thing that stands out for me (besides Donnas EXCELLENT post) is Rand having said this:"...the more Ive found that honesty (and especially transparency) does come into conflict with promises that have been made and the needs for stealth and protection." Transparency and honesty are not synonymous. Of course you, your company or any business does not need to be transparent in everything you do. That would be silly, imo.But honesty, in my opinion, should be the foundation for every business and for every person of integrity. Anything less will get found out and will create more problems then simply being honest from the outset.While I cant say whether or not Rand has been completely honest on this crawler issue or others (and quite frankly, I dont really care), his statement here that honesty comes into conflict with other things is one I would not have expected to hear.
@Michael - If we boasted, I think it was a fair boast, since all of the work we claimed has been accomplished by the people in SEOmozs offices and on our payroll. That index was built by us for use in the Linkscape tool - your continual attempt to deny it doesnt make it untrue. Its a unique data source. We designed it, and we are responsible for the how, when, what, where and why it was built.@Donna - The semantics can matter, but I think in this case, Michaels just desperate to try to show some massive fraud that was perpetrated, when thats simply not true. If I say "Donnas dog knocked over the trash" and it turns out Donna has more than one dog involved in the trash knocking over process, that might mean that I wasnt telling the whole truth, but it doesnt mean I was lying. And I admitted, as soon as we revealed our sources, that we hadnt been telling the whole truth. In fact, we admitted when we launched that we werent revealing the whole truth - that we were holding back on disclosing the source. Now that weve come out with it, and the source is against expectations (in that we may pull from one or more of the datasets listed), its being taken to mean there was a perpetration of fraud.I reject that notion. I accept that we were not fully truthful, and in fact, we still have lots of information about the index and the data that were keeping under our hats, but I maintain that a lack of disclosure is not a lie and that no claims were made about the products capabilities or value or functionality that were false.
@dazzlindonna - Actually, after reading this thread, if you had said to me "Donnas dog ate Rands cat", I would think that you had a pack of undisclosed dogs, one of which may or may not have been yours and that that undisclosed dog may or may not have been involved in eating Rands cat. Further, that Rands cat may in fact have been a litter of harmless kittens and as such, you should be found guilty of cruelty to animals and spend the remainder of your life in prison.No harm intended. You understand, yes? :)(Ok, that was my effort to bring some levity and hopefully closure, to a thread thats gone way beyond the call of duty.)and for your reading pleasure: http://www.slate.com/id/1000162/
@Pierre - We disclosed all the sources, and each of them have UAs that can be blocked. Linkscape doesnt use data thats freely available (although from a financial perspective, I wish that were the case), and the index is completely unique, designed by us and built with Linkscape alone in mind. Spiders with UAs that weve disclosed collected the URLs to our specifications (as I pointed out above, it could not have been otherwise or we wouldnt have been able to build the metrics or design the crawl structure, refresh rate, diversity, etc.).On the malicious motives - thanks. I guess that it is true that our motives for not revealing the UAs was based on a desire not to be blocked or restricted, so some might view that as malicious. I think were doing the right thing now, both by revealing sources and by allowing anyone to remove a page exclusively from SEOmoz with the meta seomoz noindex tag.@Jill - I guess I should have been more clear "full honesty" or "complete honesty" which would require complete disclosure and transparency are what comes into conflict with running a business and growing to a larger scale. I think that as this thread has illustrated, we were not fully transparent (and still are not in some areas), but I believe we haev been completely honest - certainly in spirit, even if a few semantics were off.
Rand, might I suggest that you step away from the computer today? (If only to not make Sarahs job even harder than it is already going to be on Monday morning!)
@seanmag - LOL! good one!@Rand - I guess well just have to agree to disagree on some things. Your perception is different than others. Thats what makes the world go round, and leads to a lot of disagreements.And on that note, Im gonna go play spider solitaire.
My 2c as follows. Anyone can write a crawler. Ok, its NOT that easy but fundamentally - a piece of software that runs on a server that fetches data and stores it somewhere. Its the storage, analysis and recall of the data into meaningful reporting that is impressive. All of that number crunching requires some serious equipment to run on, serious development to create and perfect and serious money to make happen. I personally dont care if Linkscape uses lots of external data sources with or without their own crawl becuase I appreciate the additional tool at my disposal and I appreciate the level of work involved to get those reports to make any real world sense. There are some interesting similarities between Linkscapes metric results and Google serp positioning. Its a nice algo!
Jill - I was accused of not caring about this problem and not addressing the concerns or charges leveled, and I think that if Im going to put in this much effort, I should probably continue to make sure theres no other questions left unanswered. Admittedly, with my 6 hours of sleep, I might be a bit more sensitive than usual (although, Im pretty sensitive in general, and let me highly recommend to any other startup CEOs out there not having that trait or limiting it as much as possible), but hopefully I can still honestly and forthrightly present our case.Regarding Sarahs job - I assume youre referring to our in-house counsel/COO? Have I said something that violates the law or suggested that were doing something thats illegal?
I think that the comments on this story are getting out of hand and going around in circles. Ill be closing comments as neither party are likely to back down.
Our bots, our crawl, our index...hand-built from the ground up. Mr. Vandemars accusations are simply and completely untrue.