Forrester Wave Web Analytics Report Wipes Out


Last week, Forrester released their Web Analytics Q2 2014 Wave on: “the Six Providers That Matter Most And How They Stack Up”. The Wave report evaluates Web Analytics platforms against a number of criteria.

It was a busy week last week, so I didn’t get a chance to read the report till this weekend, then read some colorful reviews including this one and this one :).

My first reaction when reading the report was “Whoa!” and “Are you serious?!”

Why was this my reaction? From our own experience, some of the conclusions reached in the report just don’t add up. For example, we clearly find that organizations are consistently leaving many of the platforms that this report considers “leaders” for the tool that this report considers just a “strong performer” (not to mention any names).

Now, it might be assumed that we have a biased towards Google Analytics/Google Analytics Premium (not to mention any names :)), but in fact we are vendor-agnostic, working with Adobe SiteCatalyst and other analytics platforms. Yet what we see and hear from clients day-in and day-out, from the trenches, just don’t match this report.

Here are a couple of concerns with the report:

The Blackberry of the Web Analytics Industry

(Again, not to mention names) but one of the “leaders” of this report in our view is on it’s way to becoming the “Blackberry” of the web analytics industry. (When was the last time you saw someone check their Blackberry?) We are seeing a mass exodus from this one vendor.

And this is based on a good sample size of clients we’ve worked with – people are leaving [insert name of tool :)] due to a combination of high pricing, poor support and slow pace of innovation. Maybe this vendor will turn around in the future, but for now I’m not sure how they still even made it to the list, let alone a “leader”.

Not only have we seen this trend in the private sector, but also in government. And, speaking with other consultants in the industry, they share the same experience.

So, not sure how to reconcile what we experience to be true with this report’s classification of “leaders”. Maybe “client retention” wasn’t in the criteria (speaking with ex-clients probably would be a good idea to consider for the next report).

User Friendliness –> Adoption By the Broader Audience

Some of the vendors in the “leader” category leave much to be desired when it comes to their user interface. I think the report underestimates the impact of this aspect on organizations’ adoption of web analytics.

We’ve seen it first hand with customers – most people won’t adopt a clunky interface, not matter how powerful and feature-rich the product is. On the contrary, provide a user-friendly interface and your adoption goes through the roof, as evident in the reach and deployment of Google Analytics/Google Analytics Premium and the growth this product has experienced over the last few years.

A good chunk of the internet – large brands, successful businesses, and organizations with a lot of really smart people – have adopted Google Analytics. Forrester seems to be excluding the “vote” of people who are actually using tools to make day-to-day business decisions in order to increase profit. You’d think their opinions would be a top priority when when electing analytic platform “leaders”.

Don’t Get Tunnel Vision on Technology

Just at it’s core, a report from a respectable firm like Forrester may lead you to choose the “tool” or “technology” perceived to be a “leader”. However, when choosing a tool, technology should only be A factor, not THE deciding factor.

Although the industry has matured significantly over the last few years (we’ve even changed the name of our associations from web analytics to digital analytics!), we still see an over-emphasis on technology and hardly any substantial investment in process and people.

Having a lightning fast race car without someone who knows how to drive it (or without a team that knows how to maintain it, change it’s tires, oil it up) is a useless piece of expensive machinery.

My advice to those in the market for an enterprise web analytics solution is to define your requirements thoroughly and have a vendor selection phase (prior to implementation). And in that process ensure that you have the right talent that will do wonders with the technology.

I noticed that Forrester is running both Google Analytics and Adobe/SiteCatalyst on their site. Two “leading” choices in my book! :)

7 thoughts on “Forrester Wave Web Analytics Report Wipes Out”

  1. I think that much of the criticism over the Wave has come from a misunderstanding of what it’s intended for. Forrester isn’t trying to pronounce “leadership” of the whole industry – their research (which, as one of the PMs responsible for responding, was quite thorough, I assure you) is meant to help guide large companies with sophisticated needs to choose a vendor. Hence, the prerequisites of $20M revenue, large base of enterprise clients, etc.

    Moreover, don’t be misled by some vendors’ marketing. The “Blackberry” you mentioned above (we all know who you mean) has a huge number of clients and a ton of revenue today – far more than the #measure chattering class likes to think. Demystified called out AT Internet specifically for criticism – which only makes sense if your perspective is limited to North America, in my view. (I’ve personally been involved in deals in Spain, France, Brazil and Thailand where AT is increasingly coming up as strong competition.)

    I’ve found that most perceptions of “leadership” are wholly anecdotally based – how often I see XYZ on twitter, or sponsoring eMetrics, or whose users I meet. Those are interesting datapoints, but don’t tell the whole story. From the vendor side, we all give confidential info to Forrester – revenue numbers, client names and references, an exhaustive and detailed questionnaire filled with both quantitative and qualitative questions. They’re not just throwing darts at a dartboard here.

    So, just saying… don’t be too dismissive of research like this. There aren’t many other researchers out there in a position to make as fair a judgment.

  2. Thanks for your comment and insights Blair!

    I’m not sure I agree that “Forrester isn’t trying to pronounce ‘leadership’ of the whole industry.” They did state early in the report that their evaluation is “see how the vendors stack up against each other” in general.

    We’ve been in this business for a long time advising some of world’s most recognized brands and have worked with industry analysts as well, and I’m aware of the efforts and process involved to produce such a report, but if the conclusions are off the mark, they are off the mark. Many of us in the trenches (on the consulting side), are sharing these significant “anecdotes” that do add up to a trend we see, and we’re saying it as it is :), including what my friend Stephane just published as well.

    Maybe it’s the approach/criteria that should be revisited.

    Regarding the “Blackberry” comment, this wasn’t influenced by vendors marketing, it’s what we have seen over the last many years of clients migrating off this vendor (and in a private conversation at the next networking event, I’ll share a long list of names we can share :)). And I agree with you, this vendor has a ton of revenue today, I hope they put it to good use in product development and innovation. But I don’t think a ton of revenue should place you in leadership, especially when client retention is an issue (in North America and globally).

    Last but not least, Forrester is indeed a respected research company, but the report seems to be based on a default weighting of criteria, which might not be very clearly defined, and therefore is misleading.

    We do encourage our clients and those looking for an enterprise analytics solution to tailor the feature weightings to their business using Forrester’s online customization tool.

Leave a Reply

Your email address will not be published. Required fields are marked *


You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">