The abyss of analytics

I want to talk about a mistake I see client after client making. (I work at a tech consultancy. We have a lot of clients. Not all of them make this mistake! …But many do.) That mistake is to obsess over analytics data, without any strategy; to assume that all that needs to be done is to gather as much data as possible, and then this data will magically become knowledge, and knowledge will mystically become wisdom.

I understand the temptation. Suppose you’re building a new web site, or a new app. Of course you want to know as much as you can, in as much detail as possible, about how users use it. Of course you want activity trackers, heat maps, funnel analysis; of course you want every twitch and false start from every user to be logged for all eternity. Data is the new oil, the new gold. Of course you need to gather as much of it as you possibly can, to be parsed and refined and mined later on. Right?

…Well, yes. But. I put it to you that data for its own sake is meaningless; that you should know what questions you want to ask of it, what criteria you want to measure, what targets you want to aim for, before you start collecting it. There is an opportunity cost to analytics data: time put into defining and collecting it is time not put into honing and refining your product. And if you determine what questions you want to ask of the data first, instead of saying “collect it all and let Future Me sort it out” — I think you’ll generally find that this will inform your product design in an extremely helpful way.

There is a myth, of course, a myth that grows with nigh every case study at every MBA school, that somewhere within the analytics data your site or app or service collects, in some obscure row or column, you will find the secret to your ultimate success. The famous Facebook “aha moment.” Or the famous … well, actually, that Facebook example is repeated ad nauseum because it’s generally the only one people can think of. But, more generally, the myth that your analytics data will make you understand how to hockey-stick your users.

99% of the time that is not how it works. 99% of the time you get screwed by selection bias. You get no data at all from the users who never come to you, because they can’t be bothered, because they aren’t interested enough, because they never heard of you. You get almost no data from the users who immediately bounce. The data you do get, the so-called “rich” data, are from your engaged, interested users — but making marginal improvements for them won’t help you. You want to improve the experience for the users for which you have no or little data. Explain to me again how your analytics will help you there?

I’m not saying data is valueless. I’m not saying analytics are completely unimportant. But I am saying that before you obsess about them — and believe me, with far too many of the clients I’ve had, “obsess” is the right word — ask yourself what questions you will ask of your analytics data, and what value you expect to receive. Don’t assume that its value is automatic, and just needs to be mined, when all too often it is fool’s gold at best. Don’t collect data for its own sake, collect it to answer specific questions — and know what those questions are well before you launch.