This post was written by Graham Hill as a comment to my latest post. Since it is a great read I asked Graham if I could put it up as a guest post. Fortunately he agreed. Enjoy!

Big data is held up by many (particularly those with expensive big data services to sell) as the next big thing in business. That may be partially true if your aim is to make a smarter planet. But is it also true for your average company struggling to make sense of the transactional data it already has, let alone the lorry loads of big data piled outside the door of the data center?

Making big data work depends upon pulling a number of business levers at the same time; only some of them connected directly with big data or its analysis.

The Right Data
Big data, as the name suggests, relies upon large volumes of data. Many companies already have large amounts of data. Mobile telcos, for instance, have enormous volumes of data, generated each time a customer makes or receives a call, an SMS, or in these day of the smarphone, uses a data-driven app. But this is largely transactional data about things customers have bought or done in the past. But what about the context in which the customer did something? Or the friends and family that influenced the customer to do it? Or their underlying needs that drove them to do it in the first place? If big data is to do anything more than incrementally improve the current crop of small data predictions, it will have to start collecting a much broader variety of data that captures not just what the customer did but circuemstanes leading up to it as well. Hardly any of this data is collected by companies today.

The Right Analysis
Once companies have started to gather big data they need to analyse it to generate actionable insights. This may require building the types of complex econometric models that are all the rage with advertisers struggling to justify their marketing spend on advertising. Larger data sets provide more inputs for better predictive models. But perhaps more importantly, big data may trigger the bulding of many more smaller, real-time models that can be used to influence customer behaviour. Insurance compaines, for instance, are building self-service tools for use by financial advisors, call centre agents and even customers, to provide lower-cost decision support at key points in the customer journey (albeit, largely in response to the MiFID legislation due to come into force in Europe in 2013).

The Right Value Propositions
Having big data and the tools to generate actionable insights is of no use unless they are actually actioned. That means using them to develop better value propositions for customers. Despite the groundbreaking work of Lanning & Michaels at McKinsey in the early 90s, most marketers still struggle with value propositions. They confuse propositions with lists of product features, emotional rhetoric in marketing copy and marketing gimmicks, like extrinsic gamification. Customers may have bought in the past, but they were never fooled. Today, we find ourselves in a tragedy of the marketing commons with marketers having to shout ever louder to make themselves heard over all the other marketers. And customers ignoring marketers in favour of talking to their friends and family. As Seth Godin remarked a few years ago, ‘all marketers are liars!’.

Keeping Their Side of the Bargain
Perhaps the biggest problem that marketers have with their carefully crafted propositions is their almost complete failure to recognise that they also have to ensure the propositions can be delivered exactly as it say on the tin. That is what customers expect they are paying for when they hand over their hard-earmed cash. As Prof Andy Neely pointed out in a recent blog post about the failure of his bank to offer the best rates, (itself a potential contravention of the UK’s Treating Customers Fairly legislation), marketers routinely fail to ensure their companies keep their side of the bargain. We all have many other examples of compaines treating us unfairly, particularly once they have taken our money.

Attributing Customer Behaviour
Even those marketers who diligently gather the right data, do the right analyses, develop the right value propositions and keep their side of the bargain often fail to properly attribute changes to customer behaviour to their own actions. It starts with blanket assumptions about all changes in customer behaviour being due to marketing activities and procedes to self-delusional econometric models that conveniently forgets to compund error margins in multi-stage models. If your predictive models explain less than 50% of the observed variation in customer behaviour, it is time to go back to the drawing board.

Small data and its 900lb Gorilla relative, big data, clearly have a role to play in tomorrow’s business. But to do so companies have a lot of capability building to do first. Just having big data, – even allied with the right analyses – is not enough to succeed. If companies don’t build their capabilities out at the same time as gathering big data, I can confidently predict that the era of big data will only accelerate the tragedy of the marketing commons.

Graham Hill (@grahamhill)