Graham Hill: Making Big Data Work

This post was written by Graham Hill as a comment to my latest post. Since it is a great read I asked Graham if I could put it up as a guest post. Fortunately he agreed. Enjoy!

Big data is held up by many (particularly those with expensive big data services to sell) as the next big thing in business. That may be partially true if your aim is to make a smarter planet. But is it also true for your average company struggling to make sense of the transactional data it already has, let alone the lorry loads of big data piled outside the door of the data center?

Making big data work depends upon pulling a number of business levers at the same time; only some of them connected directly with big data or its analysis.

The Right Data
Big data, as the name suggests, relies upon large volumes of data. Many companies already have large amounts of data. Mobile telcos, for instance, have enormous volumes of data, generated each time a customer makes or receives a call, an SMS, or in these day of the smarphone, uses a data-driven app. But this is largely transactional data about things customers have bought or done in the past. But what about the context in which the customer did something? Or the friends and family that influenced the customer to do it? Or their underlying needs that drove them to do it in the first place? If big data is to do anything more than incrementally improve the current crop of small data predictions, it will have to start collecting a much broader variety of data that captures not just what the customer did but circuemstanes leading up to it as well. Hardly any of this data is collected by companies today.

The Right Analysis
Once companies have started to gather big data they need to analyse it to generate actionable insights. This may require building the types of complex econometric models that are all the rage with advertisers struggling to justify their marketing spend on advertising. Larger data sets provide more inputs for better predictive models. But perhaps more importantly, big data may trigger the bulding of many more smaller, real-time models that can be used to influence customer behaviour. Insurance compaines, for instance, are building self-service tools for use by financial advisors, call centre agents and even customers, to provide lower-cost decision support at key points in the customer journey (albeit, largely in response to the MiFID legislation due to come into force in Europe in 2013).

The Right Value Propositions
Having big data and the tools to generate actionable insights is of no use unless they are actually actioned. That means using them to develop better value propositions for customers. Despite the groundbreaking work of Lanning & Michaels at McKinsey in the early 90s, most marketers still struggle with value propositions. They confuse propositions with lists of product features, emotional rhetoric in marketing copy and marketing gimmicks, like extrinsic gamification. Customers may have bought in the past, but they were never fooled. Today, we find ourselves in a tragedy of the marketing commons with marketers having to shout ever louder to make themselves heard over all the other marketers. And customers ignoring marketers in favour of talking to their friends and family. As Seth Godin remarked a few years ago, ‘all marketers are liars!’.

Keeping Their Side of the Bargain
Perhaps the biggest problem that marketers have with their carefully crafted propositions is their almost complete failure to recognise that they also have to ensure the propositions can be delivered exactly as it say on the tin. That is what customers expect they are paying for when they hand over their hard-earmed cash. As Prof Andy Neely pointed out in a recent blog post about the failure of his bank to offer the best rates, (itself a potential contravention of the UK’s Treating Customers Fairly legislation), marketers routinely fail to ensure their companies keep their side of the bargain. We all have many other examples of compaines treating us unfairly, particularly once they have taken our money.

Attributing Customer Behaviour
Even those marketers who diligently gather the right data, do the right analyses, develop the right value propositions and keep their side of the bargain often fail to properly attribute changes to customer behaviour to their own actions. It starts with blanket assumptions about all changes in customer behaviour being due to marketing activities and procedes to self-delusional econometric models that conveniently forgets to compund error margins in multi-stage models. If your predictive models explain less than 50% of the observed variation in customer behaviour, it is time to go back to the drawing board.

Small data and its 900lb Gorilla relative, big data, clearly have a role to play in tomorrow’s business. But to do so companies have a lot of capability building to do first. Just having big data, – even allied with the right analyses – is not enough to succeed. If companies don’t build their capabilities out at the same time as gathering big data, I can confidently predict that the era of big data will only accelerate the tragedy of the marketing commons.

Graham Hill (@grahamhill)

4 thoughts on “Graham Hill: Making Big Data Work

  1. Hello Graham and Wim

    Many thanks for your considered responses.

    I find myself to be a simple man. It occurs to me that one needs first to be adept at crawling before walking. That one needs to be adept at walking before running. And when needs to be adept at running before going to break the 3 minute mile.

    I ran a data mining and predictive analytics practice for 2 years. What was my experience? Usually it took months to get the organisation to determine-extract-provide the right data. Mostly it was garbage and we had to expend considerable effort to ‘clean and enrich’ it for it to be useful for data mining purpose.

    And then you get into data mining and the fun starts. When you have upward of 200 variables what comes with the territory? Relationships and trends show up simply by virtue of the number of variables! Which ones are meaningful? Which ones are spurious? Who has the requisite insight and time to figure that one out? And how do you explain that to business folks so that they act? We are conditioned to think cause-effect and once that strikes us as sensible then we act. What happens when all that you have is a correlation and you cannot figure out the cause-effect link? Then you show up as flaky in front of the business person and nothing gets done. Furthermore, if the data/insight generates ‘pain’ for the relevant business person then it is dismissed for some reason or another.

    All of this does not even take into account the severe shortage of people who have the requisite skill AND expertise in data mining/predictive analytics AND sufficient insight/grasp of the business issue under consideration.

    I wish you gents well, I get value out of your writing and thank you for that.



  2. In the world of theory: big data is great and it will revolutionise strategy, marketing, customer service, operations….

    Reality as I have experienced it: big data will make little difference – vast majority of companies cannot even get insight from the data they already have. And those that can generate insight find it hard to effect change in the organisation that actually delivers genuine value to customers.

    Summary: big data, big hype. No revolution. Just the gullible being gullible.


    • Hi Maz

      Thanks for your comment.

      I largely agree with you. Many companies have too much data already and struggle to make sense of it. And often the data is too historical and too transactional to be of real use in predicting the future. Having said that, some companies, e.g. mobile telcos, credit card companies and airlines, do collect, analyse and model data, often in real-time, and do use the insights generated to improve business effectiveness.

      Big data will certainly add to the volume of data available to be analysed, but I am not so sure it will always add much value to all the businesses that do so. But these are early days so we will have to wait and see how things develop.

      One thing in your comment is particularly interesting. Why should companies “deliver (genuine) value to customers”? The history of business analytics is all about delivering value to companies, not to customers. And companies have done very well out of it. Why should that be any different with big data?

      Graham Hill


    • Exactly my point Maz! And I very much agree with Graham that very few companies are using their current data to enhance Customer’s value creation process.

      But it’s not all gloom and doom: there are plenty examples where companies use customer generated data to help Customers choose through recommendations, insights into other Customers’ buying habits etc.. Services like runkeeper and more are helping Customers making sense of their own data and compare it with other ppl data, and I’m comfident we’ll see plenty more of those in the near future..

      When it comes to the average marketeer’s wet dream of Big Data, I’m, unfortunately also confident it’s just like you describe..

      Thx for the visit & time to comment!



Comments are closed.