by

How To (Not) Get Smart About Big Data

If you are to believe the talk of twitter-town and its suburbs, due to the connectivity of numerous devices worldwide, we (will) also have available so much data, it is just waiting to be mined and will change how we do, well..,  just about everything. All this is being referred to as Big Data. The problem with all this data of course is the filtering.

There is a lot of noise, and despite improvements in Social Media monitoring, analytics tools/solutions and what have you, we will need a lot more powerful tools to connect the dots and see patterns. We may need Watson-like technology to automate these processes and then still the outcome is not sure.

Too Big?
And because Big Data is so BIG there’s a clear call for BIG companies to join efforts to explore and undoubtedly monetize it through building a ‘smarter’ planet (pun intended). And whilst, with the right mindset, a lot of good can come from it, I don’t think we have too good of a track record that all stakeholders (and yes, that includes you!) will benefit from it sufficiently. (I suggest you also read J.P. Rangaswami post on the theme. He makes some excellent points!)

What can you do?
But what can you do? How can you find your way through the clutter? How can you, not so big company or smaller business unit in a big company, understand if you need to fear or embrace Big Data? Esteban Kolsky says it exactly like it is:

I’d say that organizations get bombarded by Big Noise, not Big Data — data is what is filtered out of that noise.  The resulting data is not  something you need to fret about how to handle; [..]

Good time to shift strategies from panic, knee-jerking mode to calculated, strategic mode – don’t you think?

I could not agree more. It’s no time to panic, and it’s not the time to go out and buy all the technological solutions you may be led to think you need. What does it mean to go into calculated, strategic mode?

5 Questions to enter strategic mode
A good way to start being calculated and strategic about this is asking yourself 5 important questions. Here the are:

  1. Start with asking (business) questions you need or want answers to. This could be any question, related to your processes, your customer needs, habits, your points of sale.. etcetera etcetera. Because, if you do not ask the right questions, you will never find the right answers in any data, let alone Big Data.
  2. Re-think what you need the answers to your questions for: what is the proposed value coming out of knowing the answer? Will knowing the answer eventually result in creating more value for the company and the Customer? Is it actionable? If not, skip the question and focus on the ones that do provide actionable insights. There’s little time and little money, so you need to be effective with both resources.
  3. Ask yourself: how can I obtain the answers to the questions fastest and cheapest? Can I get closer to the answer(s) by first asking my Customers? Can I get closer by first using data I already own? More data does not always mean better data. Relevancy is not always easy to establish, but 9 out of 10 times, the not so so sexy, not so far away, not so expensive is good enough. You don’t need to be exact all the times. You need to be closer than before.
  4. If you still think you need to tap into Big Data, or need surrounding solutions, make sure you start any project with experiments and prototyping. Evaluate and iterate in short cycles, until you get it right. And don’t waste too much of your time getting it right. People will loose interest, and even if you get it right, chances of success decrease exponentially if people hopped on the next train.
  5. Last, but not least, ask you self the question if you need all this “in (near) real time” like ‘they say’. Or that running your analysis once works just as well, because the patterns do not change that much.

To conclude: I do think though that you need to start answering these questions and jump into strategic mode. There is a lot of noise, but there’s a good chance some of the data in there is very useful to you. But you will never find it if you’re not looking for it strategically. And it will certainly not find you as fast as someone else can find it before it does.

So, what do you think?

39 thoughts on “How To (Not) Get Smart About Big Data”

  1. Pingback: About data, Big Data and People | It is all about social good!

  2. Pingback: Want Some Big-Data With That? « Wim Rampen's Blog

  3. Real Estate in Hurghada.
    Ayaro Real Estate is a leader of E-marketing real estate investment search site, is a fully developed real estate investment company, well known in Egypt by honest developers as the leading real estate network in Egypt, Hurghada, red sea, Alexandria, North Coast and Sharm Elsheikh. Our branches are in three locations in Egypt. We are known for providing the transparent and best services for those who are interested in Real Estate investment projects, property services and Rental. Our rich list of investment opportunities and properties including studio, apartment,1bedroom, 2bedrooms, 3bedrooms, villa, plot of land, building, hotel, Factory or new development and building for sale enables you to search and browse unique properties and attractive from Hurghada,across Hurghada, allover HURGHADA, SAFAGA, EL GOUNA , SAHL HASHEESH, MARSA ALAM , EL QUSIR, SHARM EL SHEIKH ,RED SEA, NORTH COAST , ALEXANDRIA, CAIRO, ASWAN, LUXOR, SINAI, DAHAB, NUWEIBA, TABA, and all over EGYPT.www.ayaroestate.com

    Like

  4. Forgive me Wim for jumping on the bandwagon (I do have original thoughts once in a while). Two critical points from a practical perspective IMO: 1) we’ve seen it before…there is an accelerating hype cycle here that, of itself, isint all that terrible. Heck, everyone needs something to get excited about. The economic impact though is that, as with others, the hype drives finite investment dollars towards the mode du jour and away from other competing innovations/projects/etc. 2) As the cost to aquire data, particularly social and other cusotmer data, screams towards zero, the focus seems to be on acquisition – more = better. Its not a new phenomenon. “data rich, information poor” is a decades-old characterization. The focus ought to be on actually doing something meaningful, creating analysis thats actionable from existing data rather than investing in acquisition and retooling data services towards new sources of data. Here’s an interesting throw-back perspective from The Economist from two years ago http://www.economist.com/node/15579717

    Thx
    Barry

    Like

  6. Good post, greater discussion Wim. :-)

    Big data is a moving target, since ask the term really means is, typically, data which id larger than what is usually handled nut the existing systems.

    What you are saying about questions, etc. is only the analysis part and then only for known issues. Can it see the unknown unknowns?

    And big data is also about collecting, storing and presenting data that has been analysed as I explained in my post mid last year: http://sfh.naasat.in/2011/06/of-pigs-elephants-hives-social-crm-and.html. Not immediate concern for the business manager, but the IT guy is losing his sleep.

    Like

    • Hi Prem,

      Excellent question! My counter question is though: why is everybody do hung up on the “unknown unknowns”.. Great innovations do not often ‘happen upon you’ by stumbling into a previous unknown unknown. Most innovations are a result of continuous refinement, digging deeper and deeper, answering small pieces on the way, until the full picture is clear. And that’s what I’m suggesting. On top of that I’m suggesting doing this with little data first. In most companies there is still plenty opportunity to innovate without the need to dig into Big Data.. Much like there is plenty to innovate in Customer Relationships without using Social Tools ;)

      And the IT guys should loose their sleep over things that worry the business, imho ;o

      Like

      • You are right, may be I should not be overly concerned with unknown unknowns, but why get so hung up with big data as such? I am agreeing with your suggestions. I am merely unsure about the knots people have in their knickers. I didn’t realize it was of concern to small businesses too.

        Like

          • Actually, my question is why everyone seems to freak out about Big Marketing Data. We’ve been dealing with “big data” in other arenas for years. A pharma clinical trial with several thousand patients over two years, Wal-Mart analyzing transactions and setting store-level pricing every day, traders tracking sub-penny movements in option pricing, SETA sifting through noise to listen for life on other planets… We know how to do this, as long as we are starting with a question to be answered or hypothesis to be disproved.

            You clearly stirred the pot, Wim. And just in time for an article I’m writing on CRM and Big Data. Thanks! :)

            Like

      • “In most companies there is still plenty opportunity to innovate without the need to dig into Big Data”
        Wouldn’t having more data about previous unknowns lead to a much better decisions in the innovation process? When having lots of data, and the ability to make sense out of it, this will give us the ability to dig deeper without setting up some big survey. It’s too easy to just dismiss it in my opinion.

        Try substituting your sentence with “computers” or “cars” and these are the kind of reactions people would have given to new developments in their era: “In most companies there is still plenty opportunity to innovate without the need to dig into computers.”

        Like

  7. Well, nice observation but..
    I look at that from another point of view – The Business point of view.
    In a competitive world everyone want a bigger piece for their business. In order to Achieve that organizations will have to be “mind readers” and better understand what their clients and prospects want at any given time and in real time !!
    One cannot ignore the reality of cost cutting and current economics but, hey, If you want to sell more and provide better value you’ll have to BE There at the Big Data ball park. 
    There are other challenges in the Big Data game such as the Infrastructure that needs to digest Hugh amount of data, network that needs to transfer very fast that Hugh amount of data, The applications that Must support that volumes etc …
    My main starting point is the Business needs and the organization strategy, targets for the short and long term. Once the Business lines understand the Hugh benefits of Big Data and How to make Hugh revenue out of that, all the rest will be of no obstacle to create a successful Big Data platform

    The main point is – Strategic Thinking, better brainstroming with no constrains. reaching the wildest “wish lists” of the Business needs.
    Everything is possible once you KNOW what You Want :-)

    Like

    • Thx Nessy,

      I was hoping for a dissenting opinion. And you show us exactly where the dilemma is with respect to this Big Data getting bigger. Thank you very much for pitching in!

      In general my thinking is that you can find out what your Customers need by plain observation, talking with them, asking them questions and what have you. Nothing earth-shocking and certainly not data intensive.

      Once you have a pretty good view of where your customer is not satisfied or unable to create as much value as he would like, you can start designing solutions to help him. And then of course it makes sense to look closely into the opportunity Big Data can provide in your solutions.

      I’m not saying that your way won’t work. I am saying that, before you take a go at ‘your’ way, it is best to use ‘my way’.. For if ‘my way’ finds you a solution it will be found faster at less costs and will certainly please your Customers.

      What do you think?

      Like

  8. Hi Wim

    Another great post.

    I couldn’t agree more with you. As the volume of data increases (and we haven’t even started to gather machine to machine data yet) then making sense of it will require a smart approach. For organisations without the vast resources of a computing technology company like IBM that means taking a very pragmatic approach; like that you suggest.

    At the end of the day, to create value, big data only needs to be consumed in small chunks that individually enable better decisioning. The key is finding the right data amongst all the noise, developing the right analytics to take advantage of it and particularly, getting it into the market as quickly as possible. The proof of analytics is in its usage. All of this using the iterative prototyping approach we have discussed before.

    As computing power increases and data visualisation gets less expensive, we may be able to handle more and more recent data. But for most organisations that is not necessary. Vodafone’s pre-paid business may need real-time data to manage customers effectively, but most companies do not.

    Graham Hill
    Customer-centric innovator
    @grahamhill

    Like

  9. Wim et al – great discussion. From my perspective marketing’s ‘Big Data’ is the result of a world where everything is interactive and people are the media. As such it supports the view that transactions, events and comments across channels, locations, and devices are all part of Big Data. The main business challenge appears to be dealing with the “Fluid Fog” – the fact that the path to purchase is both non-linear and potentially unique for each individual. The framework for (media) planning doesn’t seem to hold very well any more. Cross-posted the questions here: http://apowerpoint.blogspot.com/2012/01/dealing-with-big-data-take-deep-breath.html

    Like

  10. Interesting blog. We certainly think big data is already here. Agreed that social media creates a lot of noise-generated data. We tend to define big data as time-stamped data. We create tools to help orgs successfully prepare and mine their data quickly. We emphasize on speed so the road to accuracy is shortened.

    There’s a lot of open source help if you’re choosing the Hadoop path. But the first step is a strategic plan and wim’s questions are spot on.

    Like

  12. I’m a fan of asking good questions, so #1 is right on the mark to me. So many businesses start flailing around before they even know what questions they want answered. It’s true in all business processes, but especially when you’re trying to analyze large amounts of data.

    The other thing your post made me think of is the way we display signal vs. noise. I’m finding the new infographic craze to be slightly annoying, but it does reinforce your point about having tools to connect the dots and see the pattern. The problem with the glut of information on the web these days (social, search, or otherwise) is that we really don’t have good tools to interpret all the information. Bar graphs or pie charts are one dimensional, and data visualization techniques that are starting to be used more (scatterplots, etc..) are still fairly primitive. David McCandless has a good starting point in his Ted talk from August 2010… http://www.ted.com/talks/david_mccandless_the_beauty_of_data_visualization.html

    Like

    • Hello Joe,

      Yes, that’s an excellent video. Thx for bringing it up here. And as much as I do agree that visualizations can play a big role in better understanding what the data is representing/telling us, I believe this ‘technique’ is used extremely poorly most of the time. I hope that’s also a matter of time..

      Thx for the read & shout. Hope to see you back soon.

      Like

    • Signal vs. noise is indeed a problem, but with advanced statistical tools there is much to be discovered in the data. Presentation of the results is very important, but don’t let us forget about the skills involved before we are able to acquire these results. Analyzing big data requires very specialized people (‘data scientists’) with experience in statistics, mathematics, programming and data mining.

      Most of the problems outlined in this post can be solved with this skill set. This differs quite much from old world business problems, where common sense and some pie-charts would be enough to have success in business analytics. We saw this shift in finance in the eighties/nineties when we went from common sense to big financial models with the rise of the availability of financial “big data”, driving quants (mathematicians and physicists PhDs) to finance firms. in my prospect we are having the same development in marketing/business analytics right now with ‘big data’.

      Like

      • Definitely agreed that you need specialized people who look at data. I have a friend who’s entire job is to do quantitative analysis for a healthcare system, looking at different neighborhoods in his city and what type of health issues arise and care is needed. I think it’s clear that having someone who really understands data means you can’t just use any old person at your company. But, I do think that it’s just as important to have a creative hand involved, since using new models and thinking about things from a variety of angles may reveal advantages that wouldn’t otherwise be obvious. For ex. the NYTimes big data project where they create data visualizations around audience participation with their content. Some of that just looks like fun, but if you examine the way they present it there are trends which can be used in concrete ways to further business / increase revenue.

        Like

  13. Pingback: How To (Not) Get Smart About Big Data | CMO_On_demand's Blog

  14. For many companies, “Big Data” can simply mean the myriad of customer transactions that are currently independently managed by several external agencies and internal groups, even before we factor in social media. Until brands learn the discipline of managing this level of data to better understand their segments and the unique value propositions needed to command preference, social media data will be overwhelming. As Wim suggests, it must start with a POV and intentional strategic questions around serving customers better, not a data fishing expedition.

    Like

  15. Yes, really agree with the practicality of these suggestions. I’m really drawn to the last one. How much of this really, really needs to be real time, because real time add’s huge complexity and cost to the project.

    Like

  16. Really excellent post Wim. The fact that unstructured data coming from social world is quickly increasing doesn’t mean you need to mine every information. Time and costs are always the main variables that guide (force) your business so, even knowing that some strategic choices could take to a loss of information (as it always be), it’s mandatory to set a strategic perspective on big data management elaborating realistic informative objectives that have to be aligned to the business ones. As Esteban said probably we need to set a framework that can guide us to set new filtering techniques and approaches useful for our purposes.

    Like

  17. Pingback: Big Data: Too Big? Or, Too Noise? |WITS ZEN

  18. Nice post, thanks for sharing.

    I fully agree that the start of any (good) analysis should be defining the questions you want answers to. And though it’s ‘sexy’ to use social media data these days, not always this is the right source to start from.

    @ Esteban: I thought that by Big Data, people mean social media data. Do you feel there is a difference?

    Like

    • I am not sure yet if there is a difference, but I see that while we are consumed with Social there are other sources (mostly due to mobile and cloud coming of age) that should not be discarded as part of Big Data. The concept is far bigger than just one set of channels as I see it.

      Like

    • Hi Rob,

      Thx for the compliment, the visit & the question. Please also see my answer to Esteban’s question in the top comment.

      Thx again & hope to see you back soon.

      Wim

      Like

  19. Wim,

    First, as always – thanks for quoting me. You humble me.

    Second, I like where you are going and what you are saying, but i just have one question — are you saying social data when you talk about big data? is there a difference in your opinion between the two?

    interesting indeed.

    Like

    • Hi E,

      In my opinion Social Data is a subset of Big Data. Big Data is also ‘circumstantial’ data that is created by e.g. the channel or device through which interactions take place. Think the internet-trail, mobile/location data, speed of movement, type of connection, interactions with other devices etc..

      All these types (and more) of data can help assessing the context in which events/interactions/transactions takes place, and sometimes can be used to provide additional services (please say you noticed I did not use: to sell additional services…;))

      Juts for the record: you should plot big data on hype-cycle as well ;) Early stages I say.

      What do you think?

      Like

      • I think social data is driving Big Data. And once mobile technology becomes more ubiquitous the volume of valuable insights and noise will increase, as well. Of course, this goes back to your point about the importance of filtering noise.

        Thanks for sharing.

        Like

Comments are closed.