The Mirage of Data

The Mirage of Data

Big Data and data-driven marketing are two of the most prominent business buzzwords.

New technologies are giving businesses access to larger amounts of data and tools that help us analyze data faster, in larger volumes than ever before. It feels empowering in our quest for the answers. We want data to illuminate the way for us.

Data enables us to make the right decisions and to bulletproof our plans – Or does it?

[quote]”It is possible to be drowning in data and still none the wiser” – Paul Laughlin[/quote]

A couple years ago, one of the World’s most innovative companies was about to launch a massive project. They invested in a research project as large as they come, hiring three of the top consulting companies at the time, and spending 8 figures to interview 200,000 people in 54 cities and 22,000 individuals at 3000 corporations. All this research and data should have guided the project and ensured success, right? Continue reading “The Mirage of Data”

Does Your Marketing Formula Pass the Correlation Test?

Math FormulaFinding the return on investment for marketing activities is like the holy grail for CMOs. There are countless articles wit ideas on how to determine social media ROI, content marketing ROI, video marketing ROU, and so on. We all have heard ‘I know 50% of my marketing spend is wasted, but I don’t know which 50%‘.

Every marketer has a hypothesis of what works. Otherwise, how could you do your job, You have one, right?

We should keep in mind, marketing leaders are not always looking for a mathematical formula, we are often looking for something more basic, more essential. We need to answer the questions: is this working? Should we do more? Could we have the same results with less effort? How do we know?

This is not a post about how to determine ROI (for my POV on social media ROI you can check my slideshare from 2009). This post is about how to make sure you have found a formula that works. Success without repetition is luck. Good marketers know what works, and can have predictable success.

There are many challenges that make it very hard to understand what marketing activities work. Here are three:

  • The Delusion of a Single Explanation: We would like things to be simple. Marketers would try to find correlation between two factors: satisfaction score and revenue growth, attending a webinar and purchasing, etc. In reality, most  outcomes depend on a multitude of factors.
  • Confusing Correlation and Causation. Maybe we have observed people who visit a webinar buy more from us. But do they buy more because of listening to the webinar, or did they decide to listen to the webinar because they wanted to buy more anyway? Phil Rosenzweig, author of The Halo Effect (great book, by the way), observed the consulting company Bain & Co claims on their website “Bain clients have outperformed the stock market 4 to 1” implying working with Bain leads to better performance. It is very possible that high performing companies are the only ones that can afford to hire Bain.
  • Customers buy on emotion. Emotions are hard to understand and to map to a formula (anyone who is      married knows that), how people buy cannot be put in a spreadsheet. But marketer will try anyway. It is almost impossible to understand what are the customers intentions. A UT professor shared with our class ‘I saw someone at Costco in the checkout line with a dolly on which he had two milk jugs and a 50″ plasma TV. Did he go to Costco for the milk and when at the store  decided to buy the TV or the other way around?’

How do we solve this problem of correlation versus causality? How do we know what we think is working actually works? How do we avoid falling in the traps of market research?

I found one interesting point of view from another industry: the Bradford Hill criteria is used to determine cause and effect in medical tests. If this is the method to determine causation in an industry where being wrong can be the difference between life and death for many people, I figured it is probably good for us to learn from. These are the Bradford Hill checks to make sure your assumptions of cause and effect are correct:

    1. Consistency. “One apparent success does not prove a general cause and effect in wider contexts. To prove a treatment is useful, it must give consistent results in a wide range of circumstances.” In your marketing test, are the results consistent when you try the same technique in different situations?
    2. Plausibility. “The apparent cause and effect must make sense in the light of current theories and results. If a causal relationship appears to be outside of current science then significant additional hypothesizing and testing will be required before a true cause and effect can be found.” Which to me sounds like thinking – does this result make sense? Is this assumption aligned with what we know about customers and about marketing?
    3. Specificity. “A specific relationship is found if there is no other plausible explanation. This is not always the case in medicine where any given symptoms may have a range of possible causing conditions.”. This is a big one, especially because of the delusion of a single explanation we talked about before. What other variables are in play? Can you rule our any other possible explanations? Is your test specific enough?
    4. Evidence. “A very strong proof of cause and effect comes from the results of experiments, where many significant variables are held stable to prevent them interfering with the results. Other evidence is also useful but can be more difficult to isolate cause and effect.”. Right, test one thing at a time. A/B testing and MVT testing tools are helpful, but you need to be careful to isolate one variable at a time. Things like seasonality and external events are not factored automatically in these tools..
    5. Analogy. “When something is suspected      of causing and effect, then other factors similar or analogous to the supposed cause should also be considered and identified as a possible cause or otherwise eliminated from the investigation.” What other explanations can you find for a customer behavior? What similar activities could influence decisions?
    6. Coherence.” If laboratory experiments in which variables are controlled and external everyday evidence are in alignment, then it is said that there is coherence.”. This point encourages us to experiment with the conclusions outside of our testing environment. Examples to check for coherence in marketing: If customers seem to react to certain messages online, test the same assumption offline. Testing theories in different geographical markets, for example, or validate what you ‘learn’ in a focus groups in a real world.

Four more things to keep in mind

  • Marketing optimization,  especially online, can easily trick you into optimizing your marketing for      those buyers that will buy quickly, not for the buyers that will take      longer to buy which could produce a higher customer lifetime value
  • Don’t rule out something just because it did not work before. I have heard many times ‘We tried that last year and it does not work’ . Goes back to specificity. This is a false negative. Until you understand why it did not work, you may keep testing. Maybe you can find a way to make things work
  • Measuring direct influence behavior is easier than building a model that measures leading indicators      to understand customer behavior over time that is difficult to observe and measure.
  • As Albert Einstein said: Everything that  can be counted does not necessarily count and everything that counts  cannot necessarily be counted.