The Mirage of Data

I wrote this post in 2014 about Big Data when it was a buzzword. Now it is AI. The promise and the hazards are the same: the answer is in the data. With AI, and the mirage is more convincing than ever, so it becomes more critical that we understand the limitations of data as they will apply to anything AI produces.

New technologies are giving businesses access to larger amounts of data and tools that help us analyze data faster, in larger volumes than ever before. It feels empowering in our quest for the answers. We want data to illuminate the way for us.

Data enables us to make the right decisions and to bulletproof our plans – Or does it?

“It is possible to be drowning in data and still none the wiser” – Paul Laughlin

Back in the 90s, one of the World’s most innovative companies was about to launch a massive project. They invested in a research project as large as they come, hiring three of the top consulting companies at the time, and spending 8 figures to interview 200,000 people in 54 cities and 22,000 individuals at 3000 corporations. All this research and data should have guided the project and ensured success, right?

The company was Motorola and the project was Iridium (basically the predecesor to Starlink), which went down in history as one of the most colossal failures in business and resulted in a $5 billion write-off.  What happened? Motorola had access to experts, technology, and reams of data. Nevertheless, they failed to ask the right questions and to interpret the data correctly.

“A great part of information obtained in war is contradictory, a still greater part is false, and by far the largest part is of doubtful character” – Carl von Clausewitz

We are drowning in an ocean of data

When I first wrote this post back in 2013, Big Data was an overhyped term with a questionable path to business value. It was rhe buzzword of the day. Companies were investing millions in technology to collect more data than ever, and most of them could not explain what they would do with it.

A decade later, the hype has a new name: Artificial Intelligence. Companies are collecting more data than ever, and AI promises to finally make sense of it all. The tools have changed but the fundamental mistakes have not.

“There is an unlimited amount of data in a digital world. Many times people try to do the most complex way of using the data.”
– Rishi Dave, CMO, Dun & Bradstreet

A CMO survey found that only 6% of marketers have the talent needed to leverage marketing analytics. Most marketers are not taking advantage of the data we have today, web analytics being a prime example.

Back then  ReadWrite published an article appropriately titled “Gartner on Big Data: Everyone’s Doing It, No One Knows Why”, states most enterprises surveyed still don’t know what to do with the data they are collecting.

More data is harder. The value is not really in the quantity of the data, but on knowing what to ask for and deriving the right insights. We are entering an age where organizations will find they are rich in data but poor in customer-insights.

“But of course, the quantity of data is entirely beside the point if the data aren’t of a good quality.” – Phil Rosenzweig, the Halo Effect

The danger is all this data can be creating a Mirage

The mirage is that the answer is in the data, that more data is better, that if we collect enough data and analyze it we will arrive at the right answer.

A couple years back, Mohan Sawhney, one of the most brilliant marketing academics from the Kellogg School of Management, gave us a problem to solve. The case study included a couple data tables and multiple data points: prices, costs, volume, market research, etc. Everyone in the class, myself included, fired up our spreadsheets and came up with a couple conclusions – all wrong. Our mistake was assuming that because we had a lot of data, the answer must be in data analysis. The answer was in front of us the entire time. There was no need to analyze the data to find the answer, because the answer was found in basic marketing strategy. The data was a trap.

“A flood of data should never be allowed to wash away your common sense” – Jack Trout

The 7 Hazards That Create a Mirage of Data

1. Data only looks at the past. Despite conventional wisdom, data is limited as a predictor of the future. Otherwise, it would be easy to predict stock prices, sports scores, or even lottery numbers. By the time you collect and analyze data, the environment will have changed, limiting its usefulness.

2. Data does not tell you why and fails to reflect emotion. It is so easy to confuse information with evidence. Modern management practices focus on quantification of all the elements of a business. We are all human. We make emotional decisions and then justify them rationally. From your morning Starbucks, to the car you drive, to billion-dollar decisions in corporations, it is all driven by emotions. You cannot fully understand customers through a numerical lens. Data can show us what customers purchased, how they paid, and how often they visited. It cannot tell us why.

As Harley Manning and Kerry Bodine tell in their book, Outside In, “We recently spoke with the head of consumer insights at a major cosmetics manufacturer who was building a repository of shopper purchasing patterns from retailers PoS systems. Although he insisted that he wanted to create a ‘true 360 degree view’ of his customers, he wouldn’t even consider our suggestion to study how women actually use his company’s products in the bathroom every morning. Women of the world should be outraged that a guy in a suit has the audacity to think he understands them based solely on when and where they shop for makeup.” (P.s.  I have lived with my wife for many years, and still don’t understand her fully, or her cosmetics buying process).

HBR notes “Human behavior is nuanced and complex, and no matter how robust it is, data can provide only part of the story. Desire and motivation are influenced by psychological, social, and cultural factors that require context and conversation in order to decode. Data …reveals what people do, but not why they do it.” In other words, trusting data alone assumes the data you have is accurate, complete, and reflects or captures all relevant aspects of consumer behavior.

3. Data is always biased: Every step in data collection and analysis has potential for introducing bias. Your data is most likely biased by the method you use to collect it, the customers you collect it from, the data available to correlate it with, and the way you interpret the data. Psychologists tell us we are all victims of confirmation bias, giving more value to the data that confirms our hypothesis and ignoring the data that contradicts it.

Iridium failed to ask if people would pay for the service at a particular price. That was their bias. Surveys are biased by the subset of data you survey and the subset of those who respond. Your transaction data is biased based on your existing customers, which may not accurately represent the entirety of the market. You are blind as to why non-customers fail to buy. Collection can result in a self-fulfilling prophecy: display the most popular products on your website, make them even more popular, not because they are the best products, but because you chose them as such.

4. We often look at the data we can collect, not at the data we need. Not all data can be useful. Not all data is interesting and most is not insightful. We tend to assume the data we have (or what we can collect or observe) must have the answer to our questions. The weight of a television set has nothing at all to do with the clarity of its picture. Even if you measure to a tenth of a gram, this precise data is useless. The number of Twitter followers a person has is probably not a good indicator of actual influence, even if it is easy to measure. It takes guts to stop measuring things that are measurable, and even more guts to create things that don’t measure well by conventional means.

An online retailer can focus on conversion rates (percentage of visitors who buy), while being blind to the fact that larger group of buyers have a lower lifetime customer value (LTCV). Conversion is far easier to measure than measuring and correlating with LTCV.

5. Data makes it easy to confuse correlation with causation. In search of an ROI story, social marketers are quick to point out the higher customer value of those who follow a company on social networks, implying they buy more because they follow them. However, it is more plausible that they follow the retailer on social networks because they are loyal customers. Without understanding the why and the emotions behind people’s decisions, it is hard to properly interpret data. Your data shows a customer purchased a large ticket item and a small ticket item, which one was the original reason why he showed up at the store? Which one was the impulse buy?

6. The delusion of a single explanation. It is common to look for the (singular) root-cause for a problem – to try to find the one thing that drives customer buying decisions. The reality is that the way people make decisions is far more complex. Dozens of factors have some influence on a single decision. It is like proposing marriage: would you be able to tell why she said ‘yes’? Was it your looks, your money, your smile, or your sense of humor? As much as we would like a simple answer that clearly points to a single explanation, chances are events and decisions are results of much more complex, interdependent factors.

7. More data (or big data) is not better data. You can survey thousands of people and ask them who invented the light bulb, which will result in a false sense of security in perfectly inaccurate data. We tend to believe that more data will give us more precision, when it usually has the opposite effect. It is easy to draw a chart with two axes and it is possible to compare data using three dimensions. Adding a fourth, fifth, or sixth dimension makes data visualization an impossible problem in itself. With more information and more data points to correlate, we are forced to ignore or minimize large sets of data, increasing the chances of arriving at the wrong conclusion. Malcolm Gladwell’s Blink makes a pretty good case for the accuracy of gut-based, split-second decisions over those made after more careful and pseudo-scientific evaluation.

The Mirage gets stronger with AI

Imagine Motorola planning Iridium with today’s AI tools. They would have fed all 200,000 interviews into a model and asked it whether the project would succeed. The answer would have come back in seconds: well written, confident, and backed by data. And probably just as wrong.

AI does not eliminate the mirage of data. It makes it more convincing. A spreadsheet at least forces you to do the analysis yourself. AI hands you the conclusion, fluent and polished, and it is very easy to mistake fluency for truth. After studying how intelligent AI systems are built, I am convinced that every one of the seven hazards gets stronger, not weaker, with AI.

AI only knows the past. Every AI model is trained on data from the past. Ask it about your market and it will tell you what was true, not what is about to change. The disruptions that matter most are, by definition, the ones that are not yet in the data.

AI cannot tell you why. AI can summarize a thousand customer surveys in seconds. But it never saw the wrinkled nose of the diner or heard the sigh of the shopper waiting in line. It knows what customers said, not what they felt. And as we know, people make decisions emotionally and justify them rationally.

AI amplifies our biases. AI inherits the biases of the data it was trained on, and adds a new one: it tends to agree with you. Ask a leading question, and you will get a well-argued answer that confirms what you already believed. Confirmation bias used to require some effort. Now it is instant.

AI loves a single explanation. Ask an AI why customers are leaving, and you will get a clear, well-structured answer. Real customer decisions are messy, driven by dozens of interdependent factors. A neat answer to a complex question should make you more suspicious, not less.

AI makes more data feel manageable. We used to be overwhelmed by too much data. Now AI promises to process all of it for us. But to summarize, a model has to decide what to ignore, and it does not tell you what it left out. The danger is no longer drowning in data; it is trusting a summary we cannot verify.

The most extreme version of the mirage is synthetic data: AI-generated customers, AI-generated survey responses, AI-generated personas. They look real, they sound plausible, and they are built entirely from averages. It is insight about customers who do not exist. (I wrote more about this risk in 8 Cardinal Principles to Use Personas Effectively in Marketing.)

To be clear, I am a believer in AI. I use it every day, and it makes me better at my job. It can find patterns in data no human could see, and it can help us process real customer conversations faster than ever. The problem is not the tool. The problem is believing the answer is in the tool.

What to do, if data can create reality distortion?

The best insights come from customers, not spreadsheets or analytics

Sometimes we look for technology, sophisticated data models, complex analytics and expert advice to tell us more about customer behavior, when it would be easier to simply get out of the office and listen to them.

The best insights come from observing and listening to customers. If you make it a habit to speak with a customer or three, every day, you will gain insights that no computer technology can give you.

Bernadette Jiwa captured it beautifully: “We assume that the most valuable data is static and lives on graphs and spreadsheets….The truth of what we need to know and some of the most valuable data live in plain sight. The wrinkled nose of the diner. The sigh of the shopper waiting in line. The posture of the customer as she walks out the door. What she packed in her bag before she left home this morning. Noticing what people do is often more valuable to us than listening to what they say they think“

I am a believer in the power of Big Data technology and the business value of insights. I have seen the success stories first hand. What I am pointing to is what Gartner calls the peak of inflated expectations, that is almost always followed by a ‘through of disillusionment’, that there is a danger of following buzzwords and trends blindly, and the danger in relying purely on metrics and analysis for decision making.

Understanding the limits of what data can tell us, identifying the challenges in interpreting data, and being deliberate at capturing simple but very useful data can lead to insights and then to action. Taking advantage of data, along with non-data derived insights, results in better decision making. If he had relied purely on data, Steve Jobs would have never considered building the iPhone (click on the link for the story).

It would be a misunderstanding to conclude I am suggesting marketers ignore data. To the contrary, data is fundamental for every businessperson. Data can be especially useful for marketers. The point is to be aware of the limitations and traps of data, to better comprehend its usefulness and to extract real value from the data we have.

To read more on mirages that affect businesses, I may recommend reading The 9 delusions to high performance businesses

---

Another very relevant quote from Kenichi Ohmae “The mind of the Strategist” (published in 1982):

“Successful business strategies result not from rigorous analysis but from a particular state of mind. In what I call the mind of the strategist, insight and a consequent drive for achievement, often accounting to a sense of mission, fuel a thought process which is basically creative and intuitive rather than rational.

Strategists do not reject analysis. Indeed they can hardly do without it. But they use it to stimulate the creative process, to test the ideas that emerge, to work out their strategic implications, or to ensure successful execution of high-potential wild ideas that might otherwise never be implemented properly.

Great strategies, like great works of art or great scientific discoveries, call for technical mastery in working out but originate in insights that are beyond the reach of conscious analysis.”