Data-driven is not scientific

A Wake-Up Call

“What correlates most with willingness to recommend? This is the satisfaction with our hotline service” says David - Head of Insights at Sonos at that time.

His gut instead raised a red flag, which is why he handed over the whole dataset to us. It turned out that all numbers and correlations had been correct, but for a different reason than you would think.

New customers tend to be most excited in the time right after purchase. But in this time customers also reach out to hotline service for installation help. Hotline-satisfaction is influenced of the early enthusiasm, not caused.

A lottery company wanted to learn how to attract customers. What they believed they knew, was that the lottery target audience was seniors. That’s obvious if you see that more old people play lottery. But it turned out that instead young audiences are the receptive target group. Older people are less likely to start playing. The data and the correlations are still true but for different reasons.

Playing lottery is a habit. The more you play and the more you win, the more loyal you are. Older people simply had more time to habitualize.

I can write a book about such examples that report the same message: Data is not what we are looking for.

Instead, what 99% of market research projects do is just that. Get data. Twist it. Turn it. Paint it.

Instead, what 99% of management dashboards are doing is just that. Show data. Slice it. Dice it.

Instead, what 99% of FP&A and controlling sheets are doing is just that. Show data. Compare it. Benchmark it.

We are in the middle of the scientific area of human kind. It began slowly with Kant and Newton and picked up speed with the industrial revolution. In essence it is grounded in and enabled by the mindset that measuring things, that generating data, will enable us to understand and finally manage the world.

The tricky thing is that this mindset brought us far. It works well in physics and chemistry, engineering and software development.

But it fails when we need to manage complex systems: subjects that has many influencing factors, time-lagged cause effect cycles and lots of unknowns about all of this actually works together.

Welcome to Marketing.

It always struck me why smart people opt to study physics and math, while the true frontier of insights lies in social systems. I studied electronic engineering and experienced it firsthand: Early on I added marketing and social science to the mix. Early on I realized: Hard science is complicated but not complex.

To be able to start understanding social systems like markets, to be able to unearth the secrets of marketing, we first of all need to understand a fundamental lesson:

Data is not insights.

What you are measuring with an antenna is a signal, not the music. The signal is modulated on magnetic waves. It comes from everywhere. From the universe, from hundreds of radio stations. But on its own it is useless. Because it is noise.

This signal is like the data we look at. First of all, it is noise. When we look at our management dashboards, at the latest sales data or a market research report – you should be clear: its just data mostly covered with noise.

We need a receiver that knows the frequency that needs to be selected to get in resonance with the one promille of data that is information.

Like clouds in the sky. You can find objects if you want to. But nobody drew them. It is noise. No information.

As a corporate manager I supervised some sales reps and had monthly performance reviews. I ask Juergen “why does client X don’t buy that much anymore”, immediately he had a great explanation. “Oh, Juergen knows his market I thought” I thought. Suddenly I realized that the dashboard was tuned to the wrong year. The truth was that Client X was buying more not less. It took Juergen five seconds to give a great explanation.

As managers -day in day out- we look at our dashboards believing to spot information. In truth we stare into the sky, seeing “elephants” appear and go.

The mantra of modern manager generation is to be “data driven”, because everything else is “guesswork”. This belief structure had its role in a world of pretenders and snake oil salesmen. But when we are asking for deeper answers, this mindset is hitting a wall.

We hit the wall like a fly hitting a window glass. Bam, bam bam.

Everyday anew, “data driven” managers hitting the “window glass” – like flies. What do they learn?

Oh, we just need to try harder. Get more data, drill deeper. We hope that then finally we may surface the truth.

But they just fly faster against the wall. What is happening is that the fly will sink down one day and settle next to the other flys – the “data driven” companions.

“Data driven” is not even scientific.

Science aims to surface generalizable objective truth. When you find your market share is 20% and shampoo buyers agree to like apple scent by 45%. This is not a generalizable objective truth. It’s a boring singular fact. It has marginal use. Its usefulness has a life span of a verification code hitting your inbox.

A generalizable truth instead is e.g. to understand that beer buyers are in search for refreshment and that products fulfilling this need will sell more.

Science is interested in relationships; it is interested in cause and effect. It wants to answer the “why”.

This answer is not data. It cannot be measured. You need to infer it. Truth is hidden in the data - at best. Truth is the signal, data mainly noise.

“At best”, because you need data about most relevant factors and you need to know about the meaning and context of those factors.

What to do?

Randomized Controlled Trials are the scientific gold standard of research the truth. Even this method is tricky.

But in Marketing with have a problem with “Controlled”. In marketing we mostly do surgeries on a pounding heart. We can not set it still. We can not set all other factors constant.

What we can do is to measure what is happening outside of our test actions (=randomized trials). With this data we can run a causal analysis – special multivariate modeling exercise that can attribute outcome data back to its causal source.

In lack of solid hypothesis on how things relate (which is the default mode of nearly all practical cases) it is advised to use Causal AI not traditional causal analysis.

“Data driven” is not scientific,
causal modeling is.

Science is interested in the inner structure, the inner clockwork of the world. And it needs data to proceed. The idea that because you handle data makes your work scientific is more than naiive. Its like this toddler playing with hand grenades.

The more potent the toys are your play with the more you need to level up your know how. Otherwise it gets dangerous.

This are table stakes for sharp knifes or the gun everyone of us has in his glove compartment (😉). Its not yet for data.

So, next time someone tells you they’re “data-driven,” ask them what truth they’ve uncovered. Not a number. Not a chart. Not a dashboard.

Truth.

Because data is not the destination.
It’s the fog you must learn to see through.
And only those who dare to question, to infer, to “feel” beneath the noise - they are the ones who lead.

Not by numbers.
But by understanding.

Be insight-driven.
Or be lost in the data.