The Big Myth of Data

Good decision-makers are driven by the data, experts tell us. They cast their preconceptions aside. Indeed, as the venerable Sherlock Holmes famously uttered in Scandal in Bohemia, "It's a capital mistake to theorize before one has the data." Then, with clear eyes, the most rational of decision-makers follow the evidence wherever it may lead.

It sure sounds like good advice and, in an era where we are inundated with data at every turn, seemingly more relevant than ever. But is it?

Frankly, upon closer inspection, it's not even clear, really, what the guidance even means. Here I am, sitting in front of a large spreadsheet, wondering: how could this lifeless matrix filled with inert values ever take me anywhere? I can't download it directly into my brain like I do apps onto my iPhone. Nor can I scroll through it, absorbing its contents like some hacky speed reading infomercial from the 90s.

No, instead I must decide. I must choose which cells to look at, which queries to write, and what charts and graphs to generate. None of those decisions are driven by the data; they're driven by me. More specifically, they're driven by my theories and guesses about what is happening in the world behind the data. Guesses that originate in my own mind. Indeed, humans are only ever able to make sense of data in light of their existing guesses. As the philosopher Karl Popper pointed out, all observations are "theory-laden."

Another way to reveal the incoherence of the follow-the-data edict is to realize that the very same data are regularly used to justify completely different conclusions. A Republican and Democrat look at the same gun safety or abortion or COVID policy and walk away with polar opposite assessments. But, crucially, it's not just political partisanship that accounts for these varying perceptions. Even two sober, well-meaning scientists look at the same experimental results and commonly come away with wildly different interpretations of the intervention being investigated.

The truth is this: being data-driven is impossible. We can't help but view the world through the lens of our pre-existing theories. Which leaves data with a more humble, subordinate role. Its job is to rein in our theories, to help us fact-check them, to spot the errors they contain. As the neuroscientist Anil Seth puts it in his book Being You, "Perception happens through a process of continual prediction error minimization."

Therefore, when someone proudly announces they're "data-driven," what they often are, in actuality, is "theory-blind." They fail to appreciate that it's not the data in the driver's seat but rather their own theories.

Which is a dangerous place to be. Because if their theories happen to be wrong, and they often will be, they have no power to criticize them. No power to change them.

And that's a shame. Because it's the ability to be flexible, to conjecture alternative theories, and then view the data in light of each one of them that is truly the path to sound decision-making.