Breaking the Rules of Science: Where's the Proof?

Last fall, I was working with a student to analyze some data from an experiment we had previously conducted in my lab. The ideas we were testing were pretty cool, but I could never find time to prioritize running the analyses on that particular dataset. The student retraced our steps, learned about the methods we had used, cleaned and organized the data, and then eventually we were ready to run the real numbers.

We had two hypotheses. The first one—the far less exciting one—was clearly not supported. But then we typed up the code for the second. We ran the code. The software program I use for analyses shows this little blue spinning wheel as it’s crunching numbers. The wheel spun and spun. It must have taken less than 10 seconds, but you know how accustomed we are to instant results these days and that time felt like forever. And then the output popped up. In less time than it took the stats to run, I saw the two critical things I needed—a number that exceeded my threshold of ‘significance’ and a figure that showed my results had the pattern I expected. My fist punched the air with excitement and I practically jumped out of my chair. The audience of my student certainly contributed to the emotion of the moment, but the truth is, moments like these are why I do what I do. To get statistical confirmation of an idea that I had. To have a thought, test that thought, and then find out I was right.

Here’s the catch. The hypothesis I developed and tested was supported. But I wasn’t really ‘right.’ I found evidence that I’m right. This tiny little nuance—the difference between being right and having evidence for being right—is the crux of a scientific rule that you as a consumer need to know to up your understanding game.

The rule: Science almost never proves anything.

Let’s go back to the moment above—we found statistical support for our idea. One version of one test at one moment in time suggested our idea was correct. But not all versions of all tests in all moments. Science always leaves the door open for more understanding. There might be another way of testing my idea that finds weaker evidence or that suggests my idea is correct in some circumstances but not others. Or my evidence could have been a fluke—a randomly strange sample. Scientists learn this rule when we take statistics classes. Our statistical evidence is almost always based on rules of probability. We use statistics to determine that something (support for our idea) was unlikely to have happened by chance. Which means it might have happened because our theory is true. We have evidence for our idea, but not proof. Better research methods help us be more confident in the evidence, but in the end, it still comes down to probability.

Now, science journalists are not often well-trained in statistics. Add to that the excitement you observed me experiencing at one tiny finding, and it’s easy for this tiny nuance—the distinction between evidence and proof—to get blurred. Whenever you’re reading about the results of a scientific finding and either ‘proof’ or ‘proves’ shows up, consider it a red flag. There are a few instances where proof might be an appropriate phrase but in general, this is a good hint to step up your critical thinking when you’re reading a piece.

The Catch: Science is still the best we know.

Today’s rule is one I thought I would never tell you to break. It’s been one of the cornerstones of my teaching on statistics. But the thing is, we do need to break this rule. Gently, at least. Science is designed to chase the rainbow, not necessarily to get to the end. Even though we might not definitively ‘prove’ anything, science offers the closest we can get to proof.

Scientific recommendations will change as evidence accrues, which means we have to accept the scientific 'proof' we have as temporary. Clinging to an old conclusion when new or better or more evidence suggests otherwise because you’re holding fast to the notion that science doesn’t prove anything is foolish. Take mask wearing. I sat on my high horse and rolled my eyes at the people who wanted to wear masks back in March and early April. Because at the time, scientific recommendations were that wearing masks might not be helpful—or might even backfire—in preventing the spread of COVID-19. But then we learned more about the specific context of how the virus spreads. We also realized that people can mitigate one of the biggest downsides of masks with better handwashing. The body of scientific evidence changed.

Are masks proven to reduce the spread of COVID-19? No. But the evidence is very clearly in their favor. I had to stop defining good science by what I wanted it to say and be intellectually humble—to realize that the scientific facts I cling to every day are subject to revision. I had to acknowledge that even in the face of no proof, science was giving me an answer. The smallest, least inconvenient thing you can do right now to reduce the spread of COVID-19 (and its economic impact) is to wear a mask whenever you are indoors in public. The science could change on this again but it’s unlikely because the overall amount of evidence we have is much larger than it was at the end of March.

The Take Home Message(s): First, consider the word ‘prove’ in writing about a scientific finding to be a red flag. Turn on your critical thinking booster whenever you read it. The entire source may not be discreditable (a lot of very solid sources misuse the word 'prove') but you'll want to make sure to be convinced by the evidence, not the excitement.

Coronavirus Disease 2019 Essential Reads

Update on COVID-19 and How It Changes the Brain

How Political Partisanship Can Be Deadly

Second, remember that even though science is not definitive, it is the best knowledge we have. I used a really important phrase above—intellectual humility. The best way to accept science is with celebration of its strengths and acknowledgement of its weaknesses. Accept the imperfect proof as the best we know. Piles of evidence still aren’t proof, but they’re worth listening to.