An Insider's Tip to Understanding Statistics

• One thing that people often misunderstand about statistics is that every statistic means something only in comparison to something else.
• A different comparison point changes how you perceive the exact same statistic. For example, you may see the glass half-empty if you compare it to a full glass or half-full if you compare it to an empty glass.
• Some experiments, such as randomized controlled trials, allow researchers to create a good comparison group, which they can use to get a sense of how well an intervention works.

I try to teach statistics at least once a year. I absolutely hated statistics as an undergraduate and now that I’m on the other side of things, it brings me great joy to help students appreciate (or at least not despise) the subject. This term, I’m not teaching the class. So lucky you, you get to hear some of my thoughts on the subject.

Every semester, I ask students what they’re hoping to get out of the course and they inevitably discuss this idea that people out there are trying to fool them with statistics. That people are maliciously tricking them at every corner and that somehow this class should help them navigate the trickery. I’ll say this. There are certainly people out there who are up to no good with statistics. But for the most part, I don’t think this is driven by malintent. I think it’s driven by a lack of understanding of one key thing. And most of what feels like trickery on the part of someone else is often due to a lack of understanding in the recipients (that's you).

This one skill is the thing I think will make you a better consumer of all scientific literature—from basic readings like journal articles all the way down to the headlines of a popular press article. The bonus is that this skill requires absolutely no math at all. At least, not the kind of math you do with a calculator. Instead, it takes logic.

The secret is this: Every statistic means something only in comparison to something else. Recognizing the comparison point, and eventually developing the skill of picking the best comparison point, is the secret to understanding statistics like a pro.

The Comparison Point You Use Can Change the Statistical Story

Most of the time, people read over statistics without realizing there is an implied comparison point. Think about the classic question about a glass of lemonade—is the glass half empty or half full? Either way, you are comparing the current standing of the lemonade—halfway up the glass—with something else. You might not realize this is what you are doing but the comparison point you use is what defines your answer. If you think of the glass as half full, your comparison is an empty glass. If you think of something as half empty, your comparison point is a full glass. It’s the comparison point that changes how you perceive the exact same statistic.

Over time, you’ll realize other people are using different comparison groups. Learn from these differences. Someone else’s comparison point isn’t necessarily wrong but it is going to tell a very different statistical story. Are they pulling one over on you? Maybe. But there’s also a very good chance they don’t realize their comparison point. They see the glass as half full without realizing why. You see it as half empty without realizing why. Having a conversation about which comparison point you are using is much more likely to be productive than arguing about your end evaluation. And of course, the glass can be both half empty and half full because both comparison points could be valid.

Making Sense of Statistical Significance

Here’s another insider tip for you—a bonus tip: When scientists talk about something as "statistically significant," they are making the conclusion that a number is different than something else. Let’s say I’m trying to increase people’s motivation to save money. And I develop this brief reflection exercise that I think increases motivation to save money. I can’t just deliver that intervention and say my sample has a motivation score of 7 out of 10 and that’s high so my intervention must be good. (Well, I can try to say that but it’d be bad science and usually wouldn’t be published thanks to peer review).

Instead, I need to have a good comparison. The best comparison would be people just like the ones I give my intervention to. Some of those people will probably have pretty high motivation and some pretty low. They give me a good guess as to what the world would look like without my intervention. Then I can use this group as my comparison point. If my group gets a 7 out of 10 and this comparison group has a 6 out of 10, I can still say my intervention probably increases motivation but I have a much more cautious estimate of how much.

Motivation Essential Reads

Overcoming the Fixing Reflex

The Most Underutilized Form of Motivation

What I’ve just described to you is an experiment—a study with a control group (the comparison group) and a treatment group. Randomized controlled trials like the ones we have seen for testing vaccines and COVID treatments have used this approach. It’s the gold standard for research because of how well it allows you to create a good comparison group. When you read scientific research, look to see what their comparison group was. Was it a good one? Were the groups as similar as they could be except for one thing they wanted to change? Understanding that scientists are making comparisons will help you start to understand their findings without needing a headline or an interpreter.

Recognizing the right comparison group won’t necessarily come immediately to you. And oftentimes, there will be multiple correct groups, which I acknowledge is exceedingly frustrating. I’m sorry if I’m the first to tell you this, but the world is complex. We have to embrace the fact that the same thing can mean two different things. I don’t yet have a secret to tell you exactly how to decide what comparison group to use. But start by spending some time considering which one you are using. Then consider seriously an alternative comparison. Slowly, you should develop the skills to discriminate between the two. Not which one makes you happier or helps you make your point better, but which one leaves you with a stronger understanding of the statistic itself.