It's Always Sunny in Correlationville: Stories in Science

When I started grad school my advisor kept saying funny things. She would talk about how a new dataset fit the "story" that a previously published paper had told. Or about framing our findings, as if we were artists. I kept thinking wait, aren't we scientists? What do stories and framing have to do with anything?

I thought a scientific article was an objective summary of research findings. I figured the data should speak for themselves. The data didn't care who had collected them, so who had collected them shouldn't affect what was said in a published journal article. This seems amusingly naive to me now.

Data don't speak for themselves. I've been to a few painful lab meetings where a new grad student tried letting data speak for themselves. It is ugly. Even interesting data, without interpretation, are boring to the point of being pointless.

So scientists tell stories. Ask any scientist. If you don't tell a story you don't publish. There is nothing wrong with stories, of course, as long as they're non-fiction.

Sometimes, though, the storytelling gets too creative. This is especially true in correlational studies, which usually have a gaping hole in them: They don't determine causation. To tell a good story, one often has to speculate (i.e., make up something) about causation. The researcher's perspective inevitably influences the story they tell.

It's always sunny in correlationville

Recently it hit me: These stories always seem to have happy endings, at least in education. You know those Hollywood movies about the white teacher who comes into a poor school, and at first she clashes with her minority students, but in the end, they lift each other up educationally and morally? Correlational research in education is the same way: The news is good, the problem can be solved, the moral is uplifting, and optimism flows like water. It's always sunny in correlationville.

Optimism is great. But science is a search for truth. The truth is not always pretty. When it isn't--when the pessimists are right--too much optimism can be harmful.

I got to thinking of this because of a new study in the journal Psychological Science.

Chen, L., Bae, S. R., Battista, C., Qin, S., Chen, T., Evans, T. M., & Menon, V. (2018). Positive Attitude Toward Math Supports Early Academic Success: Behavioral Evidence and Neurocognitive Mechanisms. Psychological Science. doi.org/10.1177/0956797617735528

Here's the synopsis you'll find in the press release: If a kid is not doing well in math, it might be because of his attitude. Make him feel more positive about math and he'll start doing better.

The Method

Let's look closer. Study one examined the correlation between math attitude and math performance. 240 kids, aged 7-10, were given a battery of tests including two key measures. One was Positive Attitude toward Math (PAM), in which, Chen et al. write, 'Six questions measured two aspects of PAM, namely, strong interests (e.g., “How much do you like math?”) and self-perceived ability (e.g., “How good are you at learning math?”).'

The second was math problem solving. Again, in the authors' words, 'Children answered 18 simple addition problems, and the experimenter recorded the children's reaction time, verbal response, and strategy report (Siegler, 1987; see Fig. 1a). Retrieval rate was computed as the proportion of trials in which the child correctly answered the problem using a direct retrieval strategy. This measure, which has been shown to correlate with later math achievement, served as an index of the mastery of basic math facts (Geary, 2011).'

There were additional measures including IQ, working memory, anxiety/depression, and math anxiety.

This is a correlational study. As the authors say in their penultimate paragraph: "We could not determine the direction of causal influences between positive attitude and math achievement because of the cross-sectional nature of our study (see, however, Table S10 in the Supplemental Material)."

Yet they also say, in the very next paragraph: "In conclusion, our study demonstrates, for the first time, that PAM in children has a unique and significant effect on math achievement independent of general cognitive abilities and that this relation is mediated by the MTL memory system." In fact, the title of the article is "Positive Attitude Toward Math Supports Early Academic Success: Behavioral Evidence and Neurocognitive Mechanisms."

Education Essential Reads

Which Trendy Teaching Techniques Actually Work?

Why Students Should Resist Using ChatGPT

The words "effect" and "supports" are causal language. They are saying that positive attitude causes math success. Here's why that's great news: Change a kid's attitude and you'll make them better at math. Thus dawns another glorious sunrise in correlationville.

This study does not demonstrate causation. It doesn't involve an intervention. These optimistic conclusions should not be taken at face value because there are other, equally valid, ways to look at the data. Here's a pessimistic take on these findings.

The Pessimistic Story

First, it's a truism that people often like things they're good at. Therefore, we should expect being good at math to cause kids to like math. That alone is enough to explain the attitude-performance correlation. (If that wasn't enough, did you notice that part of the PAM attitude measure asked kids whether they're good at math? How could PAM scores not be correlated with math performance?)

In other words, attitude might not actually have any effect on performance. If this is true, then changing a kid's attitude toward math will not make them better at it. That's more pessimistic, but it gets worse.

Kids are very perceptive about how well they're doing in school, and by age 10 they're good at spotting BS, too. If you tell that math is fun and they can do it, but then they get a low score on their math test, they'll just dismiss you as being insincere. In other words, it is possible that you can't change a kid's attitude toward math, not really, except by making them better at math. So don't try to improve math scores by changing attitudes, because first, you'd have to change math scores.

Now let's get even darker. If a kid has below-average math aptitude, they will tend to struggle with math. This struggle will affect their attitude. That is, kids who are not good at math will grow to hate it. In fact, it is the very strength of the correlation in this study, between performance and attitude, that brings down the hammer on kids who don't do well in math. It's a strong correlation, which means few will buck the trend. In other words, this study shows that very few kids with low math aptitude will ever like math (which seems, anecdotally, true). They're doomed to hate it. (Hopefully, this is going too far. Remember, this is storytelling. But it's consistent with the data.)

Different stories about the data produce different headlines:

Optimist: "Math performance can be improved by changing kids' attitudes toward math!"

Pessimist: "Kids' math performance determines their attitudes, and kids with low aptitude are doomed to hate math."

Here's why it matters. The optimist is going to invest funds into improving attitudes to create a positive cycle. The pessimist is going to give extra math help to kids who are struggling at a young age to prevent a negative cycle.

Conclusion

When we read science, we want to hear the truth. But we also like to see problems (like low math scores) get solved. It's not that we want scientists to tell happy stories about a bad world. We want them to tell happy stories and we want the world to conform to the stories they tell. We want to live in correlationville. But we live on earth.

The song says keep on the sunny side, always on the sunny side. Good advice for life, but not for science.

Here's my twitter page.