Verified by 
I decided what to do with my life based on reading descriptions of psychology research written for normal, curious people who had no training in the subject.
I remember reading Malcolm Gladwell’s Blink outside my dorm and thinking how cool it was to learn about the research of John Gottman, who could predict with incredible accuracy whether a married couple would separate or not just by watching a video of them arguing.
My master’s degree involved a similar behavioral analysis of conversations between strangers meeting for the first time, and finding cues that could help me predict whether they would cooperate or compete in an economic decision-making game.
Later I moved on to Steven Johnson’s books on science and technology more broadly, where I was introduced to the idea of an ant colony as a single cohesive “super organism” that made smart decisions based on the aggregate outcome of simple rules followed by each ant when foraging for food.
The concept of emergence (the title of Johnson’s book that used the ant analogy) later became a common topic of discussion in the Dynamics of Perception, Action, and Cognition (DPAC) reading group I belonged to in graduate school.
My interest in evolutionary approaches to human behavior was fueled by reading David Sloan Wilson’s Evolution for Everyone; my interest in moral psychology by Jon Haidt’s The Happiness Hypothesis and The Righteous Mind. I became a psychologist—and continue to develop professionally—by reading descriptions of research that present broad descriptions of key findings in accessible, engaging language. (My current reading list includes Superforecasters by Phil Tetlock; so far I’d recommend it!)
When I began graduate school, I was vaguely aware that some scientists criticized Gladwell’s work as being not quite right, simplifying and getting some details wrong. At the time, this didn’t bother me, because I thought that Gladwell got the big picture right, and that I could get the details nailed down by reading the details of the articles.
The most important lesson from my eight years of professional development is that this isn’t true. Psychology research has serious methodological problems. It’s not just popular science writers like Gladwell missing some details; it’s many of the most prominent and successful psychological scientists making big mistakes due to serious misunderstandings about how to collect and think about scientific evidence.
The most important finding in psychology in the last generation was a 2015 paper attempting to replicate 100 findings from prominent journals in the subdisciplines of social and cognitive psychology. Just more than one-third of these findings were successfully replicated.
Most of the findings chosen from these top journals were not robust or general enough that a new set of scientists, following the same procedures laid out in the original article, could get the same results.
The lesson for many psychologists like me was clear: The way we as psychological scientists have been conducting research isn’t good enough. The methods we use to vet published research lets through too many “false positives.” Our current methods allow us to claim that an effect is real when it isn’t.
The most important ongoing project in psychology research, therefore, isn’t answering some new research question like “how can we figure out if couples should stay together?” or “how do individual decisions lead to good or bad society-level outcomes?”
The most important research question in psychology is “how do we know what’s true?”
This is a philosophical question with deep roots, but a working scientist does not need to have a final and comprehensive answer to be able to contribute new knowledge or to identify concrete places where scientific psychology could improve.
For example, scientists for decades have known that it is easier to publish “positive” results that find evidence of an effect than “negative” results that do not find evidence of that effect. This leads researchers to prioritize publishing only results that find evidence of an effect but not being able to publish—and not even wanting to try to publish—results that do not find evidence of the effect.
We don’t need to definitively answer the question “how do we know what’s true?” to know that a body of scientific research that contains all evidence—results both in favor of and against an effect—gives a better idea of what’s true than a body of research with only positive results.
To combat this bias, reformers have proposed a new way of publishing research that had not been in use at many prominent psychology journals: having scientific review occur before the data is collected, on the basis of the question being asked and the methods being used to answer it, and then committing to publish the findings whether they are positive or negative.
Research published using this Registered Reports format is more trustworthy than research not published using this format, because the researchers publishing it do not have any incentive to hide or minimize negative results.
There is an active community of reformers trying to improve the ways research is evaluated, proposing better ways of identifying which psychological findings are likely to be real. From creating new journal article formats like Registered Reports, to implementing small “nudges” like giving people badges for sharing their data, to developing new tools for analyzing data that demonstrate how robust a finding is to different sets of assumptions, many of these reforms are already improving the quality of published research.
My goal with this blog is partly to make new and exciting psychology research accessible to normal, curious people (like I was—and am!). But I no longer think that’s enough. I also want to help people understand why researchers are making a claim, and to help provide the context and tools needed to understand what makes findings more or less likely to be correct.
To make sense of psychological research—and much of the modern world—we need to know not only what is true, but how we know it’s true.
