The Science Reform Brain Drain

The science reform movement feels like a season of RuPaul's Drag Race, with talent leaving all the time.

Source: Photo by cottonbro on Pexels.

The last decade has seen the blossoming of a rich scientific reform movement in psychology. A movement focused on implementing straightforward solutions that fit into—and potentially improved the efficiency of—scientific workflows, this Credibility Revolution in psychology has led to widespread changes in everyday scientific practice. For example, more researchers are spelling out their predictions explicitly ahead of time (to be clear what fits with previous theory versus is a new, unexpected finding), using larger sample sizes (to estimate effects with more precision), and replicating import findings (to make sure they are reliable and consistent). At the same time, jobs for psychology researchers have become increasingly competitive. It’s incredibly hard to become a psychology professor who has time devoted to research. The combination of hostility that reformers face from inside academia, the scarcity of jobs, and the intense expectations of productivity required for tenure mean that key members of the reform movement are beginning to leave the field of scientific psychology altogether. For many of these people, this means better pay, better hours, and more appreciation of their talents. For the psychology reform movement as a whole, this suggests that a career directly devoted to scientific research—the kind of research that winds up in journals and textbooks—might not be worth the grief.

Many of my colleagues have a particular person whose decision to leave academia hit them hardest. For some, it was James Heathers, the bombastic co-host of the “Everything Hertz” podcast, whose work catching scientific fraud was written up multiple times in high-profile news outlets. Heathers developed tools to find academic journal articles whose results could not possibly be true, because the combination of numbers they reported in their papers were impossible. It didn’t completely surprise me when Heathers left the field, because criticizing researchers to the point that their journal articles are retracted makes enemies among their entire collaborative networks—and these networks sit on academic hiring committees.

For some, it was Tal Yarkoni, whose statistical work and methodological criticism in blogs and manuscripts were frequently passed around wonky psychology circles. Yarkoni recently wrote a widely-read manuscript called “The Generalizability Crisis” outlining issues that he felt had barely been addressed yet by the current reform movement. Toward the end of the article, he writes that, given how hard the problems psychology really faces are, one simple “next step” would just be for psychologists to go off and do something else. At the time, the suggestion seemed a bit flip—like, “if you can’t hack it, get lost.” Seeing now that he followed his own advice, it feels more profound—as if one of the deep thinkers in our field decided that the institutional pressures of academic research just didn’t make continuing a good use of his time.

Joe Hilgard debunked the idea that video games caused aggression.

Source: Photo by Julia M Cameron from Pexels

For me, it was Joe Hilgard. Hilgard is a casual acquaintance, but someone I looked up to quite a bit. He researched the role of video games in influencing aggression, and debunked findings purporting to show that video games make people violent. In a recent post, he details how tracking down the details of specific studies led him to strongly suspect fraud from a prominent researcher—and how his attempts to get institutional support from journals and universities to follow up on and correct errors were consistently stymied. In his “quit post," he mentions that stirring things up in this way—even if he was right—was a poor career move. More than that, he writes about how reducing the carbon footprint of a drug company feels like more meaningful work than trying to reform a psychological establishment that is consistently resistant to implementing simple solutions that have been supported by decades of findings. Fighting against a system that rejects innovation and definitive conclusions from statistical modeling is a difficult, disheartening job. Doing it for less money, with longer hours, and with less respect than you could get in an industry position means that only the hardiest (or most masochistic) young researchers will stay in the movement.

In the last couple of years, two reform-minded editors of major psychology journals had their terms expire, and were replaced by editors much less interested in the reform movement. Establishment researchers have long disliked the new directions advocated by reformers. Being tenured has meant that they were able to wait us out. I recall that when I first got involved in this movement, there was a cynical belief among older researchers that this was just the latest in a series of “crises” about the way psychology research was done that ultimately wouldn’t amount to much. “Everyone already knows these criticisms, and every decade or two someone brings up how they make certain results invalid, and then we all just go back to doing things the way we always have,” this voice said. I didn’t believe then that scientific reform would just fizzle out, given the attention and passion it elicited. Now, seeing how tenure insulates older researchers and competition weeds out those who don’t play by their rules, I understand the cynicism better.

Reform of academic psychology research can't compete with the pay, hours, and respect of the private sector.

Source: Photo by Miriam Alonso from Pexels

How has seeing this “brain drain” changed my views? First, it suggests that the best psychological research may not be going on in academic laboratories, but instead may be occurring in corporate labs and tech companies. Harvard and Stanford isn’t necessarily a better credential than Google, Facebook, or Snapchat. Second, it suggests that academic journals will often go to great lengths to prop up sensational but incorrect or deeply flawed research. The most reliable work may not be the work showing up in “prestige” journals—and we may want to be more skeptical of results in big-name journals. Third, scientific research is in dire need of “disruption” to alter the current incentives and vetting processes. I would not be surprised if, in a decade, a more reliable and stringent vetting process for scientific research was established outside the university system, and much of the work from “big name” professors at major universities did not meet these standards. Finally, it suggests that life is bigger than science, and reformers need to remember that we don’t just need to consider whether academia wants us, but whether we want academia.