A “Replication Crisis” or Academic Demagoguery?

Source: Shutterstock with Winarick Edits

There has been a new academic idol for the past few years. His philosopher king status and the influence of the “replication crisis” only came to my attention because of Twitter, the most highly respected medium for serious academic debate (insert eye roll emoji). A Stanford professor, Ioannidis, has been credited with addressing a deep dark secret in social science: The research does not replicate, across the board. While many others have drawn attention to this issue before him, his outsized public figure status may have lent more credibility to the notion, helping to amplify the idea that there is a “replication crisis.”

Reactions have often focused on the lack of funding or publishability of a replication study as one of the reasons for the cancer of replication failure. While this is undoubtedly true if taking the meaning of a replication so literally that only an identical recreation of the original study can bear on this question.

There are a number of problems with the conclusion of a “replication crisis in social science.” For one, as a social scientist with a modest publication history, scholarly academic research 101 holds that it's bad practice to make conclusions marked by excessive over-generalization. This is why the titles of most peer-reviewed scholarly articles would never generalize any finding to “social science” as a whole but to a specific domain within social science as well a specific research methodology, including design, sample, and measures as well as data analysis. To not be bothered by a claim so general as to include social science, from macroeconomics to identifying politically-relevant predictors of political behavior to basic cognitive science, psychological science research as well as staged social psychology experiments involving confederates, emotion and psychophysiology research involving measuring response latency after exposure to various stimuli, as well as, cognitive psychology research involving studies of semantic priming. In fact, “social science” is so general, I need a few more sentences.

Display of social science domains from shutterstock

Source: Shutterstock with Winarick Edits

Social science also includes longitudinal developmental psychology research on parenting, attachment style, and neurobiology. It includes studies of autobiographical memory specificity in depression, OCD, and other forms of psychopathology. It includes research on quantitative content coding of implicit motives from narrative text and qualitative content coding of field interviews with a population of interest. It includes psychometric research on the validity of measures of intelligence, personality, and psychopathology. It also includes outcome research on interventions such as psychotherapy or something else on reducing some relevant variable (pre-posttest design). It includes efficacy (RCTs) and effectiveness (naturalistic, one group pretest-posttest design) studies on various interventions social interventions.

Accordingly, the claim that there is “a replication crisis in social science” might be rejected at first glance by any logical positivist (the strictest definition of what is and is not science) or positivist (knowledge can be gained through experience and observation). The rule of thumb is the more specificity, the better if making a controversial claim. Even Karl Popper, the most hardnosed philosopher of science who dismisses anything not falsifiable as voodoo, might reject it as a pseudoscientific claim on its face. Yet, the replication crisis has been accepted as a major concern in social science in need of being addressed as practically the first to-do for the field. How did that happen? Why did it happen? Is there a replication crisis? If so, where exactly is it? This is part I of a multipart blog post on “the replication crisis.”

I plan to study the proliferation of the “replication crisis” trope, evaluate its validity by critically examining the major peer-reviewed publications that espouse this point of view, and ultimately explain what happened here. I also remain open to being proven wrong. Perhaps, there is a crisis accurately described as a problem in social science for good reason. We will see. Stay tuned.