Countering a Replication Reichstag with Behavioral Science

The “replication crisis” talking point in academic psychology represents a very under-developed conception of the scientific method that fails to grasp the sequence of stages or the complexity of theory, hypothesis, and prediction testing. By asserting that a theory should be considered disproven if a given study anywhere in the nomological net of that theory yields results that were not consistent with the study hypothesis, rapid “replication crisis” proponents wage war against the philosophy of science, positivism, and logical positivism.

Scientific Method

The way science works is that the investigator begins with an empirical observation perceived through the senses. This observation can be considered data. The investigator then comes up with the most parsimonious explanation (best guess) as to why the above observation exists. This explanation is considered a “hypothesis” in the scientific method. Scientific hypotheses generate predictions, i.e., a specified outcome for a given study. The distinction between a hypothesis and a prediction can be illustrated as follows: It was hypothesized that anxiety affects test performance. Based on this hypothesis, it was predicted that greater anxiety leads to poorer test performance. A prediction specifies a direction. An investigator who generates a hypothesis based on an observation then tests future predictions generated from that hypothesis. If the predictions turn out to be true, then the hypothesis is strengthened. If the predictions turn out to be false, then a new hypothesis is developed that generates new testable predictions.

When several hypotheses have been supported by research, i.e., the predictions generated by them have held up in scientific research, then a theory is developed which ties together the various hypotheses and explains their rationale in the most parsimonious way possible. A theory generates new hypotheses. Once an investigator has combined extant hypotheses into a theory, the investigator tests predictions generated from novel hypotheses generated by the theory. If a novel hypothesis is supported by evidence, the theory’s truth value is strengthened. If a novel hypothesis is not supported by evidence, then the theory must be reworked to generate new hypotheses and explain existing data.

Science says what "replication crisis"?

To conclude that a study did not replicate and therefore theory upon which the study was derived is false is research and logic malpractice to an absurd degree. If a study does not find the same results as prior study with the same methods, then one prediction generated from one hypothesis derived from one theory has been found to show mixed results. Some research finds that the prediction is true whereas other research does not. Assuming enough replication studies to conclude with confidence that the results for the prediction are indeed mixed, then one must go back and re-work the hypothesis that generated the prediction. Other hypotheses generated from the theory may have produced reliable predictions whereas one hypothesis did not. The next move is not to junk the theory but to modify it such that it explains only the hypotheses for which empirically supported predictions exist.

What has been labeled the replication crisis and specifically in connection to psychology and social psychology is based on erroneously rejecting an empirically supported theory because a single prediction generated from a single hypothesis of that theory produced mixed results. Modifying the theory and disposing of unsupported aspects is the gold standard in psychological science. Describing gold standard practice in psychological science as a replication crisis strikes me as spin or framing that would make George Lakoff green with envy.

While it is true that there are few exact replication studies in psychology and there should be more, it is a complete misnomer to describe testing prediction 4 from hypothesis 2 of theory A as a conceptual replication study. It is simply the logical extension of conducting a new study based on prior research.

Going forward

In future posts on the “replication crisis,” I will focus on specific fields within psychology and specific studies and theories deemed to be evidence of a “replication crisis.” Stay tuned.