Verified by 
Replication Crisis
Replication Crisis?
Replication failures are (probably) not cause for panic
Posted September 30, 2015
As many of you may know, a recent collaborative effort attempted to replicate 100 studies that have been published in leading psychology journals. Depending on the criterion used, only about 40% of the studies replicated.
There have been loads and loads of scholars and journalists commenting on these results and their implications for psychology as a science. Many suggestions have been made that are really positive steps (e.g., using larger samples, and more open and thorough reporting of data, experimental design, and samples). But I want to focus on one factor that that does not seem to be getting a lot of “airplay” in why these effects did not replicate more readily – the complexity of human psychology.
Complexity and the Replication Debate
Imagine any behaviour of yours, or any belief you hold, or any emotion you experience. Imagine the causes of any one of those emotions, beliefs or behaviors. Now do the same, but expand this quest to the people around you, then to the people in your country. Then expand that to the entire world. Now do so when people are in groups instead of by themselves. Ok, now imagine trying to predict these various aspects of others while the world outside the laboratory is constantly changing and impacting people. My guess is that you could come up with, easily, a wide range of factors that can influence any specific instance of human behaviour, thinking or feeling.
Yet, by and large, psychologists have been attempting to conclude in experiments that 1 or 2 variables can predict human emotion, cognition and behaviour. To make matters more problematic, most of these studies are done using college students, almost entirely from Eastern and Central Europe, North America (well, Canada and the United States), or Australia. It has long been a very gross oversimplification of a complex being (the human!).
When describing these failed replications, there seems to be this mass ethos among psychologists that effects found in past research should occur similarly across all samples, regardless of what is happening in the external world, and regardless of the specific characteristics of the sample. Even when using university samples (which most studies do) in original studies and in the attempted replications, it is a bit short-sighted to assume that every single sample across all states will respond similarly within a psychology study. When you have replications being attempted in different countries than the original study, this is even more enigmatic, even excluding language comprehension issues (which were not assessed in the current replication attempts).
We know from research that where people live predicts political orientation, religious beliefs and a variety of personality variables. Now, we also know that many effects in experiments occur only for people who are, for instance, high in extraversion, or politically conservative, or who believe in God. And we know that for the counter to each of those “groups” (people who are introverted, politically liberal and who are atheistic), not only might these effects not occur, but the opposite effects can occur.
Now, why this research is being ignored is puzzling to me as we attempt to explain why these studies failed to replicate. If you assume that most psychology effects are moderated (which seems a safe bet for at least the social psychology studies included in the replication attempts) by many factors, then of course switching samples from one side of a country to another, or from country to country, is going to make a difference. Not all university students are the same, everywhere. And not all worlds outside the laboratory are behaving exactly the same. (across history or culture).
If you take a sample from a conservative state, it is very likely that they will respond a bit differently (if not oppositely) to any study that involves a variable associated with political orientation, and probably religious belief too given how correlated those two variables are. And yes, it is true that there will be an equal number of conservatives (probably) in each condition of an experiment. But this doesn’t mean that the effect won’t be dominated by the conservative people in the sample. And if you have more of these individuals, you will be more likely to replicate the effect. (if you don’t measure political orientation as a moderating variable in statistical analyses, you can get a significant main effect of the manipulation on the dependent variable, but this doesn’t mean that there is not moderation occurring). If the original study doesn’t measure this moderator (in this case, political orientation) or report it, and just assumes a main effect, then if a replication is done using a sample that is lower on that moderating variable (in this case has more liberal people), the effect is not likely to replicate.
In essence, this failure of replication is not (entirely) showing that the effects in the original published research don’t exist. What it is probably showing is that things are way more complex than we make them out to be, and the characteristics of the sample (and the situations happening specifically in the experiment setting, the area outside of the setting, and the world for that matter) really matter. We know this on some level, yet it keeps getting ignored within discussions about this replication project. Perhaps this is because people have a general need for structure and simplicity, or because we crave a clear, strong understanding of people so we can clearly be labeled a science. But regardless, it really needs to be considered that what is in doubt is not only the existence of these effects, but equally our understanding of psychological processes, and, in particular, our oversimplification of results in the past. In the desire to understand, I suspect we greatly simplify.
I congratulate the scholars who put a lot of hard work into executing this massive replication effort. The immense reaction it is getting both within and outside of psychology attests to its importance. It is crucial to the development of the field, and to the subsequent positive outcome it can have on the world at large, to do our best to improve many of our scientific practices. But it is perhaps equally important to fight the urge to oversimplify the social and psychological world when doing so might hinder the very accuracy we strive for.
We live in a world of 4-5 way interactions a plenty, and are pretending to exist in a world of exclusively main effects (or at most, two way interactions).
