When Studies Don't Replicate: A Case Study

Significant changes have occurred in social science research over the past few years, mainly because many studies could not be replicated when scientists tried to replicate them.

Replication of research is at the heart of the scientific process. When scientists are unable to replicate the work of others, we cannot know whether the original work is trustworthy or the findings were made by chance, or only apply to some populations of people and not others.

Scholars have dubbed this phenomenon a “replication crisis,” which has spurred significant changes in how scientists do their work. For example, many peer-reviewed journals require scientists to provide more detail about the methods, samples, and statistics used so that others can more easily replicate their studies.

Another standard developed is the open sharing of data and statistical models that scientists use to test their hypotheses. In the past, scientists would ask other scholars to share this information directly, and often they did.

Today, scientists make this material available on websites such as the Open Science Framework so that anyone can access the information. And lately, scientists are beginning to share their data before publication. These changes have increased the transparency of the scientific work being conducted and made it easier to replicate and verify others’ research.

But what happens when open science practices are not followed?

Sometimes, scholars use research to advocate for changes in policies and laws. It is essential to closely examine the trustworthiness of the studies and conclusions made by the scholars who produce such research. If there are problems with the investigation, it can negatively impact many people’s lives.

What happens if the scholar’s work is not transparent? How can another scientist replicate the position to determine whether the conclusions made are trustworthy?

This issue recently arose in a paper published by Joan Meier and colleagues. The paper was published as part of a student-edited law paper series and received a lot of media attention. The authors advocated using this paper to change public policies and laws regarding separating and divorcing families.

Meier concluded that women’s abuse allegations in court are often discredited, so they and their children are in danger from abusive fathers. These conclusions were very concerning for many family violence scholars. They prompted psychology professor Jennifer Harman and legal scholar Demosthenes Lorandos to closely examine how these scholars came to their conclusions.

What they found was very troubling—there were very few details about the methods used by the authors to collect their study sample. In the description of how Meier’s team analyzed their data, they wrote that they “reviewed the [statistical] output, and, through numerous iterations, refined, corrected, and amplified on the particular analyses.” This sentence indicated that the authors used a data-dredging technique known as “p-hacking,” which occurs when someone manipulates their analyses until they get the statistically significant results that they want.

The results are unreliable, and when done to promote one’s expectations of what the findings “should” be, also unethical. Even more troublesome, there were no statistical models or tables reported in the paper for a reviewer or potential replication study team to see what the authors did.

Harman and Lorandos wanted to replicate the study, particularly given how important the findings are for families affected by family violence. When directly asked, Meier refused to provide them with study information (for details on this correspondence, the emails are publicly available).

Based on what was described in the paper, Harman and Lorandos found over thirty problems with the study’s research design, which they detailed in a recent article published in Psychology, Public Policy, & Law.

An exact replication of the Meier et al. (2019) study was impossible, so Harman and Lorandos identified the Meier et al. team's conclusions in their paper, created hypotheses that would test the conclusions, and then developed a study to test them.

Parental Alienation Essential Reads

Talking to Custody Evaluators: Common Mistakes

When Target Parents Reunite With Alienated Children

They used open science practices from start to finish (all details are accessible), and they failed to find any support for the conclusions made by Meier and her team. Harman is conducting another study to test the hypotheses using another sample to see if the results will replicate.

Unfortunately, Meier has continued to promote the findings from her student-edited paper while failing to acknowledge Harman and Lorandos’ critique of her study and their inability to replicate her findings using open science practices (Meier, 2021).

Given the lack of transparency, admitted p-hacking, and study design issues that Harman and Lorandos identified in the original publication of her work, the conclusions that were made and are being promoted pose a serious risk to families struggling with family violence.

For social problems like family violence and parental alienation, replications and higher standards of transparency and accountability should be expected, not ignored or undermined.