Are Journal Editors Responsible for Poor Quality COVID-19 Research?

Source: Ray in Manila on Wikimedia Commons

Key Points: Science journals have rushed to publish research during the pandemic. But a good percentage of the papers reveal shaky analysis, unsupported claims, meaningless insight, and even embarrassing typos. Poor research greatly harms the public, especially at times of crisis.

The majority of mental health research related to the COVID-19 pandemic has been poor-quality. As the editor-in-chief of the Journal of Anxiety Disorders wrote, it’s often been “Garbage in, garbage out” (Asmundson & Taylor, 2021). It wasn’t just the low-quality journals either. Researchers who conducted casual, non-rigorous studies are of course largely responsible for this, but there is perhaps a bigger problem.

The bigger problem may be the editors who published the garbage. Researchers submit garbage to journals all the time. The worst of it usually gets weeded out by peer-review. The peer-review system has its flaws, but it works well most of the time. During the COVID-19 pandemic, the normal peer review was changed at many journals that were trying to publish more rapidly.

The normal peer-review process was changed

The editor-in-chief of the Asian Journal of Psychiatry described how he changed the review process for the pandemic. After inviting submissions, he received over 550 in six weeks and published 52 of them. The changes to the normal peer-review process needed to accomplish that speed were astounding. The initial reviewers provided “cursory feedback” within two days. The authors were given one week to make revisions. Re-review after revisions had been minimized (Tandon, 2020).

The literature was flooded with studies that were meaningless and misleading. My series of blog posts showed this, and I’m not alone in saying that. Hundreds of studies on mental health were published in a short span and most were of such poor quality that they weren’t worth summarizing (DEPRESSD Project). One study using three-item or nine-item measures disseminated carelessly by social media apps was one too many.

Case reports have suffered too

Case reports were of baffling quality. The fictional disorder of “coronophobia” was diagnosed in at least two case reports. A 23-year-old female in Indonesia came to the attention of doctors (it’s not stated how) with extreme anxiety attributed to the appearance of COVID-19 in her country. However, there was no report of whether she suffered anxiety prior to the pandemic (Wulandari et al., 2020). A 38-year-old female in Peru became suddenly psychotic after a visit to a dentist in March 2020. She feared she had been infected because the dentist didn’t wear a mask and had recently traveled to France (Huarcaya-Victoria et al., 2020).

An 18-year-old Chinese male visited Wuhan in late 2019 and then two days later developed physical symptoms consistent with COVID-19. His negative COVID tests were puzzling until his doctors discovered he had a six-year history of depression. Then they concluded his fear of being infected triggered all the other symptoms including fever, sweating, and cough (Fu & Zhang, 2020). We’ve known for a long time that stress can manifest as psychosomatic symptoms. Life is full of stressors; it is not evident that COVID-19 stress is a unique variety of stress. Why the editor felt this was a new contribution to psychiatry is puzzling.

Non-mental-health professionals were suddenly mental-health experts

Some of the commentaries in mental health journals that screamed the loudest about our being in the midst of a psychological pandemic were authored by non-mental health professionals. A dentistry school biostatistician suggested that psychological “disorders are more prevalent in non-members of medical teams than in frontline healthcare professionals” through vicarious trauma (Ghaffari & Mortezapour, 2020). A health policy researcher predicted that the pandemic will have “profound mental health impacts that pervade racial, ethnic, and class lines in the United States“ (Purtle, 2020). A professor from the School of Management in Malaysia claimed economic hardships for immigrants contributed to excessive mental health problems, yet cited no mental health research (Mia & Griffiths, 2020). Why would editors publish such extraordinary, unsupported claims from individuals with no mental health expertise?

Proof-reading seemed non-existent

In many cases, it appeared that absolutely no proofreading was done by editorial assistants. Proofreading is important, especially when so many studies were from different countries. In a previous blog post (11/6/2020), I noted this sentence in a study published in the journal PLOS ONE, “Huge citizens expose to social media during a novel coronavirus disease (COVID-19) outbroke in Wuhan, China.” (Gao et al., 2020). I did not know that the mental health of huge citizens in China was impacted differently than small citizens. And I learned that outbroke is a valid Scrabble word, but not in this context.

There are too many of these grammar errors to summarize, but here’s another example. "This paper suggests the applying HBM to COVID-19 in mitigating behaviors which provokes anxiety and fear and converts individual beliefs informed by preconceived impressions of a perceived threat and direct cues of perceived benefits from perceived barriers to action inform behaviors (through perceived self-efficacy)” (Mukhtar, 2020). Got that?

Coronavirus Disease 2019 Essential Reads

Update on COVID-19 and How It Changes the Brain

How Political Partisanship Can Be Deadly

Retractions are everywhere

The problem of research quality during the pandemic was not limited to mental health. As of February 13, 2021, there have already been nearly 70 journal publications on medical topics retracted by journals because either fraud, sloppy mistakes, or unexplainable statements were discovered post-publication and pre-publication peer review failed to uncover them (Retraction Watch 2/13/21).

If you’re a researcher in any specialty, and you’re not following Retraction Watch, you’re missing one of the great stories in the history of science right now. When the good folks at Retraction Watch started tracking retractions in 2010, retractions were thought to be rare. Not so. Science is being corrupted daily by paper mills, predatory journals, fake authorship, and fake data. These are not conspiracy theory hyperbolic claims. These are based on hundreds of retractions per year by the journals themselves. And those are just the ones that got caught. Much of the current concern stems from Chinese universities that often use a pay-for-paper model with their faculty, but the problems occur at some level everywhere.

For example, the Asian Journal of Psychiatry retracted one of those rapidly published papers because the Chinese authors had managed to get it published in three separate journals. The duplicates were discovered when a staffer at a different publisher took the time to run a plagiarism check (Retraction Watch 10/29/20).

The harm of dodgy science

You have to wonder about the politics. The responses to the pandemic in every country were totems of political ideology. As psychologist Philip Tetlock has noted, scientists are politicized in ways they are often dimly aware of (Tetlock and Mitchell, 2015). During COVID-19, objectivity was abandoned in many instances.

This last year raises questions about the resilience of science journals to withstand political times and this age of weaponized ideologies. The behavior of many journal editors seems to indicate, “Yes, we stand for truth through the rigors of the scientific process, just not all the time.” Gorman and Gorman were calling for journal editors to be more responsible well before the pandemic. “Scientific journal editors must always bear in mind the possibility that articles they accept for publication will emerge on the Internet and be seen – and misinterpreted – by nonscientists who are not the intended audience” (Gorman and Gorman, 2017, p 259).

Was this the test run for changing the peer-review process?

The peer-review system through all specialties has been under attack from the explosion of pre-print repositories, lack of good framing metrics (Fleerackers et al., 2021), open access models, predatory journals, greedy reviewers demanding to get paid, and paper mills used by researchers in China under a pay-per-paper model (Else, 2020). Many reformist scientists have called for radical changes to the peer-review process like the type we’ve seen during COVID-19.

We might think of this COVID-19 period as a test run for what happens when peer review is relaxed. The result: nothing good. Neither truth nor treatments were discovered more accurately or more rapidly.

When it comes to science, speed kills. It kills integrity and trust.