Outcome Switching and Withheld Data in COVID-19 Research

Source: Angellodeco/Shutterstock

With more than 4.8 million cases worldwide of COVID-19 currently recorded by the Johns Hopkins tracking system, and the U.S. alone suffering more than 90,000 deaths from the virus, the need for reliable medical and public health data has never been greater. The information not only guides treatment in hospitals, but also helps to determine—along with the virus’s reproduction rate, the all-important R number—whether national and state officials can safely reopen sectors of the economy, along with schools, universities, and local businesses.

In the search for reliable data, concern is growing among public health experts about outcome switching, wasteful and misleading medical research, and data being withheld from key clinical trials. The problem is especially acute when incomplete, sometimes faulty data is released to news outlets eager for positive reports because the coverage skews the treatment picture, bright-siding readers with false hope. It also draws attention from nonmedical interventions, which are still the best means of mitigating community spread and reducing the strain on our health care systems.

Last week, the British Medical Journal published “Waste in COVID-19 Research,” an editorial by Paul Glasziou, Sharon Sanders, and Tammy Hoffmann at the Institute of Evidence-Based Healthcare at Bond University, Australia, that concluded bluntly, “A deluge of poor-quality research is sabotaging an effective evidence-based response.”

The article drew on several recent examples that had made international news. Of the 145 registered trials of hydroxychloroquine currently listed on ClinicalTrials.gov, the national registry, only one provided an actual study protocol. Of the remaining 144, “32 have a planned sample size of less than 100; 10 have no control group; and 12 are comparative but non-randomized. Outcome measures vary widely… Even limited registry details reveal unjustified outcome switching.”

The questionable results of such flawed studies are nonetheless treated as if they were breaking news, helping to fuel a perception that safe and reliable treatments are just around the corner, while perhaps lulling some into a false sense of security. This, in turn, adds pressure to state and local authorities to end shelter-in-place orders, even as the number of infections across the U.S. continues to rise daily.

Glasziou and colleagues note that a March 20 preprint of the first reported study of hydroxychloroquine—“a non-randomized study of 46 patients with inappropriate analyses”—has so far been cited 520 times, whereas “a larger, randomized trial of hydroxychloroquine posted on MedRxiv on 14 April showing no benefits received far less attention.” Media bias favoring the first study helped to “trigger a wave of what is likely to be largely unnecessary or misdirected research: 135 hydroxychloroquine studies have been registered on ClinicalTrials.gov since March 20.”

That date turns out to be meaningful and not just because of media reporting. On the previous day, President Trump had touted the antimalarial at his White House briefing, asserting, “It’s not going to kill anybody… and it’s shown encouraging—very, very encouraging—early results.” On April 6, he then twice asked the public and assembled press, “What do you have to lose? Take it!” as he instructed the federal government to stockpile 29 million pills of the treatment. By April 24, as part of the same White House briefing, the president claimed that an “injection inside” the human body with a disinfectant such as bleach or isopropyl alcohol could help combat the virus—comments he later claimed were “sarcastic” (itself peculiar language to adopt in a pandemic briefing, with more than 90,000 Americans already dead from the virus). Public health officials had to scramble to warn that following such advice would result in immediate fatalities.

On May 18, the president acknowledged publicly that he was himself daily taking hydroxychloroquine with zinc and had been for a week and a half, though the treatment is unproven, not recommended, and goes against the advice of the FDA and medical professionals in his own administration.

As the spotlight turned appropriately to the risks and harms of hydroxychloroquine, including on the liver, kidneys, and heart rhythm—and the Washington Post reported, “Drug promoted by Trump as coronavirus ‘game-changer’ increasingly linked to deaths”—media interest shifted abruptly to the antiviral remdesivir, originally designed to treat the Ebola virus. A battery of reports alluded generously to its promise, despite prominent notice that its first round of trials on COVID patients had been a bust, with no reduction of the pathogen in the bloodstream, negligible improvement in treatment days, and many patients opting to discontinue.

The hype intensified for several weeks, including cautious but firm support from Dr. Anthony Fauci, director of the National Institute of Allergy and Infectious Diseases, with headlines such as “Critical study of Gilead’s COVID-19 drug shows patients are responding to treatment, NIH says.” However, the “responses” in question were modest and open-ended, leaving unclear whether remdesivir crucially helped to disable the virus and prevent deaths: “In the trial, 8 percent of the participants given remdesivir died, compared with 11.6 percent of the placebo group, a difference that was not statistically significant.”

Coronavirus Disease 2019 Essential Reads

Update on COVID-19 and How It Changes the Brain

How Political Partisanship Can Be Deadly

The NIH subsequently declared the trial an “interim” analysis and decided to give the drug to its placebo group, citing the moral urgency of their decline while necessarily ending meaningful comparison of the drug with non-remdesivir patients, limiting the value of the trial. Others inside the NIH disputed that it was ever an “interim” analysis, instead calling it a “preliminary final analysis” that would justify the full release of data that Gilead Sciences, the maker of remdesivir, initially had promised but has since chosen to withhold.

Others connected to the trial worried that the earlier hyping of statistically insignificant results, coupled with the discrediting of hydroxychloroquine as a viable alternative, left remdesivir as the presumptive default, even without supporting data: “The danger is, now it’s the treatment for everybody,” Janet Wittes, president of Statistics Collaborative, told STAT News. “Now this is the base drug [… but] I think we don’t know if it’s strong enough for it to be the standard of care. I don’t think we know who should be treated.”

In a follow-up article, “Where’s the data? In a pandemic, now is no time to sit on COVID-19 trial results,” Ed Silverman pressed Gilead on its decision to withhold by stating the obvious: “The sooner trial data are available for review and scrutiny, the faster physicians and researchers can gain a clearer picture into the best use of a medicine and also understand when it should not be used.” Those who had hyped the drug were left wondering whether its manufacturer was now actively impeding clinical discovery.

In their editorial in the BMJ, Glasziou, Sanders, and Hoffmann note that such problems extend far beyond the remdesivir saga: “We found the sample frame of most studies was unclear, missing cases were undocumented, and ‘asymptomatic’ was [left] undefined,” complicating the necessary research appraisal now underway.

“The imbalance in trial topics is worrying,” they add, in particular for “the paucity of trials on non-drug interventions.” While drug-based projects currently have secured at least $74m in funding, nonmedical treatments are sidelined by misplaced emphasis and drug-related noise. They could find “just two trials of masks on ClinicalTrials.gov and none examining social distancing, quarantine effect or adherence, hand hygiene, or other non-drug interventions,” such as contact tracing.

“Given the risk that a vaccine may be ineffective, partially effective, or delayed,” they conclude, “there is an urgent need for a body” similar to the World Health Organization or Coalition for Epidemic Preparedness Innovations (CEPI) to “coordinate and support neglected research into non-drug interventions," as these “have so far been the only effective means of control” in limiting the spread of COVID-19.