Evaluating Scientific Claims in the Era of COVID-19

Source: Cottonbro/Pexels

Hydroxychloroquine! Famotidine! Ozone! Bleach! Every day there seem to be claims for new treatments to conquer COVID-19 or a host of other ills, ranging from measles to AIDS. For the last half-dozen years, my colleagues and I have been examining scientific reasoning. We have discovered, in our peer-reviewed published research, that traditional cognitive and academic skills are not good predictors of scientific reasoning. Even college-educated people, some with science backgrounds, often misinterpret scientific data.

Here is a checklist of a dozen things anyone can do to evaluate claims regarding COVID-19, or really, any claim regarding health or nutrition.

1. Is there any credible scientific research?

What does not count as scientific research: “People are saying….” “I’ve heard a lot about….” “I know a guy who took this stuff and he got better really fast.” Anecdotes are not scientific research. They make for interesting human-interest stories. Scientifically, they mean nothing. Also, remember that expertise is not domain-general. Actors may know a lot about acting, politicians about politics, and celebrities about being celebrities; but they are not scientists and their pronouncements are not substitutes for careful scientific research, any more than would be scientists’ pronouncements on acting, politics, or being a celebrity.

2. Who did the research?

Insist on knowing who did the research. It matters, a lot. There are three things to look for. First, do the researchers have serious scientific credentials? Second, are the credentials not only serious but in the field of the research? Third, do the researchers have a conflict of interest? For example, were they funded by an organization that will profit financially if the research comes out a certain way, or will they, themselves, profit from certain kinds of results?

3. Who is telling you about the research?

Most people do not read original scientific articles; rather, they read media accounts of those articles. What is the source in which you read about the research? Is it reputable? Is it objective? Does it have an ax to grind? Or is it looking for sensational headlines to garner attention and revenue?

4. Is the sample of participants in the research representative?

Who was in the sample of participants? Are they representative of the population with regard to age, gender, racial/ethnic composition, health, and or other relevant variables? For example, results from adults do not necessarily generalize to children or vice versa.

5. Is there a large enough number of participants (N) for the sample to have the power for one to be able to draw conclusions?

Look closely at the number of participants. Is it small (e.g., less than 100), medium, or large (e.g., 1000 or more)? Studies done on small samples are less likely to replicate—to yield the same results when done again.

6. Is there a control group?

Good research on medicines, foods, and the like of course has an experimental group that receives the treatments under investigation. But the research must have a control group—a group that receives a placebo treatment—that is, a treatment that looks, tastes, or smells like the experimental treatment (drug, food, or other substance) being evaluated but that is actually inert or irrelevant to the treatment. Otherwise, there is no way to know for sure that any effect in the experimental group is actually due to the experimental treatment rather than to some other irrelevant variable.

7. Are participants blind to their treatment condition (experimental or control)?

If participants know what condition they are in, the research is invalid because the mere fact of the participants’ knowing what group they are in may have affected their reaction to the treatment. If people know what their condition is, some of the effects may be psychological ones due to their perception of the treatment they received.

8. Was the assignment of participants to conditions random?

If an assignment is not random, the results may reflect the assignment of participants to conditions rather than the experimental treatment. For example, in studying sick people, if either the experimental or control group starts off sicker than the other group, the results may reflect in part the participants’ health when they started in the study.

Coronavirus Disease 2019 Essential Reads

Update on COVID-19 and How It Changes the Brain

How Political Partisanship Can Be Deadly

9. Are the outcome measures relevant ones that can be objectively quantified and interpreted?

Make sure the outcomes measured make sense. For example, with regard to medicine, are participants in the study getting better faster—fewer days to recovery? Are they showing fewer measurable symptoms of the illness? Don’t be satisfied with reports of others observing that the patients certainly look like they are doing better.

10. Are the results statistically significant?

Any number of studies get results that are “in the right direction” but that are not statistically significant. Also, results that are statistically significant at the 1 percent level are not necessarily stronger than results statistically significant at the 5 percent level. They are just less likely to be due to chance.

11. Are the results practically significant?

Practical significance refers to whether the results suggest that the medicine, food, or other substance can be practically useful in the real world. This means that a cost/benefit analysis reveals that there is a future for the drug, food, or other substance. For example, can enough of the drug be produced at a manageable cost to produce outcomes that sufficiently positive so as to be worth the cost of the drug?

12. Are there side effects, either short-term or long-term. If so, how prevalent and how serious are they?

As you know, many, and probably most drugs and some foods have undesirable side effects. Some of them show up immediately but others take a while to show up.

That’s it. If the research meets these dozen criteria, take it seriously. If it fails on any of these criteria, be skeptical, even very skeptical. It’s your life and the lives of your loved ones at stake.