Verified by 
AMERICAN ASSOCIATION FOR ADVANCEMENT OF SCIENCE (AAAS) PROCLAIMS GENDER BIAS IN PEER REVIEW WITHOUT EVIDENCE
AAAS hosted a conference in April, 2016, which presentation after presentation by famous, influential, and prestigious scientists argued for the power and prevalence of implicit gender biases in peer review. Yet not a shred of evidence of implicit bias in peer review was actually presented. Just read this report – you can see for yourselves that there was ample evidence of gaps, but none that bias accounted for those gaps. And if you believe that “gap=discrimination,” you are wrong (please read my prior blog on the topic).
Now, that AAAS report is basically a news article, not a peer reviewed scientific article. So I contacted two attendees of the conference. I knew one personally – Simine Vazire -- and consider her to be an extremely reasonable and thoughtful scientist. I hate disagreeing with Simine, but, if I have to disagree with someone, it would be Simine – because she is one of few people I have ever met capable of disagreeing without becoming insulting or hostile. Our exchange on sex bias in social psychology might be of interest to many readers of this blog.
So I asked her the bluntest of all possible questions:
“I am about to write a blog claiming that, at the AAAS conference you attended, no one presented a shred of evidence that bias explains any of the gaps that were discussed. I do not want to make that statement if it is not true. Is it true?” She confirmed, via email, that, in her view, it was a true statement, although she also pointed out that “unconscious bias” was not her main scientific area of research.
The AAAS article also included this: “’Studies of the peer-review process have shown that African-Americans and women ‘are held to higher standards to be judged competent,’ said Molly Carnes, a professor of medicine, psychiatry, and industrial & systems engineering at the University of Wisconsin-Madison.”
WHAT STUDIES?
“Aha!” I thought, “Now I can get to the bottom of this. Just because the attendees didn’t present evidence of gender or race bias in peer review, it does not mean the evidence doesn’t exist! It must be that the evidence is so overwhelming, so conclusive, that the presenters simply took it for granted that any reasonable scholar attending would be thoroughly familiar with the abundant evidence.” So, figuring that anyone with familiarity with the area would know such evidence cold and could direct me to it, I emailed Dr. Carnes a request to identify the studies to which she referred.
As it turned out, this was not the simple slam dunk it seemed at first glance.
FAILED ATTEMPT I
Dr. Carnes wrote back to me:
“Much of this work comes from Monica Biernat at University of Kansas. She refers to the phenomenon as ‘shifting standards of reference. I have attached some.”
Now, this was amusingly ironic – I went to grad school with Monica Biernat, and was there when she developed the ideas and initial studies that became her work on shifting standards. I know that work pretty well. Here is the kicker, though: Monica has never published a single study of the peer review process.
However, Dr. Carnes also sent me this paper of hers (Kaatz et al, 2015, published in Academic Medicine). Here is one of the key conclusions, quoted directly from the paper: “Critiques showed differences due to applicant sex despite similar application scores or funding outcomes: more praise for applications from female investigators, greater reference to competence/ability for funded applications from female experienced investigators, and more negative evaluation words for applications from male investigators (all P < .05).”
People are more likely to criticize and less likely to praise those for whom they hold higher standards. Carnes’ paper acknowledges this: “Such gender bias may be unconscious and derives from pervasive cultural stereotypes that women have lower competence than men in fields like academic medicine and science where men have historically predominated.” So, who received more criticism and less praise for similar quality applications? Men! Put differently, Dr. Carnes’ own research produced a result exactly opposite to the one that she was promoting at the AAAS Conference! Women were held to lower, not higher, standards than men to warrant praise.
FAILED ATTEMPT II
So, I contacted Dr. Carnes again. This time, I wrote,
“Thanks for sending those along, but I am confused. I did not see anything here by Biernat on peer review. Do you? Or is there some other reference?”
Then I also pointed out that her own paper showed a result opposite to her claim. I ended with this:
“Thanks in advance for whatever clarifications you can provide.”
After agreeing that Biernat never studied peer review, she went on to explain the result from her paper as follows:
“The Kaatz paper shows similar scores for men and women but greater praise and acclamation for women's proposals. This is an observational rather than experimental study, but it aligns with experimental work indicating that even though women may receive higher verbal acclaim for their work, they are less likely to have this accompanied by tangible rewards such as promotion, hiring, raise, or - perhaps in this case - a grant score.”
Right. They found women were held to lower standards with respect to receipt of praise and criticism. Now, one might argue that, “Aha! This is bias! People have low expectations and standards for women scientists!” And that might be a valid point, and it would be interesting if it led to lower levels of funding for women. But the Kaatz et al paper reported no actual results comparing male and female researchers’ outcomes. In seeking to explain their results, it did, however, include this curious statement: “This could be one interpretation of the comparable scores and funding outcomes for male and female investigators…” What can this (“comparable scores and funding outcomes”) possibly mean other than “We found no gender bias in funding outcomes”?
Women might be held to higher standards or they might be held to lower standards. Either might be common or general, or specific and contingent (maybe women are held to higher standards in some situations, and lower standards in other situations). But one cannot cite evidence of women being held to lower standards as evidence that they are held to higher standards.
SEMI-FAILED ATTEMPT III
However, Dr. Carnes also sent me another paper. This one really was on peer review, and it really did show sex bias favoring men. Women had to be much more productive than men to get an equal rating. In the mid-1990s. In Sweden. When applying for postdocs.
So why have I cast this as a “semi-failed” attempt to get the data? Let’s contrast the original claim with the data supporting it so far:
Claim, made at AAAS conference, in 2016: “Studies of the peer-review process have shown that African-Americans and women ‘are held to higher standards to be judged competent"
VERSUS
A single study of peer review of postdoc applications in Sweden, in the mid-1990s, showed bias against women.
FAILED ATTEMPT IV, PART I: “VARYING CONCLUSIONS”
“It Ain’t What You Don’t Know That Gets You Into Trouble. It’s What You Know For Sure That Just Ain’t So.” (Mark Twain)
I sent Dr. Carnes one last email:
“Thanks for sending that paper on Swedish grant reviews from the 1990s. I am wondering if you know of anything else? Say, something in the U.S. or in journal reviews?”
Dr. Carnes graciously replied yet again. Her email included a striking admission:
“There are a number of studies that have looked at grant awards with varying conclusions about gender bias.”
I did not ask her the following question: “If the conclusions are varying, then why were you quoted as claiming that the studies supported one specific conclusion, that peer review is [supposedly] biased against women and African Americans, without mentioning that there were also studies showing no bias or reverse bias?
Now, this would all be a trivial issue if Dr. Carnes was a unique outlier, or if her views were not taken seriously. But none of this is true. Her claims all occurred at a major conference sponsored by a major scientific organization. She was invited, presumably, because AAAS considered her an important expert on this topic.
I sent Dr. Carnes a draft of this blog, requesting that she tell me anything I had gotten wrong and offering her an opportunity to post a guest blog in reply. I also informed her when this post went live, repeating those offers. As of this writing, I have not received a reply.
And, as we shall see in my next post on this topic, rather than being an idiosyncratic outlier, this sort of thing is commonplace, when scientists allow their political agendas to drive their claims about science -- in general, but also specifically, with respect to explanations for gender gaps in STEM.
------------------------------------------------------
A follow up on this post appears here:
If you are interested in these, you may also be interested in:
Why Do Brilliant Girls Tend to Prefer Non-Stem Careers?
and
As usual, please familiarize yourself with my comment guidelines before posting. In short, no snark, sarcasm, or insults, and please stay on topic.
Follow us on Twitter for new blog postings and links on science reform, scientific dysfunction, political distortions of science, threats to academic freedom and freedom of speech, and the value of intellectual diversity
