Verified by 
As the U.S. presidential contest elbows its way into the COVID-drenched news cycle, the possibility that Donald Trump will undermine mail-in voting has made headlines around the world. But a more insidious problem that psychologists and political scientists have grappled with is how to get more voters to participate in the first place.
Efforts to “get out the vote” include door knocking, mail-outs, and telephone calls. The impact of such campaigns is modest, usually increasing voter turnout by about two to four percentage points. However, one team of psychologists has claimed that a simple intervention may boost turnout by up to 14 percent.
Noun-priming: A simple method to boost voter turnout
The idea behind this intervention is to represent voting as a fundamental expression of the self. To achieve this, the noun “a voter” is embedded in a pre-election survey, comprising questions such as “How important is it to you to be a voter in the upcoming election?”
Christopher Bryan and his colleagues first evaluated the impact of noun-priming in the lead-up to the 2008 presidential election and 2009 New Jersey gubernatorial election. Respondents received either the survey laced with noun-primes, or a control survey in which the nouns were swapped out for verbs (i.e., “How important is it to you to vote in the upcoming election?”). Those exposed to the noun prime were up to 14 percent more likely to vote than those exposed to the verb prime, an effect the study authors described as among the “largest experimental effects ever observed” on voter turnout.
But in the years immediately after these findings were published, psychology found itself weathering a storm of evidence that many published results cannot be independently reproduced. And one of the most contentious kinds of effect within psychology's “replication crisis” is behavioural priming.
Like the nouns-vs-verbs study, most behavioural priming studies feature subtle manipulations with dramatic effects on subsequent behaviour. But many of these have failed to replicate—such as the finding that being primed with age-related words (e.g., “retirement”) leads people to walk more slowly. Where priming studies do replicate, their effects tend to be modest and short-lived. For instance, brief exposure to words such as “gamble” and “wager” across 80-90 hands of online Blackjack can sightly increase the proportion of bets a player will place during the session. So priming isn’t bogus, but it isn’t magical either.
Is the noun-priming effect reproducible?
In 2016, a team led by political scientist Alan Gerber tried to replicate the effect of noun-priming on voter turnout. With more than 2,000 participants per condition (compared to just 200 participants in total in the larger of the original studies), this study was powered to detect even tiny effects of noun-priming. However, the team found no evidence that nouns-vs-verbs boosted voter turnout at all. In contrast, a standard “get-out-the-vote” message produced a modest (but more typical) ~2% increase in turnout.
In a brief commentary, Bryan and colleagues described this replication attempt as “psychologically inauthentic” because it took place in the context of low-profile primary elections, few of which were “meaningfully competitive”. Gerber’s team replied to this critique, noting that they were unaware that “the election must be high-profile or competitive” for noun-priming to work. Indeed, election competitiveness was not mentioned anywhere in the original study. But could it account for the discrepant results?
To find out, Gerber and colleagues tried again to confirm the impact of noun-priming on voter turnout. This time they focused on highly competitive election contexts—gubernatorial elections across three U.S. states, plus a contested mayoral election. They also switched to a web-based survey, among other efforts to more closely match the original methods used by Bryan’s team. But still there was no impact of nouns-vs-verbs on voter turnout. Even when restricting analyses to the more competitive of the three gubernatorial elections, amid other checks on robustness, no evidence emerged for the dramatic 14% boost in voter turnout.
The second replication attempt was again criticised by Bryan, joined by a new set of co-authors. This time it was claimed that Gerber’s team had included so many control variables as to render their analyses “flawed” and their conclusions “unreliable.” Although this concern is reasonable in principle, Gerber’s team had also presented analyses without any of the control variables (p. 933)—Bryan’s team don't mention this. And it's strange that their concern isn't raised in the critique the first replication attempt, which also involved many control variables.
Bryan’s team also objected to the timeframe of the second replication attempt. Whereas the original study was rolled-out on the day before and morning of the election, the 2018 replication spanned four days, beginning three days prior to the election and concluding at the end of the election day. Bryan’s team explain that collecting data on election day risks delivering the noun-prime to respondents who had already voted1. On the other hand, they don’t explain why delivering the intervention three days before an election is problematic, beyond noting that it represents “a substantial departure from the original experiments.”2 Again, it’s curious that this concern applied equally to the first replication attempt, yet wasn’t mentioned in the critique of that study.
Recovering the noun-priming effect
Bryan’s team then present a re-analysis of the data reported in Gerber and colleagues’ second replication attempt. They omit all control variables and include only the data collected one day before the election. They also adopt a lax criterion for statistical significance, despite the increased risk of false positives. And that is when the effect of noun-priming on voter turnout finally surfaces.
Probing further, Bryan and colleagues then deploy a technique to examine evidence for their effect under a variety of data configurations—such as including some control variables and omitting others. This amounted to a comparison of 1,200 models, around half of which yielded support for the effect of noun-priming (again, provided one adopts a lax criterion for statistical significance). After constraining these to a shortlist of 218 models, the noun-priming effect emerges in nearly all of them.
Although these re-analyses seem compelling, the models Bryan's team selected were a fraction of the total number of possible models. And the reasoning supporting their choice of models was sometimes hard to follow. For instance, they decided not to compare results across the three states sampled, which Gerber and colleagues had done to examine the impact of more versus less competitive election contexts. Indeed, they claim that the competitiveness of the election isn't even relevant, which seems to directly contradict the critique of the first replication attempt.
In the end, Bryan’s team conclude from their re-analyses that noun-priming may (under certain conditions) boost voter turnout by as many as 4.2 percentage points. This falls well short of the fourteen-point boost reported in 2011, but is in line with more typical effect sizes in this literature.
What have we learned?
So, is the effect of noun-priming on voter turnout reproducible?
One reading of this story, played out across the papers by Bryan, Gerber, and their colleagues, is that two large-scale replication attempts failed to detect an effect that was actually there all along. This seems to be the narrative preferred by Bryan and colleagues, who claim that their reanalysis provides “strong evidence in support of a noun-wording effect on turnout” (p. 25542). They attribute the null findings reported by Gerber’s team to potential “null hacking”—i.e., design and analysis decisions driven by the motivation to obtain a null result—producing “misleading” failures to replicate. Yikes.
Another reading is aligned with the conclusion drawn by Gerber's team after their first replication attempt. They suggested that the noun-priming effect is potentially reproducible “but highly sensitive” to various study characteristics. Indeed, the effect Bryan’s team extract in their re-analysis is (1) much smaller than the effect reported in 2011; (2) very sensitive to timeframe, being maximal at one to two days prior to the election but fading if this window is altered; and (3) possibly dependent upon the competitiveness of the election context (or perhaps not?).
But perhaps the evidence is not sufficiently strong to conclude even this. Whatever the limitations of the two replication attempts conducted by Gerber's team, we are yet to observe a successful independent replication of the impact of noun-priming on voter turnout. Until we do, the potential efficacy of this manipulation would seem to be an open question.
__________________________________
1 Gerber’s team checked their results based only on respondents who completed the pre-election survey in full (p. 934). Although this doesn’t guarantee that the respondents hadn’t already voted, it would seem odd for anyone to engage fully with a pre-election survey after they had already voted.
2 Gerber and colleagues acknowledged their wider timeframe in both of their replication attempts and conduct supplementary analyses to address this. Specifically, in the first replication they present analyses for a subset of the data collected the day prior to the election (they had collected no data on election day). In the second replication they present data for a subset of the data collected the day prior to and the day of the election (where available).
