Attracting WEIRD Samples

Much has been written about psychology's reliance on WEIRD samples. "WEIRD" is an acronym coined by Henrich, Heine, and Norenzayan: samples that are drawn from populations that are White, Educated, Industrialized, Rich, and Democratic. Ninety-nine percent of all published studies rely on participants recruited from populations that fit those criteria.

This reliance on a limited population of participants is problematic for many reasons. Most obviously, it is a threat to the external validity of our findings. If we only study a phenomenon in one population, we don't know if or how it works in others. We limit our generalizability.*

There is encouraging evidence that psychologists are reaching beyond our standard model of studying undergraduate psychology students. For example, Mechanical Turk is a service now frequently used by social scientists to recruit participants in studies that can be completed on the internet. Samples drawn from crowdsourcing services tend to be more diverse than those from other sources.

But who you ask to be in a study is not the same as who actually signs up. I was thinking about this while I was writing a lecture on how microaggressions, implicit bias, and stereotype threat can undermine the external and internal validity of studies. One way in which implicit bias can affect who enrolls in studies is through self-selection. If I don't feel welcome, or that the study is for "people like me," or that the way the study is framed is going to make me look bad, I will not sign up.

In writing the lecture, I pulled a list of studies from a psychology recruitment site (see the screenshot below). One thing that struck me was the differences in the samples that each might attract, based solely on the title.

Read down the list. I don't want to pick on particular titles. But it seems clear that different studies are going to appeal to different folks. This is particularly problematic for studies whose intent is to develop new measures that work well in diverse populations or who are developing materials to be disseminated for educational purposes.

We can do better

Often psychologists spend a lot of serious thoughtful work developing measures that are accessible. I think we as a field have changed our paradigm so we no longer believe that people are interchangeable and that an all-male Harvard sample can generalize to people all over the world. In addition, as psychology students have become more diverse, psychology samples have also become more diverse. However, as a developmental psychologist who studies adolescents because they are different from people who are older or younger, I am bemused by the notion that college students are "typical" of people as a whole.

We can do better. Inviting a more diverse population to participate is a necessary first step. But it is not sufficient. We also need to be alert to the messages we send with that invitation.

Screen shot of site recruiting participants for psychology studies.

Source: Nancy Darling/screen shot

* There are both advantages and disadvantages to having diverse samples within a study, depending on its goals. See "Are Intro Psych Students Lab Rats?" for a discussion of this issue.