Verified by 
Personality
Science by Convenience
Are behavioral researchers doing the best possible behavioral research?
Posted July 28, 2010
Psychologists base many of their conclusions about typical human behavior on a singularly atypical sample set: the undergraduate student. Five years ago, not far removed from being a singularly atypical undergrad myself, I wrote a story about this curious practice. As I've mentioned before in this space, I participated in a psychology study or three as an undergrad. So you might say, to borrow from Groucho Marx via Alvy Singer, that the article was inspired by my refusal to accept any conclusion about human behavior that would have me for a datapoint.
Undergraduate students are known as the "convenient" sample. Some of them are required to engage in research studies as part of their coursework. They're low-cost and easy-access—terms that, preferably, would describe a good date rather than good data. As Peter Killeen of Arizona State delicately put it to me, undergradates are "cheaper than white rats, and they're more similar to the population to which we hope to generalize. And they seldom bite."
Matriculating vampires aside, the question isn't whether undergraduates are more similar to other people than rats are, it's whether they're similar enough to warrant broad conclusions about human behavior. As Lisa Feldman Barrett of Boston College told me:
"The goal of psychology is to make nomothetic laws—laws that apply to all people. ... The question is, how well can you do that when you're sampling by convenience?"
I was hardly the first to consider the problem. In 1971 psychologist Rae Carlson called undergraduates "unfinished personalities." Fifteen years later, writing in JPSP, David Sears argued that
college students are likely to have less-crystallized attitudes, less-formulated senses of self, stronger cognitive skills, stronger tendencies to comply with authority, and more unstable peer group relationships.
But psychologists were too busy running tests with undergraduate samples to run tests on undergraduate samples. It was one of those problems—like the psychotic in the subway car, or climate change—that everyone wanted someone else to handle. So, of course, no one did. As Robert Peterson of UT-Austin told me:
"If you look at the issue of using students [for research], there is virtually no empirical evidence supporting or challenging their use."
Peterson would know. It was he who put the "virtually" in front of the "no empirical evidence" with a massive meta-analysis (pdf here) published in 2001. Some 650,000 subjects and 65 psychological relationships later, Peterson concluded that studying students alone was insufficient. He wrote:
The results augur in favor of, and emphasize the importance of, replicating research based on college student subjects with nonstudent subjects before attempting any generalizations. (my emphasis)
At long last, Peterson has some company. Three psychologists from the University of British Columbia have prepared a mammoth criticism of the "undergraduates who form the bulk of the database" in psychology, cognitive science, economics, and all other behavioral sciences. This magnum opus, set to run in the wonderful journal Behavioral and Brain Sciences, is called "The Weirdest People in the World" (pdf here). "WEIRD," in this case, is an acronym for the Western, Educated, Industrialized, Rich, Democratic subjects who aren't just populating research pools but often polluting them.
Nevermind for a moment that Canadians calling someone weird is like—well it's like something. The researchers raise some powerful questions about standard behavioral samples. According to the authors, convenient undergraduates make up at least two-thirds of subject samples in the leading social psychology publication, the Journal of Personality and Social Psychology. All told, they write, "96% of psychological samples come from countries with only 12% of the world’s population."
The authors argue that in the grand scheme of the globe these WEIRD subjects are not only "frequent outliers" but "among the least representative populations one could find for generalizing about humans." They continue at pace:
Many of these findings involve domains that are associated with fundamental aspects of psychology, motivation, or behavior—hence, there are no obvious a priori grounds for claiming that a particular behavioral phenomenon is universal based on sampling from a single subpopulation. Overall, these empirical patterns suggests that we need to be less cavalier in addressing questions of human nature on the basis of data drawn from this particularly thin, and rather unusual, slice of humanity.
Woah, Canada.
And that's just from the abstract. If I blocked all my favorite quotes from the entire paper, I might need another Internet. So I will skip right to the concluding statement:
The sample of contemporary Western undergraduates that so overwhelms our database is not just an extraordinarily restricted sample of humanity; it is frequently a distinct outlier vis‐à‐vis other global samples. It may represent the worst population on which to base our understanding of Homo sapiens. Behavioral scientists now face a choice—they can either acknowledge that their findings in many domains cannot be generalized beyond this unusual subpopulation (and leave it at that), or they can begin to take the difficult steps to building a broader, richer and better‐grounded understanding our species.
The paper, it should be told, isn't all accusatory bluster. The authors make some intriguing if hopeful recommendations for addressing the problem. Most notably, they call on granting agencies and journals to give researchers incentives to go beyond the convenient sample—fully aware that doing so will add time and money to every experiment.
The great thing about BBS is that each issue publishes numerous commentaries on the lead paper. I'm eager to post some of these reactions when they appear. (Which, if anyone knows the pub date, help a Headcase out?) As Professor Lambeau in Good Will Hunting would say, the gauntlet has been thrown down. The question now is whether the faculty will answer, and answer with vigor.
---
(HT: BPS Research Digest)
Check out a recent related post at the PT blog "Thinking About Kids"
