Verified by 
My first experience in conducting social science research occurred when I was assigned my very first term paper, for a ninth-grade history class. Sadly, our teacher's instructions for the term paper focused almost entirely on trivial matters of form (e.g., how to format footnotes). He spent virtually no time on conceptual matters, like what does a reasonable hypothesis look like and how does one gather and evaluate evidence objectively in order to properly test one's hypothesis. The impression I got from my teacher's impoverished instructions was that I was supposed to come up with a hypothesis and then gather as much evidence as I could find to support my hypothesis. Knowing that Ulysses S. Grant was considered by many historians to be a poor president, I came up with the hypothesis that U.S. presidents who were elected on the basis of their success in military leadership made the worst presidents. I ignored cases like Eisenhower to focus on Grant's shortcomings, and earned a well-deserved D+ on my term paper.
Essentially, the model I had adopted for "testing" my hypothesis was a legal model rather than a scientific model. In the legal model, prosecutors and defense attorneys are supposed to stack the evidence in favor of guilt or innocence, respectively. The legal theory behind this practice is that when opposing sides present their best arguments, an impartial judge or jury can decide which case is stronger, and truth will win out more often than not. That might be fine for the legal system, but that is not how science is supposed to work. Scientists are not supposed to begin with the goal of convincing others that a particular idea is true and then assemble as much evidence as possible in favor of that idea. Scientists are supposed to be more like detectives looking for clues to get to the bottom of what is actually going on. They are supposed to be willing to follow the trail of clues, wherever that may lead them. They are supposed to be interested in searching for the critical data that will help decide what is actually true, not just for data that supports a preconceived idea. Science is supposed to be more like detective work than lawyering.
But this is science in theory, not in practice. In practice, it looks to me like many social scientists learned about hypothesis testing from my ninth-grade history teacher. In psychology I think it is actually the norm for psychologists to begin with a preconception about what is true and a desire to convince others about this preconceived truth. Next, they set out find data and statistical procedures that will produce results that support what they already believe. And, in some cases, their desire for their pet hypothesis to be true is so strong, it colors their perception and makes them see support for the hypothesis when the support is not there—sometimes even when the data literally suggest the opposite conclusion.
One example of a research study in which one of the conclusions was exactly opposite of what the data indicated is described in a book that is now regarded as a citation classic, Birth to Maturity: A Study in Psychological Development, by Jerome Kagan and Howard Moss. This book, which was cited over 460 times in the 20 years after it was published, reported findings about personality stability in a select group of subjects studied at the Fels Research Institute in Yellow Springs, Ohio. One of the study's most frequently-cited findings about personality stability was that personality traits that are consistent with sex role expectations (e.g., dependency in females, aggressiveness in males) are more likely to show stability from childhood to adulthood because social expectations reinforce gender-related personality traits. In the 1960s when the Fels study was published, behaviorism and social learning theory were strongly in vogue, and behavior genetics was just a fledgling enterprise. It is therefore not surprising for the Fels researchers to speculate that social reinforcement explained the relative stability of gender-related personality traits. Their explanation was repeated in countless textbooks and was never challenged until I pointed out in a conference paper how any observed stability in personality traits must have occurred despite the sex role expectations rather than because of them. To my mind, the researchers' misinterpretation of their data is a classic example of theoretical preconceptions distorting the perception of data.
The Fels study, other research I have conducted, and my day-to-day experience with other psychologists has convinced me that we are not the dispassionate, unbiased detectives described by textbook accounts of the scientific method. Rather, psychologists are more like lawyers who are trying to convince an audience that their particular narrative of events is correct. There are good reasons why research psychologists act like lawyers. First, according to Mercier and Sperber (2011), the evolved function of human reasoning is to make persuasive arguments, to convince others that your position is correct. Using arguments to persuade others is a part of human nature, and because scientists are human beings, it is natural for them to use their reasoning to create persuasive arguments. Furthermore, we need to remember that a requirement for staying employed as an academic researcher is publishing a sufficient number of peer-reviewed articles in highly-regarded journals whose rejection rates hover around 90%. Editors of the top journals demand creative, ground-breaking studies, not replications or slight variations of existing studies. But being creative often means contradicting what editors and reviewers currently accept as true, and Mahoney (1977) has demonstrated that reviewers are more likely to reject articles that contradict their own theoretical predilections. This means that researchers need to be particularly skilled, persuasive arguers to be professionally successful.
What should be done about the failure of psychological researchers to live up to textbook accounts of dispassionate objectivity? Some individuals have called for reforms in the way we evaluate and reward research, so that researchers do not need to act like lawyers. Personally, I think that this is probably a losing battle because, in agreement with Mercier and Sperber, I think that most scientists possess the species-typical trait of wanting to lawyer for their own pet positions. The best we can hope for is that, when the smartest scientists make their best cases for their divergent positions in the arena of scientific discourse, eventually the best ideas will prevail, just as the truth about guilt or innocence is supposed to prevail in the courtroom.
Maybe my ninth-grade teacher did not teach me the wrong model of social science research. Maybe I simply had not yet learned how to construct a persuasive argument. Hopefully I am a little better at that now.
