When Tests Fail People

Pixabay

Source: Pixabay/Pexels

The simple answer to the question of “What should college and university admissions tests measure but don’t?” is “Almost everything.” I used to believe the situation was bad; now I think it is scandalous. Let me explain why.

In my theory of successful intelligence, there are four sets of skills that matter for success in life and, to some extent, in school.

1. Creative skills. How well a person can create, invent, imagine, suppose, discover.
2. Analytical skills. How well a person can analyze, evaluate, judge, critique, compare and contrast.
3. Practical skills. How well a person can apply, use, put into practice, implement, and persuade
4. Wisdom-based skills. How well a person can use creative, analytical, and practical skills to achieve a common good, balancing their own, others’, and larger interests, over the long as well as the short terms, through the infusion of positive ethical values.

The basic idea is that you need creative skills to come up with ideas, analytical skills to ascertain whether the ideas are good, practical skills to put the ideas into practice and persuade others of their value, and wisdom to ensure the ideas help to achieve a common good.

We have tested these ideas for their validity. For example, in the Rainbow Project, a national project, we found that including creative and practical skills in a test could double prediction of first-year GPA over the prediction obtained with just admissions-type tests of analytical skills. The tests also greatly decreased ethnic-group differences relative to the tests of analytical skills. In the Kaleidoscope Project, a project at Tufts University, we found, including wisdom as well, that our tests increased prediction of both academic and extracurricular performance over tests of analytical skills, while greatly reducing ethnic-group differences.

College and graduate-school admissions tests, such as the SAT, ACT, and GRE, I used to believe, utterly failed to measure creative, practical, and wisdom-based skills, but at least provided reasonably good measures of analytical skills (as well as of knowledge base). I was wrong.

Years ago, Wendy Williams and I found that the GRE was not a good predictor of success in the graduate program in psychology at Yale. But I knew, of course, that Yale was highly selective, and that we were looking only at psychology. As it turned out, the conclusion was generalizable. I just didn’t know it at the time.

More recently, several colleagues—Karin Sternberg, Rebel Todhunter, Vina Wong—and I decided to follow up on the work with Williams. We were particularly interested in one aspect of college, university, and life success: scientific thinking. Obviously, such thinking is important for scientists. But if you look at the strong anti-science movement in the United States, such as the anti-vaccination movement or the climate-change–denial movement, you can see that everyone needs to be able to think scientifically, or at least to be able to evaluate both scientific and anti-scientific claims for their validity.

We devised tests of scientific thinking that measure what scientists actually do in their work, and what everyone should do to be scientifically literate—generate alternative hypotheses for empirical claims, generate experiments to test theoretical or empirical claims, and draw scientifically defensible conclusions from scientific data. We also looked at some allied skills, such as evaluating quality of teaching in science. These are important skills for everyone, not just scientists. We tested students at Cornell in a wide variety of majors—scientific and otherwise—to determine their levels of scientific-reasoning skills.

What did we find? First, we found that, psychologically and statistically, the scores on the measures of scientific reasoning all clustered together. In other words, those tests were measuring similar psychological skills. Second, we found that scores on IQ-like tests, such as the SAT, ACT, number series, and letter sets, also clustered together psychologically and statistically. Third, we found that the two clusters were relatively independent. In other words, the college-admissions–type tests (which measure so-called “general mental ability,” or” GMA) did not well measure analytical skills of the kind that are central in scientific thinking.

So, it turned out, if you want to select students for their scientific-thinking skills, the conventional tests used for college and university admissions are not particularly good measures. This is not to say that they are useless. General mental ability can be used in lots of tasks in life. But taken alone, it is not a particularly good measure of scientific thinking.

Colleges and universities that rely heavily on these admissions tests are making a big mistake. When will they admit it? The college-admissions tests are hard to dislodge. Universities tend to be entrenched, continuing to do things the way they used to. They are afraid to take risks; they like things (like the tests) that don’t cost them anything (the students pay); and they have old-fashioned views of “merit.” So, I don’t know when they will admit it, but as institutions that encourage scientific thinking in their students, it would be great if they did it themselves.