Breaking the School Mismeasurement Habit

A recent study from the Massachusetts Institute of Technology (MIT) revealed that some school ratings (based on performance level) can be misleading measures of school quality, yet these ratings are used widely by educators and parents. This echoes problems that arose when schools tried to use value-added measures to evaluate teacher quality. Educators and other stakeholders are continually using data to make decisions that impact students, teachers, and communities, and they should be using data to guide decisions, but their efforts are often undermined by mismeasurement.

Source: Courtesy of Routledge

Knowing heroic educators want to avoid mismeasurement pitfalls to best help the students they serve, I interviewed one of the authors of the new book Mismeasuring Schools’ Vital Signs: How to Avoid Misunderstanding, Misinterpreting, and Distorting Data (Rees & Wynns, 2022). In this interview, author and education data expert Steve Rees helps us understand why the mismeasurement problem those MIT researchers found is so commonplace and what can be done about it.

Why do mismeasurement problems persist despite educators’ considerable training, passion, and hard work?

The scholar who has advanced the clearest and most blunt answer to your question is John Hattie. He’s a lifelong educator of educators, and in his seminal book, Visible Learning (Routledge, 2009), he writes that “The key question is whether teaching can shift from an immature to a mature profession, from opinions to evidence, from subjective judgments and personal contact to critique of judgments.”

Other scholars, Ellen Mandinach and Edith Gummer, in their book Data Literacy for Educators, point to the entrenched old guard in schools of education, who continue to defend experience-based craft knowledge and resist advances that come from outside their field (neuroscience, computational social science, economics).

I can add a few down-to-earth factors to those well explained by Hattie, Mandinach, and Gummer. I chalk it up to innumeracy, self-interest, and the insularity of the education profession.

Innumeracy is prevalent in America. Where else would you hear someone casually describe themselves this way: “I’m a people person, not a numbers person.” But I believe that the prevalence of innumeracy is higher among educators than among the general population.

Ed leaders don’t want to see evidence that might reveal that their decisions are to some degree ineffective, inefficient, or incorrect. They don’t want to discover that they adopted the wrong curriculum to teach reading. They don’t want to examine evidence that might show their district produces the highest proportion of graduates who have to take remedial writing courses when they go to college. They don’t want to listen to their own program evaluators who can prove to them that a favored program is eight times more expensive per student and producing questionable learning outcomes. If leadership values “getting along” highest, then measurable improvements that would lead to friction and disgruntled staff or board members are too often just ignored.

Insularity of the profession is perhaps the largest barrier to the advance of measurement in the school world. It is an island culture. Classrooms are islands within schools. Schools are often islands within districts. And too many districts are themselves islands within their counties and states. Where do we see unified or high school districts meeting with their colleagues over “the fence” on the college side?

The insularity of education leaders and teachers begins with their education. Schools of education sit apart from the other professional schools on campus (e.g., business, medicine). Their faculties don’t go for lunch to the same dining hall, read the same journals, or mix at parties. No wonder economists and social scientists are only seen on the staffs of some of the largest districts and some state departments of education.

What can educators do to improve measurement in their schools and districts?

The good news is that they can do a lot. Districts are governed by school boards, and some of those boards include community leaders whose work requires measurement skills. This includes people working in medicine, mental health, financial services, the military, architecture, sports, marketing, engineering, auto repair, and more. If district leaders turned to their boards for help, many would be happily surprised to find people who could provide technical guidance.

Education Essential Reads

The Bad News About Teen Sex Education Reflects Good Trends

The Rise of Homeschooling: The Confessions of an Educator

District leaders could also set standards for the people they hire in management positions. Those standards and skills were published in 2015 by the National Center for Education Statistics under the series title, “Data Use Standards.” HR directors need measurement skills, both quantitative and qualitative. Business office leaders need knowledge and skills that go well beyond budgeting and accounting: ratio analysis, cost-benefit analysis, risk analysis. Curriculum and instruction leaders should know how to run comparative cost-benefit analyses of competing curricular offerings. This requires estimating how much learning results for the dollars invested, or “bang-per-buck” analysis. Instructional leaders also need to have full command of the interpretive skills needed to estimate the growth of student learning.

Finally, districts are free to measure what matters most to them. Is it student satisfaction? Is it the number of books students read for their own pleasure? The quality of student writing? Decreasing the percentage of third-graders reading below grade level each year? This freedom is too often under-utilized.

I encourage districts to take fuller advantage of their freedom zone. That freedom includes the ability to ask better questions, more fundamental questions, the kind that kids often ask. My favorite: “How do you know what you know?”