In the Feb 14 issue of the New Yorker magazine, Malcolm Gladwell writes about US College rankings. Which brings up an interesting point for me. Why the human obsession with measuring and ranking things?
Gladwell does an excellent job of deconstructing and critiquing the ranking system used by the US News and World Report. Essentially, because of the measures used, such as professors’ pay, alumnae engagement, class size, etc, the rankings measure the wealth of an institution.
One of the major measurements is selectivity. That is, how high are the average grades of entering students, which means that those students have a greater number of choices, and still chose that institution.
The other measurement is graduation rate performance. This measurement compares the expected graduation rate of the institution to its actual graduation rate. A positive difference between the two would suggest that the school is doing a good job engaging and educating its students.
The US News College rankings emphasize selectivity. As do many graduate programs and employers. Which I think is a problem. When we use grades as a primary measure of performance it can result in significant levels of cheating. Especially to graduate professional schools like law, medicine, business and education. And that is exactly what has happened. First, grade inflation has occurred in high schools. Second, cheating has skyrocketed in the past twenty years, with over 50% of Canadian undergraduates admitting to having cheated during their university years.
Another example of this measurement fallacy is the input of school reputation. The US News rankings include school reputation at a 20% weighting. Which is great. Except, how does a school get a good reputation? By having a strong position in the US News rankings. Kind of a circular logic.
The whole point of this long-winded discussion of school rankings is to suggest that our penchant for measuring things might just be backfiring. When using rankings or measurements, consumers and managers need to have a deep knowledge of how these measurements are collected, and what underlying biases they might promote.
So the next time you look at a ranking or measurement, whether it is a performance appraisal, a consumer research report or the Mclean’s University Rankings, ask yourself the obvious. How is this data collected, and what might be the weaknesses in the data? What are the unanticipated consequences of making decisions using this information?