You Get What You Measure

In the Feb 14 issue of the New Yorker magazine, Malcolm Gladwell writes about US College rankings.  Which brings up an interesting point for me.  Why the human obsession with measuring and ranking things?

Gladwell does an excellent job of deconstructing and critiquing the ranking system used by the US News and World Report.  Essentially, because of the measures used, such as professors’ pay, alumnae engagement, class size, etc, the rankings measure the wealth of an institution.

One of the major measurements is selectivity. That is, how high are the average grades of entering students, which means that those students have a greater number of choices, and still chose that institution.

The other measurement is graduation rate performance. This measurement compares the expected graduation rate of the institution to its actual graduation rate. A positive difference between the two would suggest that the school is doing a good job engaging and educating its students.

The US News College rankings emphasize selectivity.  As do many graduate programs and employers. Which I think is a problem.  When we use grades as a primary measure of performance it can result in significant levels of cheating. Especially to graduate professional schools like law, medicine, business and education.  And that is exactly what has happened.  First, grade inflation has occurred in high schools.  Second, cheating has skyrocketed in the past twenty years, with over 50% of Canadian undergraduates admitting to having cheated during their university years.

Another example of this measurement fallacy is the input of school reputation. The US News rankings include school reputation at a 20% weighting. Which is great.  Except, how does a school get a good reputation? By having a strong position in the US News rankings.  Kind of a circular logic.

The whole point of this long-winded discussion of school rankings is to suggest that our penchant for measuring things might just be backfiring. When using rankings or measurements, consumers and managers need to have a deep knowledge of how these measurements are collected, and what underlying biases they might promote.

So the next time you look at a ranking or measurement, whether it is a performance appraisal, a consumer research report or the Mclean’s University Rankings, ask yourself the obvious.  How is this data collected, and what might be the weaknesses in the data?  What are the unanticipated consequences of making decisions using this information?


1 reply »

  1. I got caught on your first question of your first paragraph – Why?
    An answer didn’t follow, unfortunately, leaving me to speculate. When looking at the outcome of any ranking system I find that it is ego based. “I will prove that I am better than the others, any others!” The HIGHER up the rank I place the more I have to look DOWN upon. I believe it is as old a preoccupation as is the oldest profession.
    Being ego based would certainly agree with your statistics on Canadian undergraduates. The ranking of Ontario’s public schools based on the intelligence of grade 3 students seems to have had a similar ‘side effect’. What will these grade 3 students have learned from this ranking process. Was it:
    a) The perimeter of a rectangle is the sum of all sides. OR
    b) Wow, we would have been the best in Ontario if our teachers hadn’t been caught! OR
    c) Are we so bad that the only way to save face was to cheat?
    I think ‘a)’ is out. ‘b)’ is a good possibility as they may decide to become better cheaters to get higher rankings as even their teachers thought that was okay. ‘c)’ would be quite a slam in the ego for any child self aware enough to realize what happened.
    Your question is the right one, Why? As for the data used, I learned long ago, “Garbage in – Garbage out”. Skip the data and keep the question where it belongs.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s