Among the many issues with ranking schools, one of the most glaring is incorporating the input of those who are impacted by the result. Students reporting on MBA programs or University presidents ranking schools all put people influenced by the result in a position to influence the results. This creates quite the incentive problem.
Recent evidence comes from the rankings of schools (pdf) provided by University of Florida President Bernie Machen. The surveyed rankings are an integral part of the U.S. News ranking formula, and were obtained by the Gainesville Sun in a public records request. Other Florida university presidents were shrewd enough to "lose" theirs.
U.S. News treats the surveys as anonymous, meaning that a university president's ranking of his own school carries equal weight as others' rankings. On Machen's survey, the University of Florida was given the highest possible ranking, one that he granted several generally well-regarded schools only after some revision.
(excerpt of Machen's rankings)
More telling is the rankings Machen gave to other Florida public schools which are competitors for State funds. Machen rated more Florida schools as "marginal" (the lowest possible category) than schools from all other states combined.
Editors responsible for the ratings claim that "statistical methods" are used to adjust for such biases. The reality, of course, is that no statistical test can divine thoughts separate from incentives. If you asked me to rate myself as a "good person" on a scale of 1 to 10, a period of reflection would follow. If you added that my results would be anonymous, unverifiable, and come with a million dollar payment if I circled "10," you would learn nothing about me from the exercise except my responsiveness to incentives. So why would U.S. News editors contend that as-yet uninvented statistical methods protect the integrity of their results? Perhaps they, like President Machen, have a stake in the results.
UPDATE: I am not suggesting that UF does not deserve to be ranked highly along several dimensions. For example, one reader reports that UF must be at the top of its peer group in criminology, with over 4% of its students arrested annually.
Subscribe
0 comments:
Post a Comment