Gaming the rankings

Among the many issues with ranking schools, one of the most glaring is incorporating the input of those who are impacted by the result. Students reporting on MBA programs or University presidents ranking schools all put people influenced by the result in a position to influence the results. This creates quite the incentive problem.

Recent evidence comes from the rankings of schools (pdf) provided by University of Florida President Bernie Machen. The surveyed rankings are an integral part of the U.S. News ranking formula, and were obtained by the Gainesville Sun in a public records request. Other Florida university presidents were shrewd enough to "lose" theirs.

See Machen game the rankings

U.S. News treats the surveys as anonymous, meaning that a university president's ranking of his own school carries equal weight as others' rankings. On Machen's survey, the University of Florida was given the highest possible ranking, one that he granted several generally well-regarded schools only after some revision.

(excerpt of Machen's rankings)

More telling is the rankings Machen gave to other Florida public schools which are competitors for State funds. Machen rated more Florida schools as "marginal" (the lowest possible category) than schools from all other states combined.

Editors responsible for the ratings claim that "statistical methods" are used to adjust for such biases. The reality, of course, is that no statistical test can divine thoughts separate from incentives. If you asked me to rate myself as a "good person" on a scale of 1 to 10, a period of reflection would follow. If you added that my results would be anonymous, unverifiable, and come with a million dollar payment if I circled "10," you would learn nothing about me from the exercise except my responsiveness to incentives. So why would U.S. News editors contend that as-yet uninvented statistical methods protect the integrity of their results? Perhaps they, like President Machen, have a stake in the results.

---

UPDATE: I am not suggesting that UF does not deserve to be ranked highly along several dimensions. For example, one reader reports that UF must be at the top of its peer group in criminology, with over 4% of its students arrested annually.

Circular reasoning and the debasement of science

Ranking journals is a popular pastime among academics. Each of us has a favorite ranking, largely chosen by the results fitting with our favorite publication outlets. There are more debates over the methodology of journal rankings than of ranking business schools. There may be no universal agreement on the right method but there certainly is a wrong one.

Continue Reading

Kristie Engemann and Howard Wall have published a new ranking of economics journals. Their method consists of "a simple rule that considers citations only from a short list of top general-interest journals in economics." In short, they arbitrarily select the "top" journals, count the number of citations from these to other journals, add an adjustment here and there for effect, and presto! We determine the top journals by counting citations from top journals. Seems a bit circular.

If you walk into a random high school and want to know who the popular kids are, the Engemann and Wall method would have you identify them by seeing with whom the popular kids choose to hang out. The procedure might produce slightly different results if you started with the debate team than if you started with the cheerleading squad.^† It might not be a surprise, then, the top five journals in their results are included in the list of top journals by assumption. I don't disagree with the list, intuitively, but science should perhaps take a more objective path.

A more objective path does indeed exist. A commonly-used recursive algorithm initially assigns all journals an equal value. Each iteration of the algorithm assigns value from one journal to another based on citations. The iterative procedure, by the way, is at the heart of Google search results (replace "citations" with "links"). From the Google founders' monumental paper:

PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds to the principal eigenvector of the normalized link matrix of the web.

The authors of the new ranking poo-poo this mathy stuff:

[The iterative] procedure is largely a black box: It is not possible to see how sensitive the weights (and therefore the rankings) are to a variety of factors. The obvious objection to our rule is its blatant subjectivity. Our counter to this objection is to point out that the [iterative] procedure, despite its sheen of objectivity, contains technical features that make it implicitly subjective.

Ummm... Sensitivity analysis even has its own Wikipedia page.

If Engemann and Wall were to start their own search engine, the Google formula would presumably be replaced with "pages with links from pages we like."

---

^† This is not to say that cheerleaders don't often overlap with the debate team. But, seriously, they don't.

Hat tip, Mankiw

How to become (in)famous in under three hours

In a recent blog post, I took a tongue-in-cheek approach to the contentious topic of ranking business schools. The genesis of the post was a very different question: how to rank hospitals' success rates with a specific operation when some hospitals only accept less risky cases while others take on more challenging ones. Accepting only less risky cases should imply a higher success rate for obvious reasons having little to do with the quality of care. Business schools endowed with brighter, more capable students likewise should see higher success among their students independent of the quality of education the students receive.

To demonstrate this point, I provided a quick and dirty analysis, completed between the hours of 1 and 3 am, restricted to data on hand, and without the careful statistical standards that would constitute "research." The point was to show that changes to the assumptions underlying rankings can significantly change the results.

The resulting hoopla over the post, which begot university press releases and took my blog's traffic from a handful of loyal-reader friends into the thousands, is both enlightening and frightening. Below I offer a few clarifications.

See the clarifications

"What silly methodology! You've got to be kidding": I received many emails and comments of this form. The answer, of course, is "yes, I am." My friends, the frequent consumers of my blog, have sufficient personal context to inject sarcasm into my written word. I very much believe that there is insufficient discussion about what rankings try to measure, and whether they do this well. This point of my post was quite serious. However, the very aim of the post (literally stated) was to note that rankings (i) fail to account for self-selection, and (ii) are too easy to generate and cause mass hysteria among schools and students. The media coverage, some of which reported on these rankings without any hint of humor, seems to prove this point. A case in point: any ranking which offers "extra credit for sending me money ('investment index'), or publishing my papers ('scholarship discovery index')" was likely not intended to supplant Business Week.

The value of salary: I do not buy the premise that salaries equate to school quality. Assuming that some students care about social, career, and life issues apart from maximizing net present value of future salary stream, starting salaries may say more about the student's priorities than the quality of the school.

I did not "name" any school as a top school: A few universities took to releasing press releases, indicating that "an Economist names SCHOOL X Top Y." While I do consider myself an economist, that hardly confers authority status on business schools. My rankings aimed only to demonstrate that if you accept starting salary as a valid sole measure of schools, then one should contemplate the increase in salary a school provides, not its absolute value. These press releases, issued without any communication with me, became articles in papers (absent the tongue-in-cheek nature of the numerical rankings) heralding my "research."

The self-selection bias is ever-present: An editor at Business Week blogged about my rankings. Despite ever so slightly impugning my motives (and misspelling my name), he seems like an all-round good guy and found "a lot to recommend" about my methodology, though perhaps in part because he viewed my post as more critical of US News than Business Week's ranking methodology. While Business Week does not explicitly include GPA or GMAT in its rankings, recruiter evaluations of students still necessarily conflate the quality of the student with the quality of the school. Recruiters are not asked "how much do you think the school contributed to this individual's market value above what she would likely have had if she went to another school?" Instead, a recruiter simply saying "I like these students" can very well reflect that the kind of students that go to this school would be well-liked by recruiters even if they went elsewhere or did not pursue an MBA.

This is not "research": Not all mutterings by those of us in ivory towers constitute research. Some are just mutterings. While I would love for a journal editor to attest to my PTRC that my blog has contributed to general knowledge, the standards for research are quite different from those for personal blogs. While I was unaware of them at the time (my research priorities have nothing to do with ranking schools), several articles that did survive (or are currently in the process of) peer review offer similar methodologies or conceptual discussions for ranking business schools:

• Tracy and Waldfogel, Journal of Business, 70, 1.
• Dichev, Journal of Business, 72, 201.
• Bednowitz, CHERI Working Paper #6, 2000.
• Arcidiacono, Cooley, Hussey, International Economic Review, forthcoming.
• Devinney, Dowling, Perm-Ajchariyawong, Australian School of Business Working paper, 2007.

So why did you do this?

• geekiness
• sleeplessness
• curiosity

I should have learned my lesson last year. The only other post of mine ever to receive attention was my ranking of local restaurants by wine prices. It, too, resulted in well-placing restaurants citing my "study" under their list of "awards" and received its share of detractors from those lower down the list.

Methodology versus assumptions: If I was to rank dog breeds by average size, a careful methodology would account for variance, measurement error, etc. If I then labeled this list "The top dogs for kids" one should pause, wondering what breed size has to do with loyal pets. The methodology I employed had significant disclaimers, but was mostly correct. A follow-up by resident statistician extraordinaire Bruce Cooil found an empirical model that improves on mine. This is not surprising since Bruce ranks first on the noted Global Statisticians by Efficacy annual ranking, though his tweaking, while resulting in a statistically superior model, leads to few qualitative changes.

In any event, methodological questions like "did you account for standard error," or "did you account for ..." any of the other million issues miss the point entirely. Clearly, I didn't (though those questions are better directed to the publishers of rankings that people actually use to make life-altering decisions). Instead, ask what the rankings ought to measure, and if the methodology achieves that aim.

The best "breed" of dog, of course, is a mutt.

Where I take a turn at ranking business schools

Rankings of business schools generally fail to evaluate the inherent quality of an institution, instead ranking the people who choose to attend it.

UPDATE: If you came here from a source that did not make clear the wry, tongue-in-cheek nature of my rankings, also read the clarification.

An MBA student from UC-Davis will graduate, on average, with a starting salary that is $30,000 lower than a graduate from nearby UC-Berkeley. Can we take from this that two similarly-credentialed students at the two schools would have such a high difference in their market values? This reasoning ignores the selection bias. A student accepted by both schools is quite likely to choose the one often ranked in the top 10.

As long as top candidates choose to go to top programs, a higher ranking confounds the quality of students and the quality of a school. The proper interpretation of Business Week's rankings, for example, is not that Harvard is a better school than Blah College, but that the type of students who go to Harvard do better after graduating from Harvard than the type of students who go to Blah College after graduating from Blah. Thus, a high starting salary for Harvard graduates might imply that Harvard's professors can polish rough stone into beautiful rubies, but it is also possible that Harvard has the benefit of students who could have very well succeeded anywhere. (Note: I pick on Harvard because it actually does quite well in my rankings, supporting the rubies theory).

Which schools do best with the students they have? Traditional rankings fail to tell a given student with a given skill set which schools are most likely to increase his market value. That's the goal of these rankings, highlighting that a change in methodology significantly alters the results. Methodological disclaimers (and there are many) are at the very bottom.

On to the rankings ...

 

Ranking of Business Schools by Efficacy

			Data
Rank	School	Market       Value	GMAT	GPA	Adj. Sal
1	Cornell (Johnson)	$14K	682	3.31	118
2	Indiana–Bloomington (Kelley)	$13K	656	3.37	104
3	University of Virginia (Darden)	$12K	688	3.33	120
4	Texas–Austin (McCombs)	$12K	673	3.38	109
5	Harvard	$8K	713	3.63	134
6	Vanderbilt (Owen)	$7K	644	3.27	101
7	Rice (Jones)	$6K	642	3.25	100
8	Minnesota–Twin Cities (Carlson)	$7K	661	3.37	99
9	MIT (Sloan)	$6K	705	3.5	126
10	Maryland–College Park (Smith)	$6K	650	3.34	98
11	Georgetown (McDonough)	$6K	677	3.26	108
12	Ohio State (Fisher)	$6K	661	3.41	97
13	NYU (Stern)	$6K	700	3.4	123
14	Duke (Fuqua)	$6K	690	3.38	114
15	UNC–Chapel Hill (Kenan-Flagler)	$6K	681	3.27	110
16	Brigham Young (Marriott)	$5K	661	3.53	93
17	Rochester (Simon)	$5K	673	3.52	98
18	Texas A&M (Mays)	$5K	665	3.4	97
19	Northwestern (Kellogg)	$3K	704	3.5	122
20	Boston College (Carroll)	$3K	651	3.35	94
21	Univ. of Pennsylvania (Wharton)	$2K	712	3.53	130
22	Columbia	$2K	707	3.4	128
23	Southern Methodist (Cox)	$2K	640	3.3	95
24	Arizona State (Carey)	$2K	675	3.44	98
25	Wisconsin–Madison	$1K	656	3.37	92
26	Michigan State (Broad)	$1K	633	3.22	97
27	Chicago	$1K	709	3.5	126
28	Yale	$1K	700	3.47	116
29	Purdue (Krannert)	$0K	662	3.32	94
30	Penn. State (Smeal)	$-1K	650	3.3	92
31	Emory (Goizueta)	$-1K	685	3.3	106
32	Washington Univ, St. Louis (Olin)	$-2K	674	3.38	95
33	Michigan–Ann Arbor (Ross)	$-2K	700	3.3	118
34	Illinois–Urbana-Champaign	$-2K	627	3.4	92
35	UCLA (Anderson)	$-2K	704	3.6	114
36	Boston University	$-3K	668	3.38	92
37	Dartmouth (Tuck)	$-4K	713	3.46	127
38	Carnegie Mellon (Tepper)	$-5K	696	3.32	111
39	Georgia Institute of Technology	$-5K	665	3.4	88
40	Babson College (Olin)	$-5K	631	3.21	93
41	Stanford	$-5K	721	3.61	133
42	Notre Dame (Mendoza)	$-7K	673	3.2	95
43	Univ. of Southern Cal. (Marshall)	$-8K	689	3.3	102
44	Univ. of Washington (Foster)	$-8K	679	3.38	92
45	U. California–Berkeley (Haas)	$-9K	710	3.57	115
46	Univ. of California–Davis	$-11K	674	3.37	87
47	Univ. of Iowa (Tippie)	$-15K	652	3.34	76
48	U. California–Irvine (Merage)	$-16K	667	3.34	79
49	Univ. of Georgia (Terry)	$-16K	653	3.4	74
50	Univ. of Florida (Hough)	$-30K	680	3.4	70

 

These rankings consider only the 50 schools in the most recent U.S. News rankings. Specific methodological details are available at the bottom.

Adjusted salary (in thousands of dollars) reflects both the starting salary of those employed within three months of graduation, and a downward adjustment for those who are not.

Market value denotes the difference between a school's adjusted salary and what that school's students would expect to earn (given their qualifications at admission) at an average business school (for a loose definition of an "average" school in this context, see no. 29). A student with a high GMAT score and an exceptional undergraduate GPA is likely to receive higher offers than one with lower scores regardless of the MBA program he attends (not because of the undergrad GPA, but because of what it reveals about the person). Market value indicates how much a school improves on this given its actual student population.

The notion of market value is akin to the distinction between two corporate tasks: recruiting the best talent, and guiding that talent to its potential. Most of the popular business school rankings are biased towards achievements in recruiting, while the above rankings measure efficacy with the given talent. Of the U.S. News top ten, only Harvard and MIT are also in the top ten in efficacy. Conversely, Stanford and Berkeley, also top ten U.S. News schools, are in the bottom ten here, suggesting that members of their admissions staff deserve sizable bonuses.

By way of example, UT-Austin, Wash U, St. Louis, and UC-Davis admit nearly identical student bodies, quantitatively, yet differ greatly in the market's value of these students two years later. On the other hand, Yale and Cornell have nearly identical starting salaries, and therefore end up only one spot apart in U.S. News. Yet, given the superior class, in terms of GMAT, GPA, and selectivity, Yale should do better, thus ranking in efficacy 27 spots below Cornell, which takes the number one spot.

So, what's the goal of this? Perhaps there's a deep philosophical point about the purpose of education. I adopt market salaries as a measure of value purely because it is available, and is the most common quality measure in business rankings (or perhaps because of my unfaltering adherence to the social philosophy underlying classical economics). There's also a mundane point: rankings are not difficult to generate, easy to game, and even easier to tailor toward mass hysteria and overreaction. To that end, extra credit ("adjustment factors") will be applied to next year's rankings for posting a comment below ("brand management and awareness index"), sending me money ("investment index"), or publishing my papers ("scholarship discovery index").

 

METHODOLOGY

Disclaimer: This was done in great haste, and, in keeping with tradition of business school rankings, without too much regard for the appropriateness of statistical procedures.

Overview: The rankings are based on the residuals in a regression of average GPA and GMAT score on adjusted starting salary. That is, a school's score is the difference between its adjusted starting salary and the predicted salary from an ordinary least squares regression.

Data: Obtained from the 2009 U.S. News rankings of the top 50 business schools, which compiled data on the 2007 graduating class.

Adjusted salary: All students not employed within three months of graduation are (pessimistically) assumed to have a salary equal to 80% of the average salary of employed students at their institution. This biases results against schools with low placement rates. If S=average salary and e=percentage employed, then
     adjusted salary = Se+.8S(1-e)

Model selection: The only predictive variables of average earning potential available are average GPA and GMAT, though these are not bad as far as instrumental variables go. A model linear in GPA and/or GMAT badly fails specification tests. Various transformations of average GPA and average GMAT were tested. The final model is given by:
     adjusted salary = GPA + GMAT + GMAT^2
The model has an adjusted R²=0.70. Both GMAT parameters are highly significant (p<.001) and GPA is marginally significant (p=.084).

Residuals: From the estimated model, we obtain the studentized residuals by which schools are ranked. The "value" reported is simply the difference between actual and predicted adjusted salary.

Diagnostics: I ran all of the diagnostic tests that I could think of in three minutes. Heteroskedasticity is not a problem. Specification tests are mixed. Test for normality: not even close. GMAT and GPA are not significantly correlated with the final rankings. Overall, on the tests, some looked good, some didn't. After all, if the methodology was completely sound, how could I tweak it next year to produce an entirely different ranking despite very little change in the schools?

Bias: All of the above methodology was performed prior to matching the rankings to the identities of the schools. If I have a bias, it is this: those who rank schools based on whether its faculty's books make your journal's best seller list or on the number of downloads from your proprietary system antithetical to the open access concept of working papers, or who threaten to drop schools from the rankings for adhering to ethical privacy standards, are cynical freeboating knaves.

Where I suggest a new slogan for Harvard

Harvard University has decided to require faculty members to deposit all published articles in an open-access online repository. The announcement comes with the standard (and well deserved) attacks on journal publishers, but an even greater dose of hypocrisy wrapped in lip service about dissemination of knowledge.

Continue Reading

The proposal begins with the preamble:

The Faculty of Arts and Sciences of Harvard University is committed to disseminating the fruits of its research and scholarship as widely as possible.

An exception in the policy is made, however, for articles sold for profit. Whose profit? Obviously, not the profit of journal publishers.

One faculty proponent of the open-access policy explained the policy's implications:

In place of a closed, privileged, and costly system, it will help open up the world of learning to everyone who wants to learn.

Certainly, this noble institution would lead only by example, demanding of others no more than it demands of itself! I rushed to the websites of the Harvard University Press and Harvard Business Case Studies where they would now be offering free downloads of books and teaching cases, disseminating this knowledge as broadly as possible.

Surprisingly (okay, not really), I found instead:

• A popular one page classroom example on the strategic implication of rebates costs $7 per student! That's a much higher per-page cost than most academic journals.
• In sardonic irony, a Harvard teaching note titled Should Nonprofits Seek Profits? can lead to spirited classroom discussion only after depositing several hundred dollars into Harvard's coffers.

I propose a new tagline for Harvard:

Harvard. Standing on principle (when its not our principal)

---

A note to Harvard: the above tagline is protected by copyright. All rights reserved.