Rules are for sissies, not MBAs

At an online forum for aspiring MBA students, participants are discussing an application essay (400 word limit) for a top ten program:

Applicant 1: Quick question guys! How stringent is the word limit? I am at 423 words.

Applicant 2: I wouldn't worry about it. I've been following the +10% max rule.

Applicant 3: mmm... it's not like they count the words right? I'm thinking if you don't push it too much, they won't even notice.

A suggested essay topic for the above applicants: "Discuss the importance of corporate ethics, respect for the law, and sound editing skills. Compare the above applicant statements to the principles displayed in recent accounting and financial scandals."

The actual essay topic in question: "What is your greatest example of leadership and what personal qualities helped you succeed in that role? (400 word limit)." How about a 500 word essay on thinking outside the box?

How to become (in)famous in under three hours

In a recent blog post, I took a tongue-in-cheek approach to the contentious topic of ranking business schools. The genesis of the post was a very different question: how to rank hospitals' success rates with a specific operation when some hospitals only accept less risky cases while others take on more challenging ones. Accepting only less risky cases should imply a higher success rate for obvious reasons having little to do with the quality of care. Business schools endowed with brighter, more capable students likewise should see higher success among their students independent of the quality of education the students receive.

To demonstrate this point, I provided a quick and dirty analysis, completed between the hours of 1 and 3 am, restricted to data on hand, and without the careful statistical standards that would constitute "research." The point was to show that changes to the assumptions underlying rankings can significantly change the results.

The resulting hoopla over the post, which begot university press releases and took my blog's traffic from a handful of loyal-reader friends into the thousands, is both enlightening and frightening. Below I offer a few clarifications.

See the clarifications

"What silly methodology! You've got to be kidding": I received many emails and comments of this form. The answer, of course, is "yes, I am." My friends, the frequent consumers of my blog, have sufficient personal context to inject sarcasm into my written word. I very much believe that there is insufficient discussion about what rankings try to measure, and whether they do this well. This point of my post was quite serious. However, the very aim of the post (literally stated) was to note that rankings (i) fail to account for self-selection, and (ii) are too easy to generate and cause mass hysteria among schools and students. The media coverage, some of which reported on these rankings without any hint of humor, seems to prove this point. A case in point: any ranking which offers "extra credit for sending me money ('investment index'), or publishing my papers ('scholarship discovery index')" was likely not intended to supplant Business Week.

The value of salary: I do not buy the premise that salaries equate to school quality. Assuming that some students care about social, career, and life issues apart from maximizing net present value of future salary stream, starting salaries may say more about the student's priorities than the quality of the school.

I did not "name" any school as a top school: A few universities took to releasing press releases, indicating that "an Economist names SCHOOL X Top Y." While I do consider myself an economist, that hardly confers authority status on business schools. My rankings aimed only to demonstrate that if you accept starting salary as a valid sole measure of schools, then one should contemplate the increase in salary a school provides, not its absolute value. These press releases, issued without any communication with me, became articles in papers (absent the tongue-in-cheek nature of the numerical rankings) heralding my "research."

The self-selection bias is ever-present: An editor at Business Week blogged about my rankings. Despite ever so slightly impugning my motives (and misspelling my name), he seems like an all-round good guy and found "a lot to recommend" about my methodology, though perhaps in part because he viewed my post as more critical of US News than Business Week's ranking methodology. While Business Week does not explicitly include GPA or GMAT in its rankings, recruiter evaluations of students still necessarily conflate the quality of the student with the quality of the school. Recruiters are not asked "how much do you think the school contributed to this individual's market value above what she would likely have had if she went to another school?" Instead, a recruiter simply saying "I like these students" can very well reflect that the kind of students that go to this school would be well-liked by recruiters even if they went elsewhere or did not pursue an MBA.

This is not "research": Not all mutterings by those of us in ivory towers constitute research. Some are just mutterings. While I would love for a journal editor to attest to my PTRC that my blog has contributed to general knowledge, the standards for research are quite different from those for personal blogs. While I was unaware of them at the time (my research priorities have nothing to do with ranking schools), several articles that did survive (or are currently in the process of) peer review offer similar methodologies or conceptual discussions for ranking business schools:

• Tracy and Waldfogel, Journal of Business, 70, 1.
• Dichev, Journal of Business, 72, 201.
• Bednowitz, CHERI Working Paper #6, 2000.
• Arcidiacono, Cooley, Hussey, International Economic Review, forthcoming.
• Devinney, Dowling, Perm-Ajchariyawong, Australian School of Business Working paper, 2007.

So why did you do this?

• geekiness
• sleeplessness
• curiosity

I should have learned my lesson last year. The only other post of mine ever to receive attention was my ranking of local restaurants by wine prices. It, too, resulted in well-placing restaurants citing my "study" under their list of "awards" and received its share of detractors from those lower down the list.

Methodology versus assumptions: If I was to rank dog breeds by average size, a careful methodology would account for variance, measurement error, etc. If I then labeled this list "The top dogs for kids" one should pause, wondering what breed size has to do with loyal pets. The methodology I employed had significant disclaimers, but was mostly correct. A follow-up by resident statistician extraordinaire Bruce Cooil found an empirical model that improves on mine. This is not surprising since Bruce ranks first on the noted Global Statisticians by Efficacy annual ranking, though his tweaking, while resulting in a statistically superior model, leads to few qualitative changes.

In any event, methodological questions like "did you account for standard error," or "did you account for ..." any of the other million issues miss the point entirely. Clearly, I didn't (though those questions are better directed to the publishers of rankings that people actually use to make life-altering decisions). Instead, ask what the rankings ought to measure, and if the methodology achieves that aim.

The best "breed" of dog, of course, is a mutt.

Where I take a turn at ranking business schools

Rankings of business schools generally fail to evaluate the inherent quality of an institution, instead ranking the people who choose to attend it.

UPDATE: If you came here from a source that did not make clear the wry, tongue-in-cheek nature of my rankings, also read the clarification.

An MBA student from UC-Davis will graduate, on average, with a starting salary that is $30,000 lower than a graduate from nearby UC-Berkeley. Can we take from this that two similarly-credentialed students at the two schools would have such a high difference in their market values? This reasoning ignores the selection bias. A student accepted by both schools is quite likely to choose the one often ranked in the top 10.

As long as top candidates choose to go to top programs, a higher ranking confounds the quality of students and the quality of a school. The proper interpretation of Business Week's rankings, for example, is not that Harvard is a better school than Blah College, but that the type of students who go to Harvard do better after graduating from Harvard than the type of students who go to Blah College after graduating from Blah. Thus, a high starting salary for Harvard graduates might imply that Harvard's professors can polish rough stone into beautiful rubies, but it is also possible that Harvard has the benefit of students who could have very well succeeded anywhere. (Note: I pick on Harvard because it actually does quite well in my rankings, supporting the rubies theory).

Which schools do best with the students they have? Traditional rankings fail to tell a given student with a given skill set which schools are most likely to increase his market value. That's the goal of these rankings, highlighting that a change in methodology significantly alters the results. Methodological disclaimers (and there are many) are at the very bottom.

On to the rankings ...

 

Ranking of Business Schools by Efficacy

			Data
Rank	School	Market       Value	GMAT	GPA	Adj. Sal
1	Cornell (Johnson)	$14K	682	3.31	118
2	Indiana–Bloomington (Kelley)	$13K	656	3.37	104
3	University of Virginia (Darden)	$12K	688	3.33	120
4	Texas–Austin (McCombs)	$12K	673	3.38	109
5	Harvard	$8K	713	3.63	134
6	Vanderbilt (Owen)	$7K	644	3.27	101
7	Rice (Jones)	$6K	642	3.25	100
8	Minnesota–Twin Cities (Carlson)	$7K	661	3.37	99
9	MIT (Sloan)	$6K	705	3.5	126
10	Maryland–College Park (Smith)	$6K	650	3.34	98
11	Georgetown (McDonough)	$6K	677	3.26	108
12	Ohio State (Fisher)	$6K	661	3.41	97
13	NYU (Stern)	$6K	700	3.4	123
14	Duke (Fuqua)	$6K	690	3.38	114
15	UNC–Chapel Hill (Kenan-Flagler)	$6K	681	3.27	110
16	Brigham Young (Marriott)	$5K	661	3.53	93
17	Rochester (Simon)	$5K	673	3.52	98
18	Texas A&M (Mays)	$5K	665	3.4	97
19	Northwestern (Kellogg)	$3K	704	3.5	122
20	Boston College (Carroll)	$3K	651	3.35	94
21	Univ. of Pennsylvania (Wharton)	$2K	712	3.53	130
22	Columbia	$2K	707	3.4	128
23	Southern Methodist (Cox)	$2K	640	3.3	95
24	Arizona State (Carey)	$2K	675	3.44	98
25	Wisconsin–Madison	$1K	656	3.37	92
26	Michigan State (Broad)	$1K	633	3.22	97
27	Chicago	$1K	709	3.5	126
28	Yale	$1K	700	3.47	116
29	Purdue (Krannert)	$0K	662	3.32	94
30	Penn. State (Smeal)	$-1K	650	3.3	92
31	Emory (Goizueta)	$-1K	685	3.3	106
32	Washington Univ, St. Louis (Olin)	$-2K	674	3.38	95
33	Michigan–Ann Arbor (Ross)	$-2K	700	3.3	118
34	Illinois–Urbana-Champaign	$-2K	627	3.4	92
35	UCLA (Anderson)	$-2K	704	3.6	114
36	Boston University	$-3K	668	3.38	92
37	Dartmouth (Tuck)	$-4K	713	3.46	127
38	Carnegie Mellon (Tepper)	$-5K	696	3.32	111
39	Georgia Institute of Technology	$-5K	665	3.4	88
40	Babson College (Olin)	$-5K	631	3.21	93
41	Stanford	$-5K	721	3.61	133
42	Notre Dame (Mendoza)	$-7K	673	3.2	95
43	Univ. of Southern Cal. (Marshall)	$-8K	689	3.3	102
44	Univ. of Washington (Foster)	$-8K	679	3.38	92
45	U. California–Berkeley (Haas)	$-9K	710	3.57	115
46	Univ. of California–Davis	$-11K	674	3.37	87
47	Univ. of Iowa (Tippie)	$-15K	652	3.34	76
48	U. California–Irvine (Merage)	$-16K	667	3.34	79
49	Univ. of Georgia (Terry)	$-16K	653	3.4	74
50	Univ. of Florida (Hough)	$-30K	680	3.4	70

 

These rankings consider only the 50 schools in the most recent U.S. News rankings. Specific methodological details are available at the bottom.

Adjusted salary (in thousands of dollars) reflects both the starting salary of those employed within three months of graduation, and a downward adjustment for those who are not.

Market value denotes the difference between a school's adjusted salary and what that school's students would expect to earn (given their qualifications at admission) at an average business school (for a loose definition of an "average" school in this context, see no. 29). A student with a high GMAT score and an exceptional undergraduate GPA is likely to receive higher offers than one with lower scores regardless of the MBA program he attends (not because of the undergrad GPA, but because of what it reveals about the person). Market value indicates how much a school improves on this given its actual student population.

The notion of market value is akin to the distinction between two corporate tasks: recruiting the best talent, and guiding that talent to its potential. Most of the popular business school rankings are biased towards achievements in recruiting, while the above rankings measure efficacy with the given talent. Of the U.S. News top ten, only Harvard and MIT are also in the top ten in efficacy. Conversely, Stanford and Berkeley, also top ten U.S. News schools, are in the bottom ten here, suggesting that members of their admissions staff deserve sizable bonuses.

By way of example, UT-Austin, Wash U, St. Louis, and UC-Davis admit nearly identical student bodies, quantitatively, yet differ greatly in the market's value of these students two years later. On the other hand, Yale and Cornell have nearly identical starting salaries, and therefore end up only one spot apart in U.S. News. Yet, given the superior class, in terms of GMAT, GPA, and selectivity, Yale should do better, thus ranking in efficacy 27 spots below Cornell, which takes the number one spot.

So, what's the goal of this? Perhaps there's a deep philosophical point about the purpose of education. I adopt market salaries as a measure of value purely because it is available, and is the most common quality measure in business rankings (or perhaps because of my unfaltering adherence to the social philosophy underlying classical economics). There's also a mundane point: rankings are not difficult to generate, easy to game, and even easier to tailor toward mass hysteria and overreaction. To that end, extra credit ("adjustment factors") will be applied to next year's rankings for posting a comment below ("brand management and awareness index"), sending me money ("investment index"), or publishing my papers ("scholarship discovery index").

 

METHODOLOGY

Disclaimer: This was done in great haste, and, in keeping with tradition of business school rankings, without too much regard for the appropriateness of statistical procedures.

Overview: The rankings are based on the residuals in a regression of average GPA and GMAT score on adjusted starting salary. That is, a school's score is the difference between its adjusted starting salary and the predicted salary from an ordinary least squares regression.

Data: Obtained from the 2009 U.S. News rankings of the top 50 business schools, which compiled data on the 2007 graduating class.

Adjusted salary: All students not employed within three months of graduation are (pessimistically) assumed to have a salary equal to 80% of the average salary of employed students at their institution. This biases results against schools with low placement rates. If S=average salary and e=percentage employed, then
     adjusted salary = Se+.8S(1-e)

Model selection: The only predictive variables of average earning potential available are average GPA and GMAT, though these are not bad as far as instrumental variables go. A model linear in GPA and/or GMAT badly fails specification tests. Various transformations of average GPA and average GMAT were tested. The final model is given by:
     adjusted salary = GPA + GMAT + GMAT^2
The model has an adjusted R²=0.70. Both GMAT parameters are highly significant (p<.001) and GPA is marginally significant (p=.084).

Residuals: From the estimated model, we obtain the studentized residuals by which schools are ranked. The "value" reported is simply the difference between actual and predicted adjusted salary.

Diagnostics: I ran all of the diagnostic tests that I could think of in three minutes. Heteroskedasticity is not a problem. Specification tests are mixed. Test for normality: not even close. GMAT and GPA are not significantly correlated with the final rankings. Overall, on the tests, some looked good, some didn't. After all, if the methodology was completely sound, how could I tweak it next year to produce an entirely different ranking despite very little change in the schools?

Bias: All of the above methodology was performed prior to matching the rankings to the identities of the schools. If I have a bias, it is this: those who rank schools based on whether its faculty's books make your journal's best seller list or on the number of downloads from your proprietary system antithetical to the open access concept of working papers, or who threaten to drop schools from the rankings for adhering to ethical privacy standards, are cynical freeboating knaves.

Where I suggest a new slogan for Harvard

Harvard University has decided to require faculty members to deposit all published articles in an open-access online repository. The announcement comes with the standard (and well deserved) attacks on journal publishers, but an even greater dose of hypocrisy wrapped in lip service about dissemination of knowledge.

Continue Reading

The proposal begins with the preamble:

The Faculty of Arts and Sciences of Harvard University is committed to disseminating the fruits of its research and scholarship as widely as possible.

An exception in the policy is made, however, for articles sold for profit. Whose profit? Obviously, not the profit of journal publishers.

One faculty proponent of the open-access policy explained the policy's implications:

In place of a closed, privileged, and costly system, it will help open up the world of learning to everyone who wants to learn.

Certainly, this noble institution would lead only by example, demanding of others no more than it demands of itself! I rushed to the websites of the Harvard University Press and Harvard Business Case Studies where they would now be offering free downloads of books and teaching cases, disseminating this knowledge as broadly as possible.

Surprisingly (okay, not really), I found instead:

• A popular one page classroom example on the strategic implication of rebates costs $7 per student! That's a much higher per-page cost than most academic journals.
• In sardonic irony, a Harvard teaching note titled Should Nonprofits Seek Profits? can lead to spirited classroom discussion only after depositing several hundred dollars into Harvard's coffers.

I propose a new tagline for Harvard:

Harvard. Standing on principle (when its not our principal)

---

A note to Harvard: the above tagline is protected by copyright. All rights reserved.

The poor children

Jake starts school next year. Public school. In Nashville. And I'm scared.

Continue Reading

The Chief Administrator of the Nashville public school system (i) can't write, and (ii) has not come to terms with his disability, or he would have hired an editor. From Benjamin Wright's page:

Mission Statement: To provide guidance and assistance to local schools in effectively fulfilling their administrative duties and responsibilities in doing whatever it takes for all students to acquire the knowledge, skills, and experiences to become productive, responsible citizens.

Lesson 1: Schools no longer teach how to diagram a sentence.

And my tax dollars fund the electrons that make this mission statement possible.^[1]

Belief Statements: accept that leaders are responsible for insuring that others are successful.

Lesson 2: Homonyms are a bitch. Or, if your child fails, call Mr. Wright to file a claim.

UPDATE: As Ben Wright might say, I'm even scareder for my son's future now, after discovering his self-promotion site.

Mr. Wright is a visionary Transformationalist with over thirty-three years experience implementing paradigm and pedagogy changes

Mr. Wright is available for speeches: "He is beyond being a motivational speaker, but rather spiritual, knowledgeable and an action oriented presenter." Perhaps your ESL group can hire him?

Want a fuller fisking?

UPDATE June 2008: Fortunately, the web page referenced in my post is no longer Ben Wright's. Despite apocolyptic predictions that he would eventually be the Director of Schools, Mr. Wright resigned in February, presumably to concentrate full time on his transformationalist pursuits. Back in December, Mr. Wright informed me by email that he "will have the page edited and corrected as soon as possible." No need to tidy things up if moving out, but the page's new resident, Assistant Superintendent Sandra C. Tinnon, has redecorated with an editor's brush, properly distinguishing between "ensure" and "insure."

^[1] The same tax dollars send over these intertubes a 1.4 MB picture of Mr. Wright to every visitor. Note for the twelve year old empowered with the html programming job for Metro schools: setting an image size within the html code does not actually change the size of the image.

Business ethics, part 2

In order to live, man must act; in order to act, he must make choices; in order to make choices, he must define a code of values ... He cannot escape from this need; his only alternative is whether the philosophy guiding him is to be chosen by his mind or by chance.

The quote, from Philosophy and Sense of Life by Ayn Rand, is a less pithy (and less fatalistic) version of Socrates: "An unexamined life is not worth living." From A Man For All Seasons by Robert Bolt:

Margaret: In any State that was half good, you would be raised up high, not here, for what you’ve done already. It’s not your fault the State’s three-quarters bad. Then if you elect to suffer for it, you elect yourself a hero.

Thomas More: That’s very neat. But look now ... If we lived in a State where virtue was profitable, common sense would make us good, and greed would make us saintly. And we’d live like animals or angels in the happy land that needs no heroes. But since in fact we see that avarice, anger, envy, pride, sloth, lust and stupidity commonly profit far beyond humility, chastity, fortitude, justice and thought, and have to choose, to be human at all ... why then perhaps we must stand fast a little -- even at the risk of being heroes.

Now, back to grading essays on business ethics ...

On grading in a world less ordinal

We all have days where a small thing makes us unaccountably happy. One of those days, for me, was when we acquired a commercial-grade blender. It had but one speed: "on." After all, when I put something in the blender I just want it to come out, after a few whirling seconds, blended. This marked the end of unnecessary confusion with an old fourteen-speed blender, which always forced me to ponder whether the button labeled "whip" resulted in faster, slower, or roughly equivalent blending action to the one labeled "frappe." Why could it not simply be labeled "Speed 8"? After all, eighth gear on a ten speed bike is not called "expeditious" but simply 8th, comfortably nestled between 7th and 9th and respecting the natural order of integers.

How did Starbucks determine that "grande" is smaller than "venti" but bigger than "tall"? Even my four year old comprehends that "large" is bigger than "small," but understandably can't differentiate between subtle cross-cultural size differentials in translation. Why is Super High a higher frequency than Ultra High but lower than Extra High?

Yet, I have persevered and mastered the stand-ins for what should be facile ordinal comparisons. Until now.

I am confronted with another simplicity-defying reclassification.

Continue Reading

The lexicographically meaningful grades that used to reflect degrees of student performance (A comes before B before C) are being replaced with HP, LP, PA, and SP. Even an X is now offered as an option for the instructor, perhaps as a placeholder for those of us who relish single-letter simplicity. After introspection and even a bit of extraspection, I fail to comprehend how a less transparent recoding of presumably ordered evaluations aids learning, efficiency, or what's left of my hair.

My son's preschool teacher grades daily behavior on a scale of zero to four stars. Thinking that was perhaps too antiquated a numerical system, I relayed the teacher's notes to my son in a more contemporary fashion. "Grande effort following directions; cleaned up toys at frappe speed. Art project, regrettably, received only an HP." A quizzical look. "How many stars is that, daddy?"

He just doesn't get it. Neither do I.

Feel free to tell me what you think of my rant. Please rate this post on a scale of Elm to Pine.

UPDATE:I am not the only geek who values the ordinal aspect of grades. The visual brilliance that is xkcd today had this:

An unintended consequence of the new grading structure: this poor little stick figure no longer will find this bit of beauty in his life.

I was supposed to READ that?

Once a day, we receive an email listing the latest classified ads posted by students, faculty, and staff. In between grading final exams for my game theory class, I notice this ad:

“Totally new”?

When I give exams, I ask students to write their names on the back of the exam only. I prefer not to know whose exam I am grading. I guarantee that this policy benefited the student in question.

As for “never use,” I assume that this reflects poor grammar rather than an indictment of Dixit & Skeath.