Ranking journals is a popular pastime among academics. Each of us has a favorite ranking, largely chosen by the results fitting with our favorite publication outlets. There are more debates over the methodology of journal rankings than of ranking business schools. There may be no universal agreement on the right method but there certainly is a wrong one.
Kristie Engemann and Howard Wall have published a new ranking of economics journals. Their method consists of “a simple rule that considers citations only from a short list of top general-interest journals in economics.” In short, they arbitrarily select the “top” journals, count the number of citations from these to other journals, add an adjustment here and there for effect, and presto! We determine the top journals by counting citations from top journals. Seems a bit circular.
If you walk into a random high school and want to know who the popular kids are, the Engemann and Wall method would have you identify them by seeing with whom the popular kids choose to hang out. The procedure might produce slightly different results if you started with the debate team than if you started with the cheerleading squad.† It might not be a surprise, then, the top five journals in their results are included in the list of top journals by assumption. I don’t disagree with the list, intuitively, but science should perhaps take a more objective path.
A more objective path does indeed exist. A commonly-used recursive algorithm initially assigns all journals an equal value. Each iteration of the algorithm assigns value from one journal to another based on citations. The iterative procedure, by the way, is at the heart of Google search results (replace “citations” with “links”). From the Google founders’ monumental paper:
PageRank or PR(A) can be calculated using a simple iterative algorithm, and corresponds to the principal eigenvector of the normalized link matrix of the web.
The authors of the new ranking poo-poo this mathy stuff:
[The iterative] procedure is largely a black box: It is not possible to see how sensitive the weights (and therefore the rankings) are to a variety of factors. The obvious objection to our rule is its blatant subjectivity. Our counter to this objection is to point out that the [iterative] procedure, despite its sheen of objectivity, contains technical features that make it implicitly subjective.
Ummm… Sensitivity analysis even has its own Wikipedia page.
If Engemann and Wall were to start their own search engine, the Google formula would presumably be replaced with “pages with links from pages we like.”
Hey dummy, did you even read the article? Or are you afraid of all the "wordy stuff"? The top general journals are not called top because of their ranking. They are chosen in a first stage and are then ranked just like the rest of the journals. You missed the entire point that the iterative process is only seemingly objective. If you understood the math as well as your snarkiness suggests, then you might be able to understand that.
gocards -
The technique in the paper, summarized briefly, is (1) figure out the top 7 (why not 8? or 3?) cited journals, and declare these authoritative, then (2) rank journals by the number of cites they receive. This *IS* the LP method, highly simplified, which the paper critiques. The LP method, however, would (i) not assume 7, but use all, and (ii) iteratively determine their importance.
No, the LP method uses all journals in the list of journals being ranked, not all journals The list is arbitrary and the ranking is sensitive to things like the exclusion of field journals. It is not, therefore, objective. The Engemann/Wall ranking is sensitive only to the choice of the top 7 journals, which they happily admit is subjective but honest. The LP method is the former but not the latter.