The “ranking game” of universities | Interview with Professor Jeffrey Stake

0:00:13.1Getting interested in rankings

Karina Macosko: Hi, my name is Karina Macosko from Academic Influence, and I’m here with Professor Stake. And on our site, we use a ranking system to decide who we’re gonna interview, and you have gotten into sort of a ranking system in yourself, so, just kinda explain to us how you got into it, and where that led you.

Jeffery Stake: Well, so I worked in a law school, and I was working in a law school 35 years ago, but about 30 years ago, maybe 25 years ago, US News published another one of its rankings of law schools, and the... The magazine described how they did their ranking, and I read it, and I thought, "Hmm, that’s kind of odd. I wonder if they really do that." And... And so, I walked into a friend’s office, who was also a former math student, we both were law professors that used to be in math, and as undergraduates, and... And I said, "Do you think they can possibly be doing what they say they’re doing?" And... And so, he said, "Wow, yeah, I don’t know." And he said, "I know where we can get the data that they use, or almost all the data that they use, and then some of it they publish, so we can add that to it, and we can reverse engineer it and see if they really do what they say they’re doing." And so, he got the data that he could get, and we combined that with the data they published, and worked quite a while, actually, sort of trying to figure out what their exact method was. And a couple of times, I actually called Bob Morris and said, "Are you doing this?", and he’d say, "Yeah."

And so... So that filled me in in a couple places. And what he... What they did at that time... And I say he, partly because it was mostly Bob Morris, I think, but what US News did at that time was they formed a ranking on different... Five different criteria, or five different summative criteria, and then they combined the ranks. And I thought that was a pretty bizarre way to do it. Now, that... You can do that, sports... There are sports that do that, and... And so, it’s not completely unheard of, but it was sort of a strange way to do things, because ranks, as I’ll probably mention later, have some bizarre characteristics, so...

So indeed, after working on it for quite a while, we confirmed independently that they were doing what they said they were doing, which was honest, but still bizarre. And so, we decided to... And I said, "Okay, we should publish this," and they... And my colleague said, "Well, if we publish it, it’ll just look like we’re complaining, and sort of that they’re doing a bad job because we didn’t get ranked high enough," so he said, "I’ll turn it over to a person I know at the AALS," I think it was the AALS instead of the ABA, but anyhow, one of those places, and... "And they can confirm that what we did was right, and what US News was doing was what they said they we’re doing."

And when they did that, US News... Sorry, the... Whatever this group was, this organizational group, they looked at it and confirmed what we said, except they used a different method. They used a regression analysis that, basically, they later published, and a lot of people... For a lot of people, it was an eye-opener, because it turned out that LSAT was really important, and more so than people thought. And so, that’s basically how I got into it to start with, was just noticing that they were doing something that, mathematically, was a little strange, combining ranks instead of combining the underlying scores on the different criteria.

So then, I thought... And then eventually, after that was published, and they were kind of exposed for that, they changed their method to combine scores instead of combining ranks. So they standardized scores, and I... And so, in the meantime, I thought, "Well, I don’t really like the way this ranking thing is going very well anyway, and I think people should rank based on their own weights and criteria," so I created what was called... What I call the Ranking Game, and it was for law school rankings, and it allowed people to put their own weights on, basically, all of the criteria that US News already used. So if you used US News’s weights, you’d get US News’s results. But if you used your own weights, then you might get a different result, and to try to show that some of this was pretty... Sketchy, I added one criterion, which was the number of Tibetan restaurants within 300 meters of the law school, and if you add any weight on that, then we would come out near the top, because we have two Tibetan restaurants near the law school. And so... Point was, not that you should care about Tibetan restaurants, but that... What you add into the mix has an effect, and sometimes, a disproportional effect.

So... So that’s how I got started, I made the Ranking Game, and then... And then later, I got worried about the effects of rankings on... On what law schools were doing, so I wrote an article. And then a statistician friend of mine says, "Well, there’s even... There’s a better way to do the statistics than you did," so we wrote another article, basically showing what I called an echo effect in my paper, and that was that when US News publishes its rankings, it has an effect on what happens the next year, in terms of the LSAT of students, in terms of the GPA of students, in terms of the reputation of the schools among faculty members, in terms of the reputation of the schools among peers. So, they all feedback, and so, US News puts out... You know, shouts out its ranking, and then they put their ear to the tracks a half year later, and they hear their own ranking coming back, basically. And so... So that was an article I wrote on that. After that, I kept up the Ranking Game website for a few years, but... But I haven’t kept it up recently, partly ’cause I don’t have the data anymore to do it. I could get the data then, but I don’t have that data anymore, so... So I haven’t done anything now with rankings for while, although I remain interested in them, obviously.

0:06:56.3The Ranking Game

Karina: Wow, so this Ranking Game was kinda to expose US News Reports and other ranking systems about how arbitrary it can be?

Jeffery: Right, right, exactly. That depends on what criteria and... And the fairly small differences in the weights can make fairly large differences. I had another factor in there, I was just remembering. So... One of the factors they used, and now, I’ve forgotten... I’d forgotten which, but there are two factors, either of which you could use, or both of which you could use. One is student-faculty ratio. One is faculty-student ratio. Well, you’d think, "Well, that’s sort of the same thing, right?" And you’d put the same weight... Not true at all. If you put faculty-student ratio, I had both in my game. And if you flip the weight from faculty-student ratio to student-faculty ratio, the rankings would change.

Karina: Wow, that is so interesting!

Jeffery: Because mathematically... Yeah, it didn’t work out the same way, and there are mathematical reasons for it, but that’s completely counterintuitive, right? "Once you’ve got student-faculty ratio in, that’s the data. That’s the information." No. How you flip that ratio... Makes a difference!

Karina: Wow, that is so interesting. Yeah.

Jeffery: Yeah, so, that was... That was one of the sorta learning things, try flipping this. I... In the early game, I had five different steps people could do. I would say, "Try doing this, see what happens. Try doing this, see what happens. Try doing this." And then, putting your own weights, if you want.

Karina: Wow, and I think this is such a great reminder for everybody who’s applying to law school, to college, to whatever undergrad or grad school, is that the rankings are pretty arbitrary, you know? We were just talking with Martha Allman, who’s the former Dean of Admissions at Wake Forest, and she was saying, "You know, one school that is a great fit for one person might not be for another person," so, I think your Ranking Game is a perfect example of how the rankings, you know... Even if one school’s ranked higher, it might not be the best fit for you.

Jeffery: Right, so there are all sorts of things that are sort of arbitrary about the original rankings, what criteria they choose. They’re never going to choose all the right criteria for anybody. There may be things important to people that are left out of the rankings. Educational... So, we’re in a big field of... Here, essentially, educational evaluation, now we’re talking about it for particular people, but educational evaluation is actually really hard, and there are sophisticated people that think about that’s... Their whole careers, and they don’t boil their numbers down to "Okay, this is the number, and now, we’re gonna rank those," when they do serious analyses of schools, and... So this is a whole discipline that has been taken over by... People who don’t know anything about the discipline, basically publishing these magazines.

So, as you were saying, there are lots of things that are important to people. For example, what happens to people after graduation, that almost never goes into the rankings, right? Pretty big important thing that people might wanna know. Are people in our field, law schools, trained for practice, or are they trained for teaching? Are they trained for being informed citizens? A lot of people learn something about democracy when they go to law school. Some people don’t, unfortunately, but a lot of them do. And then, is the curriculum itself... Even the finer points that people would never think about, and certainly, people would never rank, and that is, is the curriculum aimed at making a person a lawyer right when they graduate, and they’re ready to practice and hang out a shingle, or is it maybe aimed at making them a great lawyer in 10 years, that they’re really not gonna be ready, but they’re gonna be ready to learn more, so that in 10 years, they’re really gonna be a good lawyer? And I don’t think those are the same thing, other people might differ. But certainly, those are the kinds of things that could be very important to a student, but they’re not in the rankings. So, are people gonna be happier? Are they gonna be more influential? Are they gonna be wealthier? These are all questions that the rankings don’t even come close to measuring.

0:11:03.5Sign off

Karina: Wow, well, thank you so much for, first of all, doing this analysis, kind of opening people’s eyes. Maybe we can look past the US News and Reports rankings when we’re looking at a college, and just a great reminder for everybody out there who’s been stressing about the rankings or trying to decide on a college. It’s really great talking to you, so, thank you so much.