Are Colleges Controlled by Rankings? | Interview with Professor Jeffrey Stake

0:00:14.7Rankings

Jed: Hi, this is Dr. Jed Macosko at AcademicInfluence.com and Wake Forest University. Today, we have Professor Jeff Stake, who is a law professor and has looked at a lot of different rankings, particularly US News and World Report, and his own ranking, the Ranking Game. So he’s kind of been around the block when it comes to ranking colleges in the ways that are typically done.

Dr. Stake, you had a chance to look at our website. So what are your initial impressions of AcademicInfluence.com?

Jeffrey Stake: Well, I haven’t looked at your whole site, but I’ve looked at it a little bit, and the first thing that I like is that you have different ways of essentially ranking or looking at the qualities of schools. And it’s very important for people to keep in mind, and this is beyond very important, it’s critical, that different people are looking for different things in schools. And so the very fact that your site starts out by having people choose what they wanna look at is an important reminder of that fact. So that’s good. How you have done the things within those sections, I don’t know that, so I can’t comment at all on with the different types of the influence or other types of things so that’s beyond me at this stage, but I like the fact that immediately you’re confronted with, "Oh, how do I want to evaluate these schools?"

The job of educational evaluation is extremely important, extremely complicated. There are people that spend their whole lives dedicated to that question. And there aren’t easy answers to a lot of this, but I think a lot of people would say it’s hard to boil it down into one number. And at least, as you start to look at different criteria, that makes sense. If somebody wants to know what the biggest school is, that might be... It’s something that does boil down to one number, but there are other things that don’t. One of the things that’s troubling really even within law schools, is that they’re not... The evaluation systems are not really designed to report on... Although it’s hard to do. I’m not saying it would be easy to do, but they’re not designed to report on what a student might wanna know to go into a given kind of field. Do they wanna be a practitioner? Do they wanna be a planner? But they’re all practitioners in a way, but do they wanna be planning? Do they wanna be doing litigation? Do they wanna be working for the government? Do they wanna be working for not-for-profits? Do they wanna be working in the environmental field? There are so many different things and schools are better or worse at different things within those categories.

And so it’s important for people when they look at any kind of academic aid in getting them to get to a right choice or a good choice, I don’t know if there is a right choice, but a good choice for a school to go to, that they’re reminded constantly that they should be thinking about lots of different factors. And US News and other single summary numbers really don’t do that at all. They reinforce just the opposite. And they reinforce in ways that are very, well, very troubling and very misleading. Ranks themselves, which US News eventually publishes, right? They do publish the numbers, so I give them credit for that, so you can actually get the numbers, but I just looked at this year’s numbers. One of my pet peeves really is that, if you look, you can imagine, and I won’t even mention the schools, but you can imagine schools ranked say, 30... Three and six or something like that, not being very far, but they’re in the top group, number three or number six, something like that. And if you look at that difference, I think that difference is five points in the US News numbers. Whatever those numbers are, and it comes out to five points, and I could check on that.

But then you take that same five points and you say, "Well, what does five points do further down the scale?" And so you look and you’d find out that it’s the same difference between a school ranked 28 and a school ranked 45. So if you could rationally say, "Well, I’ll go to school six instead of school three," you could also just as rationally, and even if all these numbers are right, setting that aside, but, you could just as rationally say, "I’ll go to 45 instead of 28." But 45 and 28 sounds like they’re a long way from each other, but they’re not. They’re just as far from each other as school three and school six are on the, on US News’s criteria, which I don’t defend to start with. And they have stuff built into it. Like for example, what would you rather do? Would you rather be a top student at a lower-rank school or a middle student at a higher-rank school?

Jed: I don’t know. Yeah.

Jeffrey: I don’t know, exactly. I don’t know either. I don’t know the answer for myself, but what does US News assume? They assume that the higher the LSAT, the better the place is for you. But the higher the LSAT and the higher the grades, the more likely you’re gonna be in the middle of the class. I don’t mean you personally...

Jed: No, no, no, I get it. Yeah.

Jeffrey: The very thing they’re pointing to and putting 22% or whatever it is of their weight on, is a factor that is ambiguous from a student’s point of view. You might say, "I wanna be with the best students around I can be with ’cause I will learn from them, and I will interact with them, and that’s how I’m gonna get great legal friend," and that could be completely right. Or you could say, "I wanna be where I’ll be at the top of the class. I’ll be on the law journal and I will get the job offers ’cause the firms come in and they look for the law review people and they don’t pay that much attention to whether I’m a little higher or lower ranked in the schools. And I will get all the credit from professors and I will have special treatment because I did things with the school that very few people could do." That’s rational too, but that’s not the way US News sees it. US News sees it as, "This is better. You should be going to the school that has a higher LSAT," that’s just not true.

0:06:43.6The echo effect

Jed: Yeah. Well, it’s just fun to talk to you. Clearly, you’ve thought about this over many years. And you have actually talked about the echo effect that US News and World Report has had on law school, probably on other things as well. What do you think, first of all describe a little bit again about the echo effect and how you proved it was there, ’cause I don’t think people really understand that.

Jeffrey: Yeah. Well, it’s not a proof in a mathematical sense, but it’s sort of a proof in the statistical sense. So what I did, and then, I got the help of a statistician to help me do a second time around, but was to take almost all the data that US News has and say, "Here are the ranks that US News publishes. And then, what happens to the different sub-scores the following Fall or the following Spring after they’ve published that?" So, if school Z goes up four points in the ranking in year zero. And or, I’ll say year one compared to year zero goes up four points. Then, what happens in year two?

When you look at the students who apply, and the LSAT and the GPA, and when you look at the professional reputation among the peer schools. What about the professional reputation among lawyers and judges? And I found that all four of those things, we found that all four of those things go up after US News publishes an increase in rank, or go down after US News publishes a decrease in rank. US News is itself influencing the criteria that they’re using to rank schools. And so, what they’re hearing is essentially an echo of their own report. And so they publish this thing and it says, "Here’s the rank." And all the different groups pay attention to it. The student pay attention to it. The faculty pay attention to it. The lawyers pay attention to it. And then, when US News does is survey, they get the same information back that they already published.

Jed: Well, I mean, I don’t disagree that you’ve found something, which I think is probably true.

Okay, but playing Devil’s advocate, couldn’t somebody say, "Well, US News is following an upward trend that just keeps going up, or following a downward trend."

Jeffrey: No. Because it can go up or down. It doesn’t just go up, or doesn’t just go down. And so, it goes both ways. And this is what it takes the statistician to do. And I couldn’t do on my own. There are some sophisticated statistical techniques that are used to determine which one is following which.

Jed: Oh, okay. Well, very cool. We’ll have to read your papers to convince ourselves that this is all real, but I’m sure it is because everybody pays attention to those rankings in a way that, yeah, it is pretty amazing when you think about it.

Jeffrey: It’s stunning.

Jed: Yeah. This was supposed to be...

Jeffrey: How much attention or not they’re paid.

Jed: Yeah, if you learn the history of this. It was supposed to be sort of a gimmick that increased readership of the US News and World report, it was falling behind Times and Newsweek. And so, this is trying to boost it up. And now, it’s definitely... The journal itself has kinda gone by the way side, this is US News and World Report, it’s kind of funny that...

Jeffrey: That’s exactly right.

0:10:17.1The impact US News and World Report has…

Jed: Yeah, the tail is wagging the dog now, sort of, in this ranking. But there are a lot of other rankings out there.

So you’ve mainly focused on US News and World Report? Have you looked at other?

Jeffrey: There are but if... And I acknowledge that in some areas, other rankings are either more important or equally important to US News. One of the things that bothers me is that in the law area really US News is the big player. Students walk into the law fairs... So for example, admissions committees, I’m the chair of admissions at IU, but the admissions committees will go and send a representative to a school, Illinois or anywhere, Ohio State, any place else and say, and they all sit at tables. And it’s law fair day or whatever they call it. And the students walk around and they talk to different schools. And so, they’ll look at the school, and there will be a little poster that says Indiana University. And they’ll look at the school. And they’ll walk up and talk to them, "What do you have in Indiana University, and how much does it cost?" And things like that. And a lot of students will have US News in their hand. And they will look up, and they go, Indiana University, and then, they’ll look down, and they’ll see whatever it is this year, and they’ll go, "Okay." And they move on to the next school, or they come to talk to us. Whatever it is. But they are making initial cut decisions based on US News and almost nothing else. And it’s sort of horrifying actually to think that they’re doing that.

Another horrifying thing, another colleague of mine, and we didn’t publish this, a different colleague. And I studied from, I think it was lawschoolnumbers.com. There’s a site where students used to put in the amount of scholarships that they got from different schools. And then they put in what school they chose to go to eventually. And of course, some people... This is all self-reported data. So, you can take it with a grain salt. So we scrape that data and we analysed it. And students on average were paying $10,000 to $20,000 for a step-up of one in the rankings.

Jed: Wow, that does not seem cost effective. That does not seem cost effective.

Jeffrey: No. No.

Jed: Interesting.

Jeffrey: No, definitely not. So, it’s not really rational at that point.

Jed: Wow.

Jeffrey: And then, and again, back to my earlier point about rankings, some of these rankings are very small. Let me add a caveat, that was at the top of the range. So once you get down to the middle of the range of the schools, they were not paying $10,000 or $20,000 for a step up one, but once you got into the top 10, now they’re starting to pay big money to step up one, and it’s not even clear, to go back to the criteria, that these are even better schools because of they’re being ranked as higher in US News or better for that student.

Jed: Well, it definitely has changed the game. It used to be probably that you would go to a law school for the education that you get and for the smarts that you would get, that then you would use when you got out. But now it seems like you’re going to get a little trophy or a little pedigree that says, "I went to this ranked school."

Do you think that that’s partly why more of the recent presidents have had Ivy League or top schools in their pedigree, and the previous presidents, it wasn’t the case, presidents of the United States, and is this where we’re headed?

Jeffrey: Wow. Well, I thought of a lot of effects of US news, but I’d never thought of effects on choosing a president.

Jed: Well, or other effects. These are just things that came to my mind, the last five presidents, I was like... Go ahead.

Jeffrey: Yeah, no. I like your question, I’m not making fun of it. I like your question because it does get into the heads of people, and people start to think, "Oh well, they’re the best because they went to an Ivy League school," or whatever, ignoring that, some of the great people we’ve had throughout history came from a lot of places that they had never heard of, and that schools don’t necessarily make that much difference anyway. One of the things that I wanted to make sure I mentioned, and you probably know about this, but your audience might not, was a study by Stacy Dale and Alan Krueger , an economic study. Now, this is more FDA news money, so I acknowledge that’s a limitation, but showing that the... How did this go? The school that you applied to and didn’t get in, the SAT of the college that you applied to and didn’t get into, is a better predictor of your economic future than the school you went to.

Jed: Now, that is interesting. So, people who reach for the stars may be either A, an optimist that then that determines that they’re gonna make money in the long run, or B, they’re from a wealthy family that says, "You should apply to these top schools," and they’re also from a wealthy family that helps them get a start in life and business, and therefore they do well.

Of those two choices, which one do you think is the bigger factor here?

Jeffrey: Yeah. I don’t know, I don’t know, and then there may be others as well. But it’s mind-boggling to think that it’s not the school you went to, but the school you didn’t go to that determines more of your future economically.

Jed: Wow.

Jeffrey: And I think this goes back to the point of people paying a lot of money for stuff that they may not... This is not really necessarily rational behavior for them to be doing this. They had done another study as well, so it’s worth reading them, but they’re basically undermining the idea that all these prestige factors are necessarily gonna lead to a better economic outcome in the future. Now, there are sub-groups actually within some of the studies that do better, so if you’re from an under-privileged background and you manage to get into one of these schools that will put you in contact with people who are in the elite, then that may be an important step for you to take. But if you’re already within the elite or already in a fairly well-educated family, then what school you go to doesn’t make as much difference as you might think. So, a lot of the whole point of the ranking is to find something that’s gonna be beneficial in the long run, and yet some of their studies undermine the idea that these differences play out in the way that you’d expect them to.

Jed: Yeah.

Jeffrey: One more thing about, that I can’t resist saying...

Jed: Go ahead.

Jeffrey: That bugs me about this, and I’m happy to answer more questions, but I just gotta get this off my chest in a sense, is the effect that these rankings have on the schools themselves. And so they start to give priority in a greater way than they used to, to LSAT or SAT, and grades from before that, and other factors, because that’s what fits into the ranking and makes them look better. Instead of trying to choose the people, for example, "Oh, that person was a person who served in the military, did a great job in overseas, when they were in the military." "Well, yeah, but their LSAT is no good, forget about that person." And so we’re overlooking people, I think, across the nation in law schools and undergrad institutions as well, who might be really good candidates, but they’re not doing anything for our numbers, and so we’re saying, "Ah, well, maybe not so important."

Jed: Yeah.

Jeffrey: Now, if you go back two slides to what I was saying before, maybe it doesn’t hurt that person that much, but it still seems unfair to me that people are being overlooked because their numbers weren’t so hot when at least the data 30 years ago would have said, that that shouldn’t have been that important. We changed not because we realized how important the SAT was, but we changed because we realized how important US News was, and US News says the SAT is important to their rank. And so there are other effects like that, law schools now have transfer programs that they didn’t used to have, barely at all when I was in law school, and now they have these big transfer. Why? Because you can shrink your class, have higher LSAT and grade point average, and then to get your money, you bring in students from other schools after that, and then they fill out the ranks and you’re getting tuition from those.

Jed: Wow.

Jeffrey: And so what do they do? It just takes 100 people or 50 or whatever the number is, and it means that they have to switch law schools in the middle of their law school years. So it’s just damaging to the curriculum, I get a student like that, and I can’t count on them having a certain thing in the first year, because they took a first year course at some other law school. But that’s what schools are doing it because US News is so important, the rankings are so important to these choices, so it’s having a feedback effect on our educational system as well as on the students.

0:19:39.7The future for law school

Jed: Wow. Well, I do have more questions, and since you’re willing to take them, one of the questions I have is…

…what is the future for law school in terms of online programs and programs at smaller schools, programs at state schools, because from my vantage point in the undergraduate and physics graduate world, people are less and less likely to pay big money for programs that are not what they would consider premier. So unless you’re in the top 50 undergraduate universities in the nation, or if you’re in one of the inexpensive state schools in your home state, you’re really not that excited to spend $70,000 a year on an education that you could get online. Is that the same way it feels to students in law school who don’t get into one of the top law schools or into a state law school like where you’re teach?

Jeffrey: I’m not sure how it feel to students, at least comparing what they had hoped to get into versus what they did get into. So for some students, the best they could get into was Indiana University and they’re glad to be here, for other students, they might have turned down, perhaps, although it doesn’t happen much, but turn down a higher ranked school because we gave a better scholarship or something like that, so I don’t know whether they ever regret their decisions if they had other choices, but I agree that they view... Well, the cost is going up a lot, and so there’s more consumerism sort of naturally, because the cost has gone up so much. When I went to undergrad in law school, it cost a lot less, and so I didn’t worry about the money very much. Now they have a rational worry about the money and the debt that they’ve incurred in order to go to law school. So in order to make it feel good, they have to feel like there’s gonna be an outcome that will be worth this. In the legal world, there’s been a study on this that was done years ago, and I don’t remember...

I don’t know if it’s now updated at all, but the study that was done a decade or two ago indicated that the pay off from law school is still pretty good, now not necessarily from the very sketchy law schools that many people don’t even pass the bar after going to the law school. So the schools that have a very low bar pass rate, you have to wonder about whether they’re getting what they’re paying for it. Now they could be just wanting an education in the law and that’s fine, I have nothing against that. But if they’re thinking they’re gonna be a lawyer, and then they don’t pass the bar, that may not be a real great pay off for them. So count out some of the schools, I’m not including all law schools, but within a lot of the law schools, the payoff was pretty good to a law degree, and so it was a good investment for the students. They might feel like they wish they’d gone some place else where the payoff might be better but I think once they get into law school and they go and they put in a lot of time and effort, for the most part, I don’t think many are looking back at, "I wish I had gone to somewhere else."

Now, there are exceptions to that, and that would be the people who had hoped to be employed in... Well, in the extreme case wanted to go to the Solicitor General’s office, if they even knew what that was, and in the Department of Justice. Well, not many people make it there, it’s only a few schools that send very many people there, and if they wanted to go there and then they went to law school and thought I can’t get to the Solicitor General’s office, they could be disappointed about that, but I don’t think they’re very disappointed comparing the different schools they could have gotten into. With regard to online, which was the other half of your question.

Jed: Yes.

Jeffrey: I would have been... I was just two years ago, very worried about that because I thought, "Just do it online, it’s a lot cheaper, we’re gonna all be replaced by a great lecture, by somebody," and that’s the future of legal education. Now, I knew it wasn’t the complete future of legal education, but I was pretty concerned about it. The pandemic changed my thinking entirely. My students are so much happier to be in person, in a classroom, now this semester, than they were last year when they were online, I asked them... I just happened to... This is just coincidental that you’d asked me about this, but I asked them just last week, "How are you feeling about this semester?" ’Cause for me, it’s kind of a mixed bag. I can’t see their faces because they’re behind masks, and so I can’t see them smile whereas when it was online, I could see them smile or laugh or whatever, and so I don’t get as much feedback. And because I wear glasses a lot of the time, my mask fogs up my glasses, so I can’t even see what’s left of their face. So for me, it’s a lot harder than it was when I was teaching online. For them, the ones I asked, it’s a world of difference.

Jed: Okay.

Jeffrey: They are so happy to be in person, seeing the teacher, talking to their peers, they don’t mind the masks, they don’t mind the all the other routines that we’re going through. They are delighted. All the ones that I talked to. Now, maybe I didn’t get the right people. But I was pleasantly surprised, it was like, "Oh good." Well that reaffirms that I’m glad to be back in the classroom and doing something useful for them. But if that’s the way they feel, then we’re not gonna wipe out in-person law schools with online law schools very quickly, and you can easily imagine why, but yeah. But I would have thought, especially because the students are so flexible. They’re young, you could think, "Oh, well, they got used to online and now they come back in person and it’s sort of disappointing that it’s not really that much different and have to go through these stupid routines and they have to walk to class." No, that was not their attitude at all.

Jed: Wow.

Jeffrey: Students was so glad to be back here.

Jed: Well, that is encouraging, yeah.

Jeffrey: So I think that... Well, there is... We sometimes hope as teachers that there’s this interpersonal connection that we’re making and that the students saying something and we’re responding to something, and when you get... Now it could happen online, but it doesn’t happen as well online, and so I think that... And especially in a big class where they don’t seem to talk very much at all because there’s so many other people on the screen or whatever, but I can have a class of 60 or 70 students and they’re raising their hands and they’re talking, and so I was relieved, if you wanna put it that way, but I’m really glad to hear that in-person teaching will carry on for a while, and that the pandemic at least has reinforced the importance of developing that relationship between the teacher and the student, and the dialogue that goes on between the teacher and the student. If you had just a plain old lecture, then I don’t know, maybe a lecture online would be just about as good. But I lecture a lot and the students were much happier to have me lecture in person than have a lecture online.

0:27:20.1Sign off

Jed: Good. Well, it has been really fun to talk to you Professor Stake and I’m really glad that you’ve spent so much time thinking about the issues of ranking and also the issues of what is the future for law education in United States and elsewhere. So thank you for taking the time to share all that with us, we really appreciate it.