US News Law School Rankings -- 5 Failures

(Rankings, Profiles, Tuition, Student Life, . . . )
User avatar
jenesaislaw
Posts: 996
Joined: Mon May 19, 2008 6:35 pm

US News Law School Rankings -- 5 Failures

Postby jenesaislaw » Mon Mar 09, 2015 2:16 pm

I just published a piece on Law.com entitled, Five Failures of the U.S. News Rankings.

Here is the body:

Tomorrow, the law school world will overreact to slightly-shuffled U.S. News rankings. Proud alumni and worried students will voice concerns. Provosts will threaten jobs. Prospective students will confuse the annual shuffle with genuine reputational change.

Law school administrators will react predictably. They’ll articulate methodological flaws and lament negative externalities, but will nevertheless commit to the rankings game through their statements and actions. Assuring stakeholders bearing pitchforks has become part of the job description.

If the rankings measured something useful, the entire charade would be much easier to stomach. The unfortunate irony is that these rankings adversely affect the decision-making process for law school administrators and prospective law students alike. The stakes are high. Our profession and society need law schools that don’t figure inefficient metrics into annual budgets. Dollars spent chasing U.S. News rankings diverts funds away from students’ education. It also stands in the way of reducing tuition.

In this post, I examine five U.S. News rankings failures. I consider the methodology and underlying rankings theory from the perspective of a student who features job prospects prominently in his application and enrollment decisions. Considering the near universal support for prioritizing job outcomes in the process, these failures demonstrate just some of the reasons annual consternation hardly seems worth it.

Image
Word cloud of LSAT test-takers surveyed about the reasons they plan to obtain a J.D. degree. Source: Above the Law.


First, the rankings pay insufficient attention to what matters most to prospective students: job outcomes. In a survey of 600 students studying for the October 2012 LSAT, Breaking Media’s research arm found that the two most popular words associated with the students’ purpose of getting a J.D. were “career” and “work” (image above). These are not exactly shocking results. Despite the importance of job outcomes, they account for only 18% of the rank and credit schools for jobs few attend law school to pursue.

Second, the rankings use a national scope, which places schools on the same scale. Only a handful of schools have a truly national reach in job placement. The rest have a regional, in-state, or even just local reach. The relative positioning of California Western and West Virginia in the rankings is virtually meaningless. Graduates from these schools do not compete with one another.

Image
Percentages in above graphic are of the entire class.

It turns out that 158 schools place at least half of their employed class of 2013 graduates in one state. The top state destination for each school accounts for 67% of employed graduates. A much smaller 8.2% of employed graduates go to a school’s second most popular destination, with just 4.5% of employed graduates working in the third most popular destination. Only 20.4% of employed graduates (16.7% of the entire class) end up in a state other than the top three. Comparing schools across the country just doesn’t make sense.

Third, U.S. News rankings follow an ordinal system that fails to show the degree of difference between schools. Are Columbia and NYU virtually tied? Or does the two-rank difference represent a wide gulf in quality? Is the so-called difference between Columbia and NYU the same as the difference between Cornell and Vanderbilt? Students weighing school prices need to know not just which school is better but how much better it is.

Fourth, performance changes over time but year-to-year comparisons are virtually impossible using the U.S. News rankings. U.S. News will tell you that Stanford knocked Harvard out of the #2 spot in 2012-13, but the swap in rankings does not provide any clues to your average reader as to why. Stanford may have improved while Harvard declined. Or, Stanford may have improved while Harvard’s quality stayed the same. Or perhaps both schools saw a decline in quality but Harvard’s decline was more severe. In fact, if every single school saw a marked decline in quality the U.S. News rankings would not indicate that this happened. Instead, students can know only relative performance.

Finally, U.S. News inexplicably places every ABA accredited law school on the list of "The Best." The best at what? U.S. News doesn’t say. But it implies that every school on the list is good. The truth is that once costs and employment outcomes are considered in comparison to personal career goals, many schools are bad choices. The U.S. News rankings provide no help in drawing the line.

Rankings are not inherently bad. In fact, they are conceptually quite useful. They order comparable things to help people sort through more information than they know how to or can weigh. However, ranking credibility may be lost when methodologies are unsound, through irrational weighting or meaningless metrics, or when the scope is too broad. The legal profession is worse off for elevating the importance of a publication that falls victim to these flaws each and every year.

User avatar
zombie mcavoy
Posts: 428
Joined: Mon Mar 02, 2015 10:11 pm

Re: US News Law School Rankings -- 5 Failures

Postby zombie mcavoy » Mon Mar 09, 2015 2:24 pm

motion to sticky

User avatar
ILoveYou
Posts: 490
Joined: Thu Jan 29, 2015 3:42 pm

Re: US News Law School Rankings -- 5 Failures

Postby ILoveYou » Mon Mar 09, 2015 2:26 pm

zombie mcavoy wrote:motion to sticky


+1

Mal Reynolds
Posts: 12630
Joined: Mon Oct 03, 2011 12:16 am

Re: US News Law School Rankings -- 5 Failures

Postby Mal Reynolds » Mon Mar 09, 2015 2:29 pm

Great post.

User avatar
runinthefront
Posts: 1105
Joined: Wed Jan 15, 2014 2:18 am

Re: US News Law School Rankings -- 5 Failures

Postby runinthefront » Mon Mar 09, 2015 2:31 pm

I think your article was great, but I do wish there was a bit more detail about the actual "methodology" that US News uses. I was vaguely familiar with the methodology, but after looking at http://www.usnews.com/education/best-gr ... s-rankings , I now feel even more disgusted by how nonsensical their approach to ranking schools is, and how much weight 0Ls place on them.

User avatar
Aftermath
Posts: 40
Joined: Tue Jan 20, 2015 1:10 am

Re: US News Law School Rankings -- 5 Failures

Postby Aftermath » Tue Mar 10, 2015 10:28 am

I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.

Mal Reynolds
Posts: 12630
Joined: Mon Oct 03, 2011 12:16 am

Re: US News Law School Rankings -- 5 Failures

Postby Mal Reynolds » Tue Mar 10, 2015 10:30 am

Aftermath wrote:I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.


Except the metrics are hard to find and based on bullshit criterion.

User avatar
Aftermath
Posts: 40
Joined: Tue Jan 20, 2015 1:10 am

Re: US News Law School Rankings -- 5 Failures

Postby Aftermath » Tue Mar 10, 2015 10:43 am

Mal Reynolds wrote:
Aftermath wrote:I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.


Except the metrics are hard to find and based on bullshit criterion.

I wasn't really arguing the validity, just that it's there. The article made it seem like no such thing existed.

User avatar
jenesaislaw
Posts: 996
Joined: Mon May 19, 2008 6:35 pm

Re: US News Law School Rankings -- 5 Failures

Postby jenesaislaw » Tue Mar 10, 2015 12:02 pm

Aftermath wrote:
Mal Reynolds wrote:
Aftermath wrote:I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.


Except the metrics are hard to find and based on bullshit criterion.

I wasn't really arguing the validity, just that it's there. The article made it seem like no such thing existed.


A 93 one year is different than a 93 another year. It's normalized based on points for #1. So there's no meaningful way to tell what the gulf between schools means.

Mal Reynolds
Posts: 12630
Joined: Mon Oct 03, 2011 12:16 am

Re: US News Law School Rankings -- 5 Failures

Postby Mal Reynolds » Tue Mar 10, 2015 1:27 pm

Aftermath wrote:
Mal Reynolds wrote:
Aftermath wrote:I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.


Except the metrics are hard to find and based on bullshit criterion.

I wasn't really arguing the validity, just that it's there. The article made it seem like no such thing existed.


I'm sure people will love you in law school.

User avatar
JCougar
Posts: 3175
Joined: Tue Dec 30, 2008 8:47 pm

Re: US News Law School Rankings -- 5 Failures

Postby JCougar » Tue Mar 10, 2015 2:05 pm

Great article, Kyle.

That's really important info about picking a law school in the state you really want to work in.

The flip side of that is that you can be from one of the coasts, and then pick a school in a city you have no ties to. If you do this, don't expect to place into that school's home market. Even cities like Chicago are sensitive about hiring locals. No city wants to be a training ground for people who will flee to NYC or SF in two years. This applies even to the elite schools.

So my advice would be to simply attend a school in your home state with expectations of working in your home state. Degree mobility is a concept that is overrated even at elite schools.

User avatar
Aftermath
Posts: 40
Joined: Tue Jan 20, 2015 1:10 am

Re: US News Law School Rankings -- 5 Failures

Postby Aftermath » Tue Mar 10, 2015 3:00 pm

Mal Reynolds wrote:
Aftermath wrote:
Mal Reynolds wrote:
Aftermath wrote:I understand why the third point is being made, but it's not entirely true. The rankings are based on a number grade which takes a bit of digging to find. To use the examples in the post, Columbia received a grade of 93 while NYU got an 89. As for Cornell, their score was 79 and Vanderbilt's was 74. So the gap between Cornell and Vanderbilt is a little wider than it is between NYU and Columbia. One could argue that this grade is not much more informative, but it is there and shows differences in quality moreso than the rankings themselves.


Except the metrics are hard to find and based on bullshit criterion.

I wasn't really arguing the validity, just that it's there. The article made it seem like no such thing existed.


I'm sure people will love you in law school.

Lol sorry if I ruffled feathers. I honestly thought it was a great article.

Mal Reynolds
Posts: 12630
Joined: Mon Oct 03, 2011 12:16 am

Re: US News Law School Rankings -- 5 Failures

Postby Mal Reynolds » Tue Mar 10, 2015 3:03 pm

I'm not ruffled at all, it was just a stupid point to make.

User avatar
Aftermath
Posts: 40
Joined: Tue Jan 20, 2015 1:10 am

Re: US News Law School Rankings -- 5 Failures

Postby Aftermath » Tue Mar 10, 2015 3:16 pm

Mal Reynolds wrote:I'm not ruffled at all, it was just a stupid point to make.

I just thought it'd be fun to try to defend the rankings. :roll:

User avatar
duckspeaker
Posts: 9
Joined: Thu Feb 05, 2015 4:39 pm

Re: US News Law School Rankings -- 5 Failures

Postby duckspeaker » Sun Mar 15, 2015 9:14 pm

I think Aftermath is mostly right here. S/he pointed out that there is reason to believe that the rankings are cardinal not ordinal, even if they are hard to find. And the cardinal/ordinal point seemed to be partially what the OP was upset about. What's more, I think that it's a stretch by the OP to say that what the rankings are measuring is exact quality. The rankings are measuring the collective criterion, which are in turn a rough heuristic for quality. Certainly not exact quality, and I don't think USNWR would claim that they are in any way exact, and thus they probably wouldn't insist on the rankings being cardinal. It's kind of a dumb point to make because asking the rankings to be cardinal and measure objective quality is a nearly impossible task, regardless of the whether or not they focus more on books in the library or employment outcomes.

That being said, the rankings could be improved. But none of OP's reasons are particularly novel - see ATL or many of these forums and you'll see that all of these issues have been raised in the past.

User avatar
lacrossebrother
Top 17 consensus poster
Posts: 6875
Joined: Wed Jul 30, 2014 11:15 pm

Re: US News Law School Rankings -- 5 Failures

Postby lacrossebrother » Sun Mar 15, 2015 9:56 pm

Dumb article.
1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.

what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment :lol: :lol: . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. :roll: :roll: dope.

moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."

2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.

3. :lol: There's a raw score you fucking moron
4. :lol: it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap.
5. ok ya i agree this is dumb. what failures.

User avatar
lacrossebrother
Top 17 consensus poster
Posts: 6875
Joined: Wed Jul 30, 2014 11:15 pm

Re: US News Law School Rankings -- 5 Failures

Postby lacrossebrother » Sun Mar 15, 2015 9:58 pm

ILoveYou wrote:
zombie mcavoy wrote:motion to sticky


+1

-1 get off this author's jock.
Maureen Dowd's letter to hillary clinton yesterday is how i feel about LST. http://www.nytimes.com/2015/03/15/opini ... ilcom.html
The fact that there might be some misinformation out there doesn't mean we need to accept you just because you're the loudest voice. This article was insanely bad.

User avatar
duckspeaker
Posts: 9
Joined: Thu Feb 05, 2015 4:39 pm

Re: US News Law School Rankings -- 5 Failures

Postby duckspeaker » Sun Mar 15, 2015 11:16 pm

Agreed, this thread is best seen as a shameless plug for a relatively unsuccessful blog and a pretty uninventive blog post. -1 and motion to not sticky.

roman law
Posts: 1
Joined: Tue Apr 07, 2015 4:03 am

Re: US News Law School Rankings -- 5 Failures

Postby roman law » Tue Apr 07, 2015 4:08 am

well done, great article

it was benefitial info.....

.......
SPAM

CanadianWolf
Posts: 10439
Joined: Wed Mar 24, 2010 4:54 pm

Re: US News Law School Rankings -- 5 Failures

Postby CanadianWolf » Tue Apr 07, 2015 11:14 am

OP & his website, LST, have done a lot of good work in advancing consumer protection matters for prospective law students. Transparent placement numbers are the least that should be expected of law schools. Many believe that law schools inflate & even fabricate placement statistics to stay competitive in the USNews rankings, while OP & LST fight for transparency & honesty in reporting so that consumers can be well & fairly informed before investing substantial amounts of money pursuing a law degree.

Whether or not one agrees with a particular approach taken by OP & LST, at least they keep the discussion going & keep the pressure on the law school industry & USNews in an effort to better inform consumers.

User avatar
Aftermath
Posts: 40
Joined: Tue Jan 20, 2015 1:10 am

Re: US News Law School Rankings -- 5 Failures

Postby Aftermath » Wed Apr 08, 2015 12:31 pm

CanadianWolf wrote:OP & his website, LST, have done a lot of good work in advancing consumer protection matters for prospective law students. Transparent placement numbers are the least that should be expected of law schools. Many believe that law schools inflate & even fabricate placement statistics to stay competitive in the USNews rankings, while OP & LST fight for transparency & honesty in reporting so that consumers can be well & fairly informed before investing substantial amounts of money pursuing a law degree.

Whether or not one agrees with a particular approach taken by OP & LST, at least they keep the discussion going & keep the pressure on the law school industry & USNews in an effort to better inform consumers.

Well put.

User avatar
Clyde Frog
Posts: 7150
Joined: Sun May 26, 2013 2:27 am

Re: US News Law School Rankings -- 5 Failures

Postby Clyde Frog » Sat Apr 18, 2015 2:11 pm

lacrossebrother wrote:Dumb article.
1. the goal is to come up with some proxy to determine what the best law schools are. anyone can list last year's employment rankings. you'll find however that those numbers fluctuate. the fact that everyone says you want to be a lawyer --that's the point of the lsat --doesn't mean that the fucking purpose of legal education is to maximize employment outcomes only.

what's fucking even more insane is that you started an entire website called "law school transparency" whereby you challenge the notion that a single percentage of employment is an appropriate means to gauge employment :lol: :lol: . so you fucking say "only 18% of your ranking is based on this really important number..." even though you think it's a shitty number to begin. :roll: :roll: dope.

moreover, "only 18%"?? you give us absolutely no idea if that's actually high or low. do a sensititivy analysis with this number you hate so much and then offer your still super layman's opinion that "this is failure number 1."

2. i don't even get this point. it's impossible to try to determine if university of arizona is a better school than uconn because people who go to school in arizona won't want to work in uconn? No shit. Perhaps this is why they don't do a fucking 100% employment ranking?? Instead, they ask judges and employers to rate their perception of the products of these schools. They try to rate the faculty. And ya, they take a look at the fucking incoming product because it's pretty easy to see that if one school, on average, has a bunch of dopes attending and the other doesn't, the fact that they aren't in competitive markets doesn't preclude reaching a conclusion. They try to gauge the respective quality of the school's infrastructure and commitment to research by looking at the size of the library (volumes is an imprecise measure, but whatever). The fact that they attempt to do this for everyone, for their magazine, doesn't make the whole thing a failure.

3. :lol: There's a raw score you fucking moron
4. :lol: it's a failure that the magazine doesn't do a more in-depth analysis of each 2-3 swap? Mr. "only 18%" is upset that the magazine doesn't explain with data graphics somewhere what the most heavily involved coefficient contributing to each swap.
5. ok ya i agree this is dumb. what failures.


This post kinda makes you look like a whiny bitch.

User avatar
lacrossebrother
Top 17 consensus poster
Posts: 6875
Joined: Wed Jul 30, 2014 11:15 pm

Re: US News Law School Rankings -- 5 Failures

Postby lacrossebrother » Sun May 10, 2015 7:45 pm

How so

Teresa
Posts: 1
Joined: Mon Jun 01, 2015 6:12 am

Re: US News Law School Rankings -- 5 Failures

Postby Teresa » Mon Jun 01, 2015 6:23 am

spam

User avatar
FullRamboLSGrad
Posts: 49
Joined: Thu May 07, 2015 6:47 pm

Re: US News Law School Rankings -- 5 Failures

Postby FullRamboLSGrad » Tue Jun 23, 2015 5:40 pm

I've always felt that while USNWR are just a bunch of hogwash it does create a sort of 'accountability'.

Every law school wants to be ranked the highest
The highest ranked law schools enjoy larger endowments and preftige allowing for their grads to take lucrative career options
Lucrative career options then help the rankings and preftige
Since these schools have to compete to be ranked as highly as they possibly can be, they have to admit quality students, and produce reasonable outcomes (obviously debatable)
If these schools didn't have to maintain medians, job placement stats and the like, then what would stop them from admitting classes of 500 0Ls without too much regard to quality? As all of graduates know, LS is a money making venture.

So while OP is right, the rankings are very imperfect, they help create an accountability and they also help students decipher that schools like
Cooley, Arizona Summit, Charleston, Pacific McGeorge are simply poor schools that aren't really worth attending.

IMO, and this may vary from the elitist hive that regularly contributes to these boards, 0Ls should do their best to attend HYS if they can't then a T14, if they can't then a T40-50 (some in this range are not good schools like Wake Forest, Hastings or Washington & Lee) and if they can't get into one of those schools they should try to get a discounted rate to attend a solid public state university with a large alumni base, in a state where they wish to live for a long time.




Return to “Choosing a Law School”

Who is online

Users browsing this forum: MikkelVilla, mtf612, tangers91 and 4 guests