Something wrong with the US News rankings?... Forum

[Clever_User_Name]

The problem with the rankings has nothing to do with the rankings themselves. It has to do with the buy-in that they have.

This.

It is naive to think that US news puts serious effort into starting from scratch every year and evaluate schools on an individual basis. Obviously there is vested interest on both sides that ensures that the top 10 of any category is going to stay virtually the same on any given year. As with all polls, surveys, and questioners, it is not a bad idea to consider who is involved, what relationship they might have, and of course, what benefit(s) they may gain from said raking, poll, etc.

That being said, I don't think US news is entirely inaccurate, but I do think it is highly subjective at a certain level; there really is not that big a of a difference between a 1st ranked school and a 10th ranked school IMO.

[MURPH]

I think the most interesting thing from that article is that for the top law schools GPA should matter more than the LSAT. "Differences in high LSAT scores are minimized when converted into
percentiles; differences in lower LSAT scores are exaggerated. For
example, the one-point difference between a 172 (98.6 percentile) and a
173 (98.9 percentile) converts to a .3 difference in percentile points; the
same one-point difference between a 153 (54.6 percentile) and a 154 (59.3
percentile) converts into a 4.7 difference in percentile points – more than
15 times larger. Although differences in LSATs accounted for 12.5% of
differences in overall scores on average, at the high end they accounted for
much less, at the low end for more."

and
"Because the effects of
median LSATs were understated at the top and overstated at the bottom,
median UGPAs ended up having a more significant effect on overall
scores and therefore on relative rankings for top-ranked schools; for lowerranked
schools, the reverse was true."
If schools strictly followed the rankings methodology when admitting students then we should be struggling to get very high GPAs and mediocre high LSATs. That doesn't seem to be the case, though. Most schools (except Berkeley and Stanford) seem to prefer very high LSATs.
One other thing that he points out, which I didn't know is that US News publishes the 25 and 75% but it uses the median score when calculating the value of LSAT score. We assume that if you are above the 75th in either GPA or LSAT you should be OK. But in fact if the schools were strictly trying to game the system they wouldn't be pushing for LSAT score above the 75th%. They would be pushing to increase their medians by one point or their GPAs by a fraction of a point.

He makes the point that schools may be trying to game the system but fail to be doing it correctly, either because they don't understand the math or don't understand how to make subtle changes that work for their benefit. For example a school could charge an extra 10K in tuition and then offer everyone a 10K scholarship. That would immediately improve the schools score at no cost to the student or school.

[Mattalones]

Check out this Loyola Law professor Theodore Seto's study...
http://papers.ssrn.com/sol3/papers.cfm? ... _id=937017
From the intro:

"Indeed, as I began playing with a spreadsheet I had written to
replicate the 2007 U.S. News computations, I discovered that even if
Harvard had reported a perfect median LSAT of 180, it still would have
been ranked third. And even if Yale had reported a median LSAT of just
153 (placing it in the “fourth tier” of law schools ranked by LSAT),9 it still
would have been ranked first. Indeed, Yale would have been ranked higher
than Harvard even if both had been true – if Harvard had reported a perfect
median LSAT and Yale a 153. I was stunned. Was Yale really that much
better than Harvard in all other material respects? If not, what might the
parts of U.S. News’ methodology that led to these counterintuitive results
tell us about the validity of U.S. News’ ranking of other schools?"

Brannigan??? Is that You>!

uhh no - I'm not Brannigan. And no, I don't really care whether Yale or Harvard is #1. I'm just trying to make some tough decisions.For non-USNews rankings I've also been looking at:
the lawdragon rankings - --LinkRemoved-- and
super lawyers rankings - http://www.superlawyers.com/toplists/la ... ates/2009/
and the Princeton review rankings - http://www.princetonreview.com/law-school-rankings.aspx

I like that you are looking at other places for ranking, but the princeton review link requires a membership. Is there any way that you could PDF the website with their ranking and link it?

[JustDude]

Alternative Ranking system:


--LinkRemoved--


Its pretty interesting I think.

Harvard is still # 1 though

Those are factors they are using:


Library Seating Capacity - To study, a student needs a place to sit. The more crowded a library, the more likely it will not be able to serve as a resource to the entire student body.

[Havaianas]

uhh no - I'm not Brannigan. And no, I don't really care whether Yale or Harvard is #1. I'm just trying to make some tough decisions.For non-USNews rankings I've also been looking at:
the lawdragon rankings - --LinkRemoved-- and
super lawyers rankings - http://www.superlawyers.com/toplists/la ... ates/2009/
and the Princeton review rankings - http://www.princetonreview.com/law-school-rankings.aspx

I like that you are looking at other places for ranking, but the princeton review link requires a membership. Is there any way that you could PDF the website with their ranking and link it?

Oh yeah I did forget about that - but the membership is actually free. I had forgotten that I even had a membership but I did from undergrad and I haven't gotten a spam mail since so I think its worth it. Also their rankings are broken down by the categories not just one overall ranking:
Best Career Prospects
Best Classroom Experience
Best Environment for Minority Students
Best Quality of Life
Most Liberal Students
Most Conservative Students
Most Competitive Students
Most Diverse Faculty
Most Chosen By Older Students
Best Professors
Toughest to Get Into

I haven't looked into their methods too much but it seems like they rank largely based on student surveys and some school statistics (i.e. amount of money spent per student on educational matters). For those of us who are willing to sacrifice some prestige for quality of life and classroom experience - I think its nice to hear at least some rankings based on student views.

The lawdragon and superlawyers rankings rate on how many lawyers are in their lists of best lawyers choosen annually. I like that this looks bast UGPA and LSAT - which must matter minimally in the real world and focuses on the actual success of their students in the real world. One of the issues with this method is whether or not you should correct for the size of the student body...

I realize that all rankings have their issues; I do not purport to hold any of these rankings higher than any others, nor do I condemn the US News rankings or their relevance. I think the best way to counter their biases and faults is to take all of them with a grain of salt and get your information from many sources.

[Havaianas]

Alternative Ranking system:


--LinkRemoved--


Its pretty interesting I think.

Harvard is still # 1 though

Those are factors they are using:


Library Seating Capacity - To study, a student needs a place to sit. The more crowded a library, the more likely it will not be able to serve as a resource to the entire student body.

Thanks for this - it looks like a good one. They say that none of their factors can make more than a 3% change in a schools ranking.

A more direct link to the ranking might be: --LinkRemoved--

edit: If you go to http://www.cooley.edu/rankings/search you can sort by different factors as well. Pretty nifty. I like the number of states in which graduates are employed factor.

[Flanker1067]

I was just looking at the Princeton review rankings, and they give BU and BC better job prospects then Harvard.

[Blindmelon]

I was just looking at the Princeton review rankings, and they give BU and BC better job prospects then Harvard.

Yea, these "alternative" rankings are even stupider than US News. Judging a school by how many attorneys sign up to be super lawyers or dragon things is the dumbest thing ever - doesn't take into account class size, or really anything important.

US News sucks, but its the best we have. If you read it in the right way (chunking schools and not just a straight ranking) its actually not that bad. E.g. GW/Fordham/BC/BU are peer schools - pretty much 20-30 is about the same. 10-16 is also pretty similar, 1-3, etc.

The dragon ranking thing put Suffolk in the top 50, and Stanford at 13... haha. Its absurd.

[Unemployed]

Loyola? Never even heard of it.

seriously there's three of em?

Bitter Lawyer fail.

What's the matter? Didn't wanna go to Harvard?

[thegor1987]

Alternative Ranking system:


--LinkRemoved--


Its pretty interesting I think.

Harvard is still # 1 though

Those are factors they are using:


Library Seating Capacity - To study, a student needs a place to sit. The more crowded a library, the more likely it will not be able to serve as a resource to the entire student body.

Thanks for this - it looks like a good one. They say that none of their factors can make more than a 3% change in a schools ranking.

A more direct link to the ranking might be: --LinkRemoved--

edit: If you go to http://www.cooley.edu/rankings/search you can sort by different factors as well. Pretty nifty. I like the number of states in which graduates are employed factor.

This is offensive, they ranked a school I want to go to lower than theirs? Should I send an e-mail complaint? Their median LSAT is gettin close to the 130s which means their class is half retard/half lazy.

This is how the e-mail would go:

Dear thomas cooley pseudo law school,

Please stop sending me advertisements in the mail as it is highly demoralizing during the difficult time of the law school admissions process. Also please change your ranking system which places Cooley ranked higher than many respectable schools, your lsat is 143, a score that a half retard could beat. Thank you,

[Havaianas]

This is offensive, they ranked a school I want to go to lower than theirs? Should I send an e-mail complaint? Their median LSAT is gettin close to the 130s which means their class is half retard/half lazy.

This is how the e-mail would go:

Dear Thomas Cooley pseudo law school,

Please stop sending me advertisements in the mail as it is highly demoralizing during the difficult time of the law school admissions process. Also please change your ranking system which places Cooley ranked higher than many respectable schools, your lsat is 143, a score that a half retard could beat. Thank you,

I must admit - the elevated ranking of Cooley is more than a bit suspicious - but again - if you look around at their rankings and sort by different factors (i.e. minority students, states employed in, etc) I think you can find some useful information. And yes Cooley ends up being high on some of the lists because they have a ridiculously large size and admit basically everyone - but we all know that so just ignore it.

[SandyC877]

Yeah, if Yale had a median 153 or Harvard had a median 180, then the acceptance rate, employment/salary, peer assessment and ALL of those factors would be different than they actually are. They would move with that variable. If anything, he proves his own point by showing that the slight variations in LSAT scores are not the exact cause for the ranking differentiation.

This is why lawyers shouldn't get involved in multiple regression analysis.

150ish vs 180 is a slight variation?

[SandyC877]

2. Assessment scores by lawyers/judges.—A second input variable
reported the results of a similar survey of lawyers and judges in the fall of
2005. Again, U.S. News did not disclose how its respondents were chosen
– how they were distributed geographically, between large and small
firms, or, in the case of judges, between state and federal or trial and
appellate courts. The 2007 issue did report that only 26% of those to
whom the survey was sent actually responded. It did not report whether
members of the group that responded differed demographically from those
to whom the survey had initially been sent. As was true of peer assessment
scores, average scores for the various law schools were published in the
2007 issue and apparently not adjusted before being incorporated in U.S.
News’ further computations.

A large part of the US News ranking is determined by Lawyer/Judge responses. Above quote in bold is highly highly highly disturbing.

[ggocat]

A few more alternative rankings:
Leiter Rankings: http://www.leiterrankings.com/new/index.shtml
TaxProfBlog used the Princeton Review raw data across multiple categories to rank the top 50: http://taxprof.typepad.com/taxprof_blog ... iew-1.html.

The Seto article, in my opinion, provides the best overall analysis of the U.S. News rankings. But Professors William Henderson and Andrew Morriss have also published a few articles analyzing particular aspects of the U.S. News rankings:
1. Student Quality as Measured by LSAT Scores: Migration Patterns in the U.S. News Rankings Era. http://papers.ssrn.com/sol3/papers.cfm? ... _id=720122
2. Measuring Outcomes: Post-Graduation Measures of Success in the U.S. News & World Report Law School Rankings. http://papers.ssrn.com/sol3/papers.cfm? ... _id=954604

The second article is pretty interesting. The conclusion is something that has been requested by current and prospective students for quite some time: "We conclude that the best solution to law schools' complaints about the impact of U.S. News rankings is greater data availability and transparency, particularly on post-graduation outcomes and other factors affecting students' eventual employment prospects."

Right now, prospective students use U.S. News as a proxy for employment prospects, but U.S. News measures employment prospects in only a limited fashion. And even those variables (employment rates) are easily manipulated.

Probably the only viable alternative to U.S. News would be a system that collects employment information such as: (1) employment rate at graduation, (2) employment rate at nine months, (3) percentage of graduates employed by each type of legal employer, (4) adjusted median starting salaries for each type of legal employer (adjusted for both cost of living AND hopefully some way of adjusting for the percentage of graduates who report salary info so schools couldn't game this factor as much), (5) adjusted number of employers participating in on-campus interviewing, and perhaps some other factors. Ideally, this information could be added to a web database for which visitors could assign their own weighting to each category (so users could decide what is most important to them and how much each factors affects the overall ranking).

[ggocat]

2. Assessment scores by lawyers/judges.—A second input variable
reported the results of a similar survey of lawyers and judges in the fall of
2005. Again, U.S. News did not disclose how its respondents were chosen
– how they were distributed geographically, between large and small
firms, or, in the case of judges, between state and federal or trial and
appellate courts. The 2007 issue did report that only 26% of those to
whom the survey was sent actually responded. It did not report whether
members of the group that responded differed demographically from those
to whom the survey had initially been sent. As was true of peer assessment
scores, average scores for the various law schools were published in the
2007 issue and apparently not adjusted before being incorporated in U.S.
News’ further computations.

A large part of the US News ranking is determined by Lawyer/Judge responses. Above quote in bold is highly highly highly disturbing.

This is pretty constant from year to year. This year, however, U.S. News averaged the surveys from the two preceding years. http://www.usnews.com/articles/educatio ... l?PageNr=1 ("About 31 percent of those surveyed responded. The two most recent years lawyers' and judges' surveys were averaged and are weighted by .15.")

[SandyC877]

2. Assessment scores by lawyers/judges.—A second input variable
reported the results of a similar survey of lawyers and judges in the fall of
2005. Again, U.S. News did not disclose how its respondents were chosen
– how they were distributed geographically, between large and small
firms, or, in the case of judges, between state and federal or trial and
appellate courts. The 2007 issue did report that only 26% of those to
whom the survey was sent actually responded. It did not report whether
members of the group that responded differed demographically from those
to whom the survey had initially been sent. As was true of peer assessment
scores, average scores for the various law schools were published in the
2007 issue and apparently not adjusted before being incorporated in U.S.
News’ further computations.

A large part of the US News ranking is determined by Lawyer/Judge responses. Above quote in bold is highly highly highly disturbing.

This is pretty constant from year to year. This year, however, U.S. News averaged the surveys from the two preceding years. http://www.usnews.com/articles/educatio ... l?PageNr=1 ("About 31 percent of those surveyed responded. The two most recent years lawyers' and judges' surveys were averaged and are weighted by .15.")

All that says is the suspected flaw is repeated each year. You need to know if that the sample response is legitimate.

[SandyC877]

Truth is, HYS and other T14 law schools, like their undergrads, will always be ranked at the top. It's a game less "traditioned" institutions cannot beat based on numbers alone. But who cares about them. What concerns me is that there's absolutely nothing in the US NEWS ranking to suggest that the quality of education or life at the top schools are better than some of the lesser ranked. Have you been inside the Yale library? it's shit, and that's just the library. Or Imagine Harvard spending even half of their tuition $ to services that directly impact their student body. No chance of that happening. They'll hoard that shit to increase their endowment for display. Maybe if all the prestige whores, by whom I mean every single one of us applying to law schools, cared about something a little more than rankings, we wouldn't see losers posting their T14 degrees on Ebay.

/end rant

[imchuckbass58]

Have you been inside the Yale library? it's shit, and that's just the library.

/end rant

The Yale Library is actually quite beautiful, but that's just my opinion.

[ggocat]

2. Assessment scores by lawyers/judges.—A second input variable
reported the results of a similar survey of lawyers and judges in the fall of
2005. Again, U.S. News did not disclose how its respondents were chosen
– how they were distributed geographically, between large and small
firms, or, in the case of judges, between state and federal or trial and
appellate courts. The 2007 issue did report that only 26% of those to
whom the survey was sent actually responded. It did not report whether
members of the group that responded differed demographically from those
to whom the survey had initially been sent. As was true of peer assessment
scores, average scores for the various law schools were published in the
2007 issue and apparently not adjusted before being incorporated in U.S.
News’ further computations.

A large part of the US News ranking is determined by Lawyer/Judge responses. Above quote in bold is highly highly highly disturbing.

This is pretty constant from year to year. This year, however, U.S. News averaged the surveys from the two preceding years. http://www.usnews.com/articles/educatio ... l?PageNr=1 ("About 31 percent of those surveyed responded. The two most recent years lawyers' and judges' surveys were averaged and are weighted by .15.")

All that says is the suspected flaw is repeated each year. You need to know if that the sample response is legitimate.

I agree, and it's likely not legitimate, considering Professor Leiter's assertion that "one-third of the law firms surveyed by U.S. News are in New York City, while U.S. News surveys no law firms at all in half the states in the country." http://www.leiterrankings.com/faculty/2 ... news.shtml

Further, although we don't know the sample size in recent years, about 10 years ago it seemed particularly small; for the 1998 edition, apparently the judge-lawyer survey was sent to 1310 individuals, and roughly 430 people responded. --LinkRemoved--.

[RickyRoe]

What I got out of this thread is that Yale should accept me because it wouldnt hurt their ranking at all. My average intelligence would add to their diversity.

[invisiblesun]

I honestly think that reporting LSAT scores by percentile would change the problem greatly. The whole T14 would range from 98-99 to 97-99 and it would really make the difference between getting 2 or 3 more questions right as inconsequential in admissions as it is in real life. It would be much more of "if you pass this baseline, you're set" than "you got two fewer questions right than the other guy, so sorry, you don't get in."

[quickquestionthanks]

Yeah, if Yale had a median 153 or Harvard had a median 180, then the acceptance rate, employment/salary, peer assessment and ALL of those factors would be different than they actually are. They would move with that variable. If anything, he proves his own point by showing that the slight variations in LSAT scores are not the exact cause for the ranking differentiation.

This is why lawyers shouldn't get involved in multiple regression analysis.

150ish vs 180 is a slight variation?

I'm referring to reality, as opposed to some strange hypothetical scenario in which Yale and Harvard become magically retarded/delicious.

So in reality, the Yale LSAT is slightly higher than Harvard's, but apparently that is not the cause for the difference in ranking. Apparently the LSAT is weighted so lightly that all of the ancillary factors overwhelm its effect. And that's probably a good thing. I think this shows the strength in the US News rankings, not the weakness.

I also agree with someone who posted earlier, that the flaw isn't in the formula, but in the process which becomes self fulfilling.

[Havaianas]

I honestly think that reporting LSAT scores by percentile would change the problem greatly. The whole T14 would range from 98-99 to 97-99 and it would really make the difference between getting 2 or 3 more questions right as inconsequential in admissions as it is in real life. It would be much more of "if you pass this baseline, you're set" than "you got two fewer questions right than the other guy, so sorry, you don't get in."

[strike]If you read Seto's article you see that basically do. They convert the LSAT scores to percentile to do the rankings. They still list them as numbers. But he also says that is a big problem because the differences in percentiles in the higher ranges are like .01% versus the much larger difference of like 10% in the medium-low ranges.[/strike]
edit - nm I reread that and I see what you're saying. Man that would be nice that once you're basically above 168 - the difference matters very little.

[brigun]

The problem with the rankings has nothing to do with the rankings themselves. It has to do with the buy-in that they have.

+1

[GeePee]

Why is the possible variation in LSAT scores to keep the same ranking among the top law schools a problem? In my mind, that is a point for the USNews Rankings, not against them.