USNWR reliability

(Applications Advice, Letters of Recommendation . . . )
User avatar
masochist
Posts: 247
Joined: Tue Jun 29, 2010 3:14 pm

USNWR reliability

Postby masochist » Fri Dec 17, 2010 3:02 pm

There have been a lot of debates on these forums about the validity of the USNWR rankings, but nobody has commented upon the reliability. Since reliability establishes an upper limit for validity, this seemed important to me. It also seemed like something someone should look at more closely because there appeared to be some big variations in ranking from year to year. Here is what I found …

All of these stats look only at 2010 and 2011 data and use Spearman’s rho for the metric.

Overall reliability = .971
Reliability of tier (t-3, t-6, t-14, t-30, t-50, and t-100) = .916
Reliability within t-14 = .987
Reliability within t-30 = .976
Reliability within t-50 = .971
Reliability for t-51 to 100 = .840
Reliability for t-70 to 100 = .627

This kind of confirms what I suspected. USNWR rankings are really reliable for the top schools and really unreliable for the lower-ranked schools (.627 is not good). Thus, it doesn’t make as much sense to pay attention to the rankings for the schools that are not in the top 50 since the ranking could change dramatically in a very short period of time.

And yes, I am a huge nerd.

User avatar
tgir
Posts: 314
Joined: Wed Jul 14, 2010 7:01 pm

Re: USNWR reliability

Postby tgir » Fri Dec 17, 2010 3:03 pm

This is awesome. Thanks!

User avatar
megaTTTron
Posts: 980
Joined: Thu Jan 21, 2010 4:26 pm

Re: USNWR reliability

Postby megaTTTron » Fri Dec 17, 2010 3:03 pm

masochist wrote:There have been a lot of debates on these forums about the validity of the USNWR rankings, but nobody has commented upon the reliability. Since reliability establishes an upper limit for validity, this seemed important to me. It also seemed like something someone should look at more closely because there appeared to be some big variations in ranking from year to year. Here is what I found …

All of these stats look only at 2010 and 2011 data and use Spearman’s rho for the metric.

Overall reliability = .971
Reliability of tier (t-3, t-6, t-14, t-30, t-50, and t-100) = .916
Reliability within t-14 = .987
Reliability within t-30 = .976
Reliability within t-50 = .971
Reliability for t-51 to 100 = .840
Reliability for t-70 to 100 = .627

This kind of confirms what I suspected. USNWR rankings are really reliable for the top schools and really unreliable for the lower-ranked schools (.627 is not good). Thus, it doesn’t make as much sense to pay attention to the rankings for the schools that are not in the top 50 since the ranking could change dramatically in a very short period of time.

And yes, I am a huge nerd.


I don't understand your math (i'm not smart enough) BUT your conclusion backs up widely accepted TLS-lore.

User avatar
masochist
Posts: 247
Joined: Tue Jun 29, 2010 3:14 pm

Re: USNWR reliability

Postby masochist » Fri Dec 17, 2010 3:06 pm

Sorry, I probably should have explained the numbers a little better. The coefficients listed are correlations between 2010 and 2011 rankings. High numbers mean that the relationship is strong, and low numbers mean there is not a very strong relationship between 2010 and 2011 rankings. By convention, .70 is considered the lowest acceptable reliability, .80 is marginal, and .90+ is good.




Return to “Law School Admissions Forum”

Who is online

Users browsing this forum: Bing [Bot] and 4 guests