Top Law Schools

There have been a lot of debates on these forums about the validity of the USNWR rankings, but nobody has commented upon the reliability. Since reliability establishes an upper limit for validity, this seemed important to me. It also seemed like something someone should look at more closely because there appeared to be some big variations in ranking from year to year. Here is what I found …

All of these stats look only at 2010 and 2011 data and use Spearman’s rho for the metric.

Overall reliability = .971
Reliability of tier (t-3, t-6, t-14, t-30, t-50, and t-100) = .916
Reliability within t-14 = .987
Reliability within t-30 = .976
Reliability within t-50 = .971
Reliability for t-51 to 100 = .840
Reliability for t-70 to 100 = .627

This kind of confirms what I suspected. USNWR rankings are really reliable for the top schools and really unreliable for the lower-ranked schools (.627 is not good). Thus, it doesn’t make as much sense to pay attention to the rankings for the schools that are not in the top 50 since the ranking could change dramatically in a very short period of time.

And yes, I am a huge nerd.

This is awesome. Thanks!

masochist wrote:There have been a lot of debates on these forums about the validity of the USNWR rankings, but nobody has commented upon the reliability. Since reliability establishes an upper limit for validity, this seemed important to me. It also seemed like something someone should look at more closely because there appeared to be some big variations in ranking from year to year. Here is what I found …

All of these stats look only at 2010 and 2011 data and use Spearman’s rho for the metric.

Overall reliability = .971
Reliability of tier (t-3, t-6, t-14, t-30, t-50, and t-100) = .916
Reliability within t-14 = .987
Reliability within t-30 = .976
Reliability within t-50 = .971
Reliability for t-51 to 100 = .840
Reliability for t-70 to 100 = .627

This kind of confirms what I suspected. USNWR rankings are really reliable for the top schools and really unreliable for the lower-ranked schools (.627 is not good). Thus, it doesn’t make as much sense to pay attention to the rankings for the schools that are not in the top 50 since the ranking could change dramatically in a very short period of time.

And yes, I am a huge nerd.

I don't understand your math (i'm not smart enough) BUT your conclusion backs up widely accepted TLS-lore.

Sorry, I probably should have explained the numbers a little better. The coefficients listed are correlations between 2010 and 2011 rankings. High numbers mean that the relationship is strong, and low numbers mean there is not a very strong relationship between 2010 and 2011 rankings. By convention, .70 is considered the lowest acceptable reliability, .80 is marginal, and .90+ is good.