Ranking law schools by crowdsourced paired comparisons Forum

[Helmholtz]

Leiter conducted a study in which a large number of people answered a series of questions, viz. is X school better than Y school? So, e.g. Chicago and Columbia pop up on your screen, and you have to choose which one you think is better. There were more than 60,000 votes total and here are the results:

1. Yale University (94)
2. Harvard University (92)
3. Stanford University (91)
4. University of Chicago (87)
5. Columbia University (85)
6. New York University (83)
7. University of California, Berkeley (80)
8. University of Michigan, Ann Arbor (77)
8. University of Virginia (77)
10. Northwestern University (76)
10. University of Pennsylvania (76)
12. University of California, Los Angeles (73)
13. Duke University (72)
13. University of Texas, Austin (72)
15. Cornell University (70)
15. Georgetown University (70)
17. Vanderbilt University (69)
18. University of Southern California (67)
19. George Washington University (59)
19. Washington University, St. Louis (59)
21. University of Wisconsin, Madison (58)
22. University of Illinois, Urbana-Champaign (57)
23. Boston University (56)
23. University of Minnesota, Twin Cities (56)
25. University of Iowa (53)
26. Boston College (50)
26. Emory University (50)
28. Fordham University (49)
28. University of North Carolina, Chapel Hill (49)
28. University of Notre Dame (49)
31. University of California, Irvine (46)
32. Indiana University, Bloomington (45)
33. College of William & Mary (43)
34. Ohio State University (42)
35. University of California, Davis (40)
36. University of California, Hastings (39)
37. Washington & Lee University (36)
38. University of Colorado, Boulder (35)
38. University of Washington, Seattle (35)
40. University of Arizona (33)
40. University of San Diego (33)
42. Florida State University (32)
42. George Mason University (32)
44. University of Georgia (30)
45. Cardozo Law School/Yeshiva University (29)
46. Arizona State University (28)
46. Tulane University (28)
48. Brooklyn Law School (27)
49. Chicago-Kent College of Law (26)
49. University of Florida, Gainesville (26)

Likewise, another blog (The Conglomerate) has been doing roughly the same thing and has roughly 280,000 votes in, and is still rising. Here are the results so far:

› Yale University (CT)
89
› Harvard University (MA)
87
› Stanford University (CA)
87
› University of Chicago
87
› University of Michigan-Ann Arbor
86
› University of Pennsylvania
86
› University of California-Berkeley
86
› University of Virginia
85
› Columbia University (NY)
85
› New York University
84
› Georgetown University (DC)
84
› Cornell University (NY)
83
› University of California-Los Angeles
83
› Vanderbilt University (TN)
83
› University of Texas-Austin
83
› Duke University (NC)
81
› George Washington University (DC)
81
› Northwestern University (IL)
80
› Boston College
80
› Boston University
79
› University of North Carolina-Chapel Hill
79
› University of Southern California (Gould)
79
› Washington University in St. Louis
79
› University of Notre Dame (IN)
79
› Fordham University (NY)
77
› University of Wisconsin-Madison
77
› Emory University (GA)
76
› College of William and Mary (Marshall-Wythe) (VA)
76
› Washington and Lee University (VA)
75
› University of California (Hastings)
74
› University of California-Davis
74
› University of Washington
72
› Wake Forest University (NC)
72
› University of Illinois-Urbana-Champaign
72
› University of Iowa
71
› American University (Washington College of Law) (DC)
69
› University of Georgia
69
› University of Maryland
69
› Indiana University-Bloomington
69
› Ohio State University (Moritz)
69
› University of Colorado-Boulder
69
› Tulane University (LA)
69
› University of Minnesota-Twin Cities
68
› George Mason University (VA)
67
› University of California-Irvine
67
› University of Arizona (Rogers)
65
› University of Florida (Levin)
65
› University of Oregon
64
› University of Connecticut
64
› Cardozo-Yeshiva University (NY)
63

[094320]

..

[bdubs]

Thanks

[Helmholtz]

Source?

http://www.theconglomerate.org/2011/01/ ... kings.html
http://leiterlawschool.typepad.com/leit ... ls.html#tp

[bjsesq]

The second one is awesome. Bout time someone recognized UNC, Iowa, and George Washington are better schools than Northwestern. Wisconsin too.

[bdubs]

Source?

http://www.theconglomerate.org/2011/01/ ... kings.html
http://leiterlawschool.typepad.com/leit ... ls.html#tp

Still a bit confused by the method of gathering this info. Did they just post links on their websites? That would heavily influence the composition of the voting population.

[Zarathustraspoke]

Source?

http://www.theconglomerate.org/2011/01/ ... kings.html
http://leiterlawschool.typepad.com/leit ... ls.html#tp

Still a bit confused by the method of gathering this info. Did they just post links on their websites? That would heavily influence the composition of the voting population.

For the second one basically anyone can go on the website and vote.

[09042014]

Michigan is a TTT in decline.

[bdubs]

For the second one basically anyone can go on the website and vote.

I saw that, but is their website the only one on which that link is published, or are they trying to expose a more general populace to the poll?

[Helmholtz]

Michigan is a TTT in decline.

There's still time for you to make a push at a transfer to UW-Madison. Just sayin'.

[09042014]

Michigan is a TTT in decline.

There's still time for you to make a push at a transfer to UW-Madison. Just sayin'.

I'm not top ten 10%. If I'm going to jump a mini tier I need good grades.

[Zarathustraspoke]

For the second one basically anyone can go on the website and vote.

I saw that, but is their website the only one on which that link is published, or are they trying to expose a more general populace to the poll?

Not sure but I first saw it on the Above the Law blog...HTH

[Sentry]

Chicago > Columbia confirmed.

[bdubs]

Chicago > Columbia confirmed.

Funny how Leiter always reaches that conclusion. I don't disagree, it's just suspicious that Chicago always fares so well in his assessments.

[secondcareer]

I don't see how the methodology is all that accurate. I ran through 40 or so of the votes on the crowdsource link. The comparisons were pretty bad in some cases. Comparing Standford to Thomas Jefferson for example doesn't say much, but it does skew the numbers. Many of the other pairings particularly for top schools were against 3rd tier or worse.

[johnnyutah]

I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

[ATR]

I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

Then explain how NW was below Texas, Vanderbilt, UCLA and GW.

[masochist]

It is really interesting that the rankings are so different form one another. Crowdsourcing is supposed to generate exceptionally reliable results. They can still be wrong, of course, but they should at least be replicatable. These two are markedly different.

[bjsesq]

I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

Then explain how NW was below Texas, Vanderbilt, UCLA and GW.

Because they are better schools than NW. Have you noticed which TLS users go to NW?

[RPK34]

I'm not really sure what value these rankings have. Some pretty odd results.

[AreJay711]

I don't see how the methodology is all that accurate. I ran through 40 or so of the votes on the crowdsource link. The comparisons were pretty bad in some cases. Comparing Standford to Thomas Jefferson for example doesn't say much, but it does skew the numbers. Many of the other pairings particularly for top schools were against 3rd tier or worse.

That is true but if it wasn't that way then the best TTT would get the better results than the worst of the top comparison group. Also, I had to compare NYU and Columbia when I did it so their are a few tough decisions that people have to make. The real issue is that people might vote based on how much they like a given school rather than how good it is academically.

[run26.2]

It is interesting how closely Leiter's poll tracks the US News rankings. He acknowledges that at the link above. Faculties' perceptions are likely reinforcing the rankings given that peer assessment counts for 25% of the overall score used to produce the rankings.

Rankings could change more slowly if faculties were slow to pick up on other schools' academic improvements, e.g. increase in number of published articles or hiring of new faculty members. I did not realize until I looked at the US News methodology how influential the faculties' perception was to the rankings.

[The Stig]

I'm not really sure what value these rankings have. Some pretty odd results.

Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)

[RPK34]

I'm not really sure what value these rankings have. Some pretty odd results.

Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)

Yeah, there are issues like this that could definitely come up.

I would be much more interested if they did this type of survey with law professors, law partners, etc.

[bdubs]

I'm not really sure what value these rankings have. Some pretty odd results.

Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)

Yeah, there are issues like this that could definitely come up.

I would be much more interested if they did this type of survey with law professors, law partners, etc.

The opinions of "lawyers and judges" are a big part of USNWR as well, although the response rates are abysmal (something south of 20% IIRC). They tend to mimic the faculty perception rankings pretty closely.