Ranking law schools by crowdsourced paired comparisons

(Applications Advice, Letters of Recommendation . . . )
User avatar
Helmholtz
Posts: 4394
Joined: Wed Sep 17, 2008 1:48 pm

Ranking law schools by crowdsourced paired comparisons

Postby Helmholtz » Thu Jan 20, 2011 12:49 pm

Leiter conducted a study in which a large number of people answered a series of questions, viz. is X school better than Y school? So, e.g. Chicago and Columbia pop up on your screen, and you have to choose which one you think is better. There were more than 60,000 votes total and here are the results:

1. Yale University (94)
2. Harvard University (92)
3. Stanford University (91)
4. University of Chicago (87)
5. Columbia University (85)
6. New York University (83)
7. University of California, Berkeley (80)
8. University of Michigan, Ann Arbor (77)
8. University of Virginia (77)
10. Northwestern University (76)
10. University of Pennsylvania (76)
12. University of California, Los Angeles (73)
13. Duke University (72)
13. University of Texas, Austin (72)
15. Cornell University (70)
15. Georgetown University (70)
17. Vanderbilt University (69)
18. University of Southern California (67)
19. George Washington University (59)
19. Washington University, St. Louis (59)
21. University of Wisconsin, Madison (58)
22. University of Illinois, Urbana-Champaign (57)
23. Boston University (56)
23. University of Minnesota, Twin Cities (56)
25. University of Iowa (53)
26. Boston College (50)
26. Emory University (50)
28. Fordham University (49)
28. University of North Carolina, Chapel Hill (49)
28. University of Notre Dame (49)
31. University of California, Irvine (46)
32. Indiana University, Bloomington (45)
33. College of William & Mary (43)
34. Ohio State University (42)
35. University of California, Davis (40)
36. University of California, Hastings (39)
37. Washington & Lee University (36)
38. University of Colorado, Boulder (35)
38. University of Washington, Seattle (35)
40. University of Arizona (33)
40. University of San Diego (33)
42. Florida State University (32)
42. George Mason University (32)
44. University of Georgia (30)
45. Cardozo Law School/Yeshiva University (29)
46. Arizona State University (28)
46. Tulane University (28)
48. Brooklyn Law School (27)
49. Chicago-Kent College of Law (26)
49. University of Florida, Gainesville (26)

Likewise, another blog (The Conglomerate) has been doing roughly the same thing and has roughly 280,000 votes in, and is still rising. Here are the results so far:

› Yale University (CT)
89
› Harvard University (MA)
87
› Stanford University (CA)
87
› University of Chicago
87
› University of Michigan-Ann Arbor
86
› University of Pennsylvania
86
› University of California-Berkeley
86
› University of Virginia
85
› Columbia University (NY)
85
› New York University
84
› Georgetown University (DC)
84
› Cornell University (NY)
83
› University of California-Los Angeles
83
› Vanderbilt University (TN)
83
› University of Texas-Austin
83
› Duke University (NC)
81
› George Washington University (DC)
81
› Northwestern University (IL)
80
› Boston College
80
› Boston University
79
› University of North Carolina-Chapel Hill
79
› University of Southern California (Gould)
79
› Washington University in St. Louis
79
› University of Notre Dame (IN)
79
› Fordham University (NY)
77
› University of Wisconsin-Madison
77
› Emory University (GA)
76
› College of William and Mary (Marshall-Wythe) (VA)
76
› Washington and Lee University (VA)
75
› University of California (Hastings)
74
› University of California-Davis
74
› University of Washington
72
› Wake Forest University (NC)
72
› University of Illinois-Urbana-Champaign
72
› University of Iowa
71
› American University (Washington College of Law) (DC)
69
› University of Georgia
69
› University of Maryland
69
› Indiana University-Bloomington
69
› Ohio State University (Moritz)
69
› University of Colorado-Boulder
69
› Tulane University (LA)
69
› University of Minnesota-Twin Cities
68
› George Mason University (VA)
67
› University of California-Irvine
67
› University of Arizona (Rogers)
65
› University of Florida (Levin)
65
› University of Oregon
64
› University of Connecticut
64
› Cardozo-Yeshiva University (NY)
63

User avatar
acrossthelake
Posts: 4432
Joined: Sat May 16, 2009 5:27 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby acrossthelake » Thu Jan 20, 2011 2:49 pm

I'm a little confused by UMich ranking that high in the second one.

bdubs
Posts: 3729
Joined: Sat Feb 06, 2010 2:23 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby bdubs » Thu Jan 20, 2011 2:52 pm

Thanks
Last edited by bdubs on Thu Jan 20, 2011 2:54 pm, edited 2 times in total.


User avatar
bjsesq
TLS Poet Laureate
Posts: 13383
Joined: Fri Nov 19, 2010 3:02 am

Re: Ranking law schools by crowdsourced paired comparisons

Postby bjsesq » Thu Jan 20, 2011 2:55 pm

The second one is awesome. Bout time someone recognized UNC, Iowa, and George Washington are better schools than Northwestern. Wisconsin too.

bdubs
Posts: 3729
Joined: Sat Feb 06, 2010 2:23 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby bdubs » Thu Jan 20, 2011 2:56 pm



Still a bit confused by the method of gathering this info. Did they just post links on their websites? That would heavily influence the composition of the voting population.

User avatar
Zarathustraspoke
Posts: 173
Joined: Tue Nov 02, 2010 1:19 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby Zarathustraspoke » Thu Jan 20, 2011 3:02 pm

bdubs wrote:


Still a bit confused by the method of gathering this info. Did they just post links on their websites? That would heavily influence the composition of the voting population.


For the second one basically anyone can go on the website and vote.

09042014
Posts: 18282
Joined: Wed Oct 14, 2009 10:47 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby 09042014 » Thu Jan 20, 2011 3:03 pm

Michigan is a TTT in decline.

bdubs
Posts: 3729
Joined: Sat Feb 06, 2010 2:23 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby bdubs » Thu Jan 20, 2011 3:03 pm

Zarathustraspoke wrote:For the second one basically anyone can go on the website and vote.


I saw that, but is their website the only one on which that link is published, or are they trying to expose a more general populace to the poll?

User avatar
Helmholtz
Posts: 4394
Joined: Wed Sep 17, 2008 1:48 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby Helmholtz » Thu Jan 20, 2011 3:05 pm

Desert Fox wrote:Michigan is a TTT in decline.


There's still time for you to make a push at a transfer to UW-Madison. Just sayin'.

09042014
Posts: 18282
Joined: Wed Oct 14, 2009 10:47 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby 09042014 » Thu Jan 20, 2011 3:07 pm

Helmholtz wrote:
Desert Fox wrote:Michigan is a TTT in decline.


There's still time for you to make a push at a transfer to UW-Madison. Just sayin'.


I'm not top ten 10%. If I'm going to jump a mini tier I need good grades.

User avatar
Zarathustraspoke
Posts: 173
Joined: Tue Nov 02, 2010 1:19 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby Zarathustraspoke » Thu Jan 20, 2011 3:13 pm

bdubs wrote:
Zarathustraspoke wrote:For the second one basically anyone can go on the website and vote.


I saw that, but is their website the only one on which that link is published, or are they trying to expose a more general populace to the poll?


Not sure but I first saw it on the Above the Law blog...HTH

User avatar
Sentry
Posts: 1235
Joined: Tue Mar 02, 2010 6:38 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby Sentry » Fri Jan 21, 2011 3:25 pm

Chicago > Columbia confirmed.

bdubs
Posts: 3729
Joined: Sat Feb 06, 2010 2:23 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby bdubs » Fri Jan 21, 2011 3:38 pm

Sentry wrote:Chicago > Columbia confirmed.


Funny how Leiter always reaches that conclusion. I don't disagree, it's just suspicious that Chicago always fares so well in his assessments.

secondcareer
Posts: 3
Joined: Wed Dec 22, 2010 10:19 am

Re: Ranking law schools by crowdsourced paired comparisons

Postby secondcareer » Fri Jan 21, 2011 5:09 pm

I don't see how the methodology is all that accurate. I ran through 40 or so of the votes on the crowdsource link. The comparisons were pretty bad in some cases. Comparing Standford to Thomas Jefferson for example doesn't say much, but it does skew the numbers. Many of the other pairings particularly for top schools were against 3rd tier or worse.

User avatar
johnnyutah
Posts: 1709
Joined: Tue Aug 10, 2010 6:00 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby johnnyutah » Fri Jan 21, 2011 6:19 pm

acrossthelake wrote:I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

ATR
Posts: 1119
Joined: Sun Oct 03, 2010 9:18 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby ATR » Fri Jan 21, 2011 7:50 pm

johnnyutah wrote:
acrossthelake wrote:I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

Then explain how NW was below Texas, Vanderbilt, UCLA and GW.

User avatar
masochist
Posts: 247
Joined: Tue Jun 29, 2010 3:14 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby masochist » Fri Jan 21, 2011 8:00 pm

It is really interesting that the rankings are so different form one another. Crowdsourcing is supposed to generate exceptionally reliable results. They can still be wrong, of course, but they should at least be replicatable. These two are markedly different.

User avatar
bjsesq
TLS Poet Laureate
Posts: 13383
Joined: Fri Nov 19, 2010 3:02 am

Re: Ranking law schools by crowdsourced paired comparisons

Postby bjsesq » Sat Jan 22, 2011 12:58 pm

atresia wrote:
johnnyutah wrote:
acrossthelake wrote:I'm a little confused by UMich ranking that high in the second one.

Poll was full of Big 10 trolls.

Then explain how NW was below Texas, Vanderbilt, UCLA and GW.

Because they are better schools than NW. Have you noticed which TLS users go to NW?

RPK34
Posts: 522
Joined: Tue Mar 02, 2010 6:31 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby RPK34 » Sat Jan 22, 2011 1:10 pm

I'm not really sure what value these rankings have. Some pretty odd results.

User avatar
AreJay711
Posts: 3406
Joined: Tue Jul 20, 2010 8:51 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby AreJay711 » Sat Jan 22, 2011 1:22 pm

secondcareer wrote:I don't see how the methodology is all that accurate. I ran through 40 or so of the votes on the crowdsource link. The comparisons were pretty bad in some cases. Comparing Standford to Thomas Jefferson for example doesn't say much, but it does skew the numbers. Many of the other pairings particularly for top schools were against 3rd tier or worse.

That is true but if it wasn't that way then the best TTT would get the better results than the worst of the top comparison group. Also, I had to compare NYU and Columbia when I did it so their are a few tough decisions that people have to make. The real issue is that people might vote based on how much they like a given school rather than how good it is academically.

run26.2
Posts: 896
Joined: Thu Jul 01, 2010 1:35 am

Re: Ranking law schools by crowdsourced paired comparisons

Postby run26.2 » Sat Jan 22, 2011 1:57 pm

It is interesting how closely Leiter's poll tracks the US News rankings. He acknowledges that at the link above. Faculties' perceptions are likely reinforcing the rankings given that peer assessment counts for 25% of the overall score used to produce the rankings.

Rankings could change more slowly if faculties were slow to pick up on other schools' academic improvements, e.g. increase in number of published articles or hiring of new faculty members. I did not realize until I looked at the US News methodology how influential the faculties' perception was to the rankings.

User avatar
The Stig
Posts: 567
Joined: Sat Oct 30, 2010 12:27 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby The Stig » Sat Jan 22, 2011 2:05 pm

RPK34 wrote:I'm not really sure what value these rankings have. Some pretty odd results.


Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl :lol: :lol: )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)

RPK34
Posts: 522
Joined: Tue Mar 02, 2010 6:31 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby RPK34 » Sat Jan 22, 2011 2:34 pm

The Stig wrote:
RPK34 wrote:I'm not really sure what value these rankings have. Some pretty odd results.


Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl :lol: :lol: )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)


Yeah, there are issues like this that could definitely come up.

I would be much more interested if they did this type of survey with law professors, law partners, etc.

bdubs
Posts: 3729
Joined: Sat Feb 06, 2010 2:23 pm

Re: Ranking law schools by crowdsourced paired comparisons

Postby bdubs » Sat Jan 22, 2011 2:37 pm

RPK34 wrote:
The Stig wrote:
RPK34 wrote:I'm not really sure what value these rankings have. Some pretty odd results.


Maybe a proxy for how rabid a alumni/fan base is? Similar when Michigan State was ranked much higher in the football rankings during the season this year as voted by fans while they were still ranked lower by journalists etc. (then they proceeded to get pummeled by Bama in the bowl :lol: :lol: )... living in Michigan, I can say that Michigan State fans are unreasonably proud of their school... (not to say Michigan fans aren't, which is why they might be so high in these rankings!)


Yeah, there are issues like this that could definitely come up.

I would be much more interested if they did this type of survey with law professors, law partners, etc.


The opinions of "lawyers and judges" are a big part of USNWR as well, although the response rates are abysmal (something south of 20% IIRC). They tend to mimic the faculty perception rankings pretty closely.




Return to “Law School Admissions Forum”

Who is online

Users browsing this forum: Bing [Bot] and 4 guests