"The Most Important Chart In the History of Legal Education?"
http://prawfsblawg.blogs.com/prawfsblaw ... -mone.html
« The En Banc Fourth Circuit Decides al-Marri, Sort Of... | Main | Do Women Blog? »
Wednesday, July 16, 2008
Paul Caron, Moneyball, and the Most Important Chart In the History of Legal Education
I’ve been talking about how to compare "value added" across law schools for the annual U.S. News survey of law professors, lawyers and judges, which amounts to 40% of the overall rankings. Yesterday, I laid out the kind of data we’d want to see in a Voters’ Guide to U.S. News. One reaction has been: sure, that would be nice, but it seems like a lot of work. And in the absence of the ABA, AALS, Carnegie Foundation, U.S. News, Princeton Review, or Vault (market opportunity here) funding such an effort, it will never happen. Which may or may not be right.
But we can create the race to the top right now, without any additional work, thanks to Blog Emperor and Moneyball guru Paul Caron. A basic Moneyball principle is the use of data-driven analysis to identify things that are systematically overvalued or undervalued. The chart below, created by Caron and an assistant, does that.
This fall, if it’s the day the survey is due, and you have 10 minutes to fill it out, here’s what you ought to do:
(1) Look at the chart below (click to enlarge).
(2) Give everyone in the top half a 4 or a 5, and the bottom half a 1 or a 2.
(3) Put “don’t know” for the rest or leave blank.
Princeton_review_v_us_news_2008_3
Source: Paul Caron, TaxProf Blog, October 22, 2007
This would be infinitely better than what we do now, because when the rankings come out, the “value-added” schools would gain, and the “value-not-add-so-much” schools would lose. The race to the top would be on. Why?
Here’s how the chart works. To create the Princeton Review "rankings," (Princeton Review itself does not do rankings -- they do ratings), Caron added up the following ratings from The Princeton Review: Professors Accessible/Interesting, Admissions Selectivity, Academic Experience, and Career Preparation. So The Princeton Review data includes basically everything U.S. News does, absent a few low-weight items such as volumes in the library, and the high-weight items, which are the surveys of law professors (25%), and lawyers and judges (15%). But the surveys amount to noise -- all they have done over the last few years is replicate the rankings from the previous year! So they make no real difference.
So the difference between the Princeton Review ranking and the U.S. News ranking is attributable to what is in the Princeton Review -- and not U.S. News. And that is mostly the professors accessible/interesting ratings, and responses by students in questions having to do with the "academic experience" (range of available courses, school's research resources, good mix of theory and practice in the curriculum, open to diverse opinions, how intellectually challenging the coursework is), and "career preparation" (how much does the school encourage practical experience, opportunities for externships, internships and clerkships, how prepared do you feel to start practice). In addition, Princeton Review has average starting salaries of graduates, which U.S. News does not.
Basically, the schools at the top of the chart are ones where the teaching is rated very highly, and students feel very prepared for practice. The bottom half of the chart, the school does particularly poorly on both these metrics. This chart, then, tells you what schools are doing a particularly good and bad job of adding value for students, relative to their competitors.
The key lesson here? Assessing “value added” on a relative basis by school is not only knowable; we actually know it for many schools. So we just need to present the data in a convenient and user-friendly way for survey respondents, and then the rankings will move for at least a handful of schools, beginning next spring – and then we get our race to the top, beginning next summer. Caron's chart tells us: it can be done. Which is why, in my humble opinion, this chart might well be the most important in the history of legal education.
Posted by Jason Solomon on July 16, 2008 at 12:45 AM in Life of Law Schools