Leiter's open letter to USNWR Forum

[MURPH]

Prof. Brian Leiter at Chicago wrote this open letter to USNWR. http://leiterlawschool.typepad.com/
Anyone who read's Leiter's blogs will be familiar with his complaints about the popular ranking system that we applicants rely on to decide what school to go to. He has been complaining about it for at least a decade and has his own well known ranking system. http://www.leiterrankings.com/ His rankings are more nuanced. There are different lists for student quality, faculty quality, # of students who work in clerkships, academia, etc.
His complaints about USNWR include the fictional employment and salary information that schools publish, the easily manipulated and meaningless spending on student expenditures, the fee waivers offered to hopeless applicants, etc. He implies that USNWR does not investigate these numbers, rather they just accept whatever numbers are submitted by the school. Particularly, he cites Berkeley's 99% employment upon graduation. In recent years this number was as low as 74% but somehow shot up during an economic downturn. Since it is a state school, journalists should be able to easily investigate how this miracle occurred rather than just print whatever numbers are offered.

[Hattori Hanzo]

Prof. Brian Leiter at Chicago wrote this open letter to USNWR. http://leiterlawschool.typepad.com/
Anyone who read's Leiter's blogs will be familiar with his complaints about the popular ranking system that we applicants rely on to decide what school to go to. He has been complaining about it for at least a decade and has his own well known ranking system. http://www.leiterrankings.com/ His rankings are more nuanced. There are different lists for student quality, faculty quality, # of students who work in clerkships, academia, etc.
His complaints about USNWR include the fictional employment and salary information that schools publish, the easily manipulated and meaningless spending on student expenditures, the fee waivers offered to hopeless applicants, etc. He implies that USNWR does not investigate these numbers, rather they just accept whatever numbers are submitted by the school. Particularly, he cites Berkeley's 99% employment upon graduation. In recent years this number was as low as 74% but somehow shot up during an economic downturn. Since it is a state school, journalists should be able to easily investigate how this miracle occurred rather than just print whatever numbers are offered.

Crackberry will come by shortly to explain how that happened

[SandyC877]

why not quote the actual letter? It was very insightful

An Open Letter to Bob Morse of U.S. News
MOVING TO FRONT FROM MARCH 31, 2008, SINCE IT'S THAT TIME OF YEAR AGAIN

Dear Mr. Morse:

A recent article in the ABA Journal reports your continued interest in suggestions for improving the law school rankings. I know from experience with you and members of your staff that U.S. News does take suggestions seriously, and, to your credit, you made several important changes during the 1990s. (Contrary to popular mythology, I also know that U.S. News does not make formula changes each year--indeed, US News has not made any major adjustments to the ranking formula since 1999, the year you began adjusting expenditures for differences in cost-of-living in different regions of the country. Since then, U.S. News has made some adjustments to definitions of certain items of data, but those were all minor and were, correctly, designed to increase reliability by more closely tracking ABA data.)

The rationale for your law school rankings are to provide information and assistance to students, the "consumers" of legal education. In some measure you have done that through the data your magazine collects each year. Unfortunately, the success of the enterprise has largely undermined its original rationale. Because of keen student interest in the rankings, the US News rankings have become, as I am sure you know, the tail that wags the law school dog. Because moving up in the US News rankings requires no explanation, while falling invariably does, schools have grown increasingly sophisticated--or sometimes just duplicitous--in how they report data to the ABA and to US News in order to secure favorable results, results that are increasingly unhinged from any actual educational or professional accomplishments.

In consequence, the almost exclusive way in which a school improves its US News rank (apart from some arbitrary fluctuations in reputational scores, which schools can not control) is very clear: manipulation, trickery and, at worst, deceit. You know this as well as I do. Schools hire unemployed graduates as research assistants, hand out fee waivers to hopeless applicants to improve their acceptance rates, inflate their expenditures data through creative accounting or simply fabrication, cut their first-year enrollment (to boost their medians) while increasing the number of transfers (to make up the lost revenue), and so on. Because more than half the total score in U.S. News depends on manipulable data, schools intent on securing the public relations benefits of a higher rank simply "cook the books" or manipulate the numbers to secure a more favorable U.S. News outcome. Schools vary, to be sure, in how aggressive they are about data manipulation, and one expects that public law schools, whose records are subject to scrutiny, are especially careful. But there is no one in legal education who will deny, with a straight face, that a significant number of law schools, probably the majority, now "massage" their reporting, often within the letter, if not the spirit, of the rules.

So the question you confront is how to restore the integrity of the ranking enterprise, so that you continue to provide meaningful consumer information. Here are a few suggestions, in something like ascending order of importance:

1. Contrary to what one sometimes hears, it is clear to me, and I imagine any other informed observer of school evaluations, that the reputational surveys are the one component of the U.S. News ranking that actually keeps the results tethered to reality. Unfortunately, as Professor Stake of Indiana has shown, the superficial survey method U.S. News employs is increasingly producing an echo chamber effect, with the reputation of a school essentially tracking the overall rankings from prior years by U.S. News. In order to minimize that effect, I suggest you switch to an on-line survey system with academics (your response rate from academics is already quite high, and I imagine that for an on-line survey it will be even higher), in which evaluators are presented with concrete information about each school, rather than simply a school name: e.g., a current faculty roster, numerical credentials of the student body, a list of distinguished alumni (let the school provide a list, limited to 50 names, say), and so on. Ask academics to evaluate the scholarly and professional excellence of the school, not simply the "reputation" they associate with a name.

The lawyer/judge survey, by contrast, currently gets such a low response rate that the results are highly suspect. I do not know how you can increase the response rate, but you may need to put in place measures to insure geographic and practice-area diversity in the response pool. In any case, you need to make public the geographic distribution of the respondents, since I suspect that will shed important light on the reputational results.

2. To the extent you continue to employ data self-reported by the schools, you really must undertake more aggressive audits of the data. This year--to take the most notorious example that has already attracted widespread attention--the University of California at Berkeley claimed an astounding 99% of its students employed at graduation, a fact to which Professor Lindgren of Northwestern has already called attention. In prior years, Berkeley has reported (going backwards by year) 97.2% employed at graduation, 74.4%, 89.8%, 88.7%, 96.8%, and 93.2% . Berkeley is a state school, subject to open record requirements. Have you assigned a reporter for your magazine to investigate anomalous data reporting by schools? The integrity of the enterprise surely demands an occasional follow-up investigation.

3. Since what can only be facetiously called the "objective" data that schools self-report is the source of most of the egregious trickery and deceit that renders the results dubious, why not take steps to reduce your reliance on this data? (That was a primary consideration in the Canadian law school rankings I designed for MacLean's.) Eliminate expenditures altogether: that alone would put a halt to the worst offenses. What schools spend on utilities and secretaries and landscaping has nothing to do with anything. Per capita expenditures systematically penalize larger schools for their economies of scale and reward inefficiency: there is simply no denying this. Even expenditures on faculty salaries is a very poor proxy for faculty quality, and would be, in any case, redundant upon well-done reputational surveys or citation studies, which would provide a direct measure.

You should also eliminate the self-reported employment data, which is, as you well know, a work of fiction: it bears some resemblance to reality, but it is mainly a work of the imagination. Substitute data in the public domain, like the representation of school graduates as associates at leading law firms nationwide, or in federal clerkships. Eliminating expenditures data, and substituting public data on employment success for self-reported employment statistics, would immediately increase the credibility of the results, and would get U.S. News out of the business of rewarding trickery and deceit.

I hope you will make some significant changes to the ranking formula this coming year. "Gaming" the rankings only works because schools know the rules of the "game." Change them, and do so in ways that will increase accuracy and that won't permit new gaming. Your existing methods are discredited and are now disserving students, rather than informing them. (If you think the methods are not discredited and are not disserving students, then I hope you will make public a defense of the methods in light of the problems noted above, and noted here.) Do keep a reputational component, but improve your survey methods, so that evaluators are asked to respond to concrete information about schools, such as their current faculty rosters. Avoid self-reported data as much as possible, substituting information in the public domain such as the success of graduates in securing clerkships or employment at leading law firms. Scrap the nonsense about expenditures, which is responsible for some of the worst offenses in data reporting and which, in any case, has at best only a distant relationship to educational quality or professional outcomes. These kinds of changes are fully consistent with--indeed, demanded by--the standards of objectivity and accuracy that are the aspirations of any reputable news organization. I trust U.S. News will rise to the occasion, and seize the opportunity to restore the integrity of its law school evaluations as a source of consumer information.

I would be happy to discuss these issues with you further, either in private or in a public forum.

Sincerely yours,
Brian Leiter

http://leiterlawschool.typepad.com/leit ... tte-1.html

[Flanker1067]

Leiter is the only one who makes any sense 'round these parts.

[Havaianas]

good stuff - thanks

[Rand M.]

Leiter is the only one who makes any sense 'round these parts.

+1 He is seriously the only person that seems to understands how to compare schools effectively. There is no one I trust more on these issues.

[OneKnight]

nice

[Grizz]

Leiter knows what's up.

[entrechatsix]

haha awesome, i can't wait to see what they say back.

[MURPH]

He has posted open letters to them in previous years. They ignore the critics because they are in the business of selling newsmagazines not in the business of evaluating schools.

[Grizz]

He has posted open letters to them in previous years. They ignore the critics because they are in the business of selling newsmagazines not in the business of evaluating schools.

Which is funny, because USNWR is basically irrelevant as a publication except for its various school rankings.

[Flanker1067]

He has posted open letters to them in previous years. They ignore the critics because they are in the business of selling newsmagazines not in the business of evaluating schools.

I kind of think that by making their rankings more accurate and relevant they will sell more magazines. However, I am not sure that they will make up the cost of doing this, which could be guiding their decisions.

[TigerBeer]

They're selling to the American public, they don't have to be accurate. Even when 200k is on the line.

[T14_Scholly]

[MURPH]

The other way of looking at this is that they are right because their rankings are what people use. Deans of admission have gotten fired for falling in the ranks. Schools change admission standards based on their rankings and students pick schools based on their rankings. What they say goes until someone designs a ranking system that is not just better but more popular.
If TLS were to design a ranking system that worked better for students it would probably have more of an impact then Leiter's just because TLS has a more popular audience among the relevant audience.

(awesome picture posted above me BTW)

[nick637]

quick noob question on reporting job stats.

The way i understand law school job statistics is that the school contacts only the students who they think have the best chances of obtaining a job post-graduation. Is this correct?

If so, lets say if I graduate from a tier 2 school but am unable to find a job. Although the school has not contacted me, asking for information about my current job situation, I still report that I am unemployed.

My question is this: Is the law school forced to include my information into their job statistics? Or can the school just say that I never contacted them?

[Big Dog]

Obviously, beauty is in the eye of the beholder. Thus, I disagree with some of Leiter's thoughts because some of the gaming can actually improve LS life. For example, I do care if the lawn and weeds are trimmed ('landscaping'); if you don't like this factor, then why not hold classes in an unheated quonset hut. I do care that grads are employed. Indeed, isn't it better to have an RA position than none at all? Thus, IMO, it's a good thing to give a grad a job even if it boosts a ranking point or two. Why are smaller first year classes (to increase transfers) a BAD thing? Yes, I do care that the data is accurate, but USNews is not the data police (or govt). If the Dean Edley or his surrogate signs off on Boalt's submission to USNews, that's good enough for me. If they lie, the truth will come out eventually, and the embarrassment will be extreme. (Think about South Carolina's reports for undergrad that were recently published.)

[SandyC877]

Obviously, beauty is in the eye of the beholder. Thus, I disagree with some of Leiter's thoughts because some of the gaming can actually improve LS life. For example, I do care if the lawn and weeds are trimmed ('landscaping'); if you don't like this factor, then why not hold classes in an unheated quonset hut. I do care that grads are employed. Indeed, isn't it better to have an RA position than none at all? Thus, IMO, it's a good thing to give a grad a job even if it boosts a ranking point or two. Why are smaller first year classes (to increase transfers) a BAD thing? Yes, I do care that the data is accurate, but USNews is not the data police (or govt). If the Dean Edley or his surrogate signs off on Boalt's submission to USNews, that's good enough for me. If they lie, the truth will come out eventually, and the embarrassment will be extreme. (Think about South Carolina's reports for undergrad that were recently published.)

You should consider editing, or at the very least read it to yourself what you just wrote, for sake of coherence.

[stratocophic]

Obviously, beauty is in the eye of the beholder. Thus, I disagree with some of Leiter's thoughts because some of the gaming can actually improve LS life. For example, I do care if the lawn and weeds are trimmed ('landscaping'); if you don't like this factor, then why not hold classes in an unheated quonset hut. I do care that grads are employed. Indeed, isn't it better to have an RA position than none at all? Thus, IMO, it's a good thing to give a grad a job even if it boosts a ranking point or two. Why are smaller first year classes (to increase transfers) a BAD thing? Yes, I do care that the data is accurate, but USNews is not the data police (or govt). If the Dean Edley or his surrogate signs off on Boalt's submission to USNews, that's good enough for me. If they lie, the truth will come out eventually, and the embarrassment will be extreme. (Think about South Carolina's reports for undergrad that were recently published.)

You should consider editing, or at the very least read it to yourself what you just wrote, for sake of coherence.

I hope this was meant to be ironic.

[Grizz]

Obviously, beauty is in the eye of the beholder. Thus, I disagree with some of Leiter's thoughts because some of the gaming can actually improve LS life. For example, I do care if the lawn and weeds are trimmed ('landscaping'); if you don't like this factor, then why not hold classes in an unheated quonset hut. I do care that grads are employed. Indeed, isn't it better to have an RA position than none at all? Thus, IMO, it's a good thing to give a grad a job even if it boosts a ranking point or two. Why are smaller first year classes (to increase transfers) a BAD thing? Yes, I do care that the data is accurate, but USNews is not the data police (or govt). If the Dean Edley or his surrogate signs off on Boalt's submission to USNews, that's good enough for me. If they lie, the truth will come out eventually, and the embarrassment will be extreme. (Think about South Carolina's reports for undergrad that were recently published.)

The problem is, the ranking methodology is misleading for actually choosing a law school. People only nominally go to the school for the facilities or "money spent per student," they go mainly so they can get good legal jobs after. I'd go to school in the quonset hut if I knew I could get a $160k job afterward. The problem is, many people assume rankings are a good indicator of jobs, so when you can boost rankings by just spending a ton of money, it's not a good system. Money spent per student does not necessarily correlate in any way to hiring.

People don't go to law school to become RA's, they go to become lawyers. Kids look at employment stats and see 99% employed, assuming those are lawyers, but at TTT schools, a bunch of people don't get a legal job at all, they're just employed in any old job. People wouldn't go into $120k debt if they knew they had an excellent shot of being a waiter afterward. That's one think I like about Vanderbilt and Duke. They publish statistics on where every grad went and send it out. If TTT schools did that, and people saw that tons of grads were working at PF Chang's and Chili's, they might be reluctant to go. But these schools can hide behind the rankings.

Basically, the rankings give people a modicum of useful information, but not that much. They're basically useless without the published NLJ250 hiring data and clerkship data to see where the grads really go, and what kind of jobs you can get.

[notanumber]

Basically, the rankings give people a modicum of useful information, but not that much. They're basically useless without the published NLJ250 hiring data and clerkship data to see where the grads really go, and what kind of jobs you can get.

This.

And I'd argue that Leiter's rankings aren't that much much more useful than the USNWR rankings (and usually seem to give Chicago, his school, a bit of a Cooley-like-boost). Though, in his defense, he's upfront about it when his rankings extrapolate from weak data (as in his "student quality" rankings).

Why even bother ranking schools at all? A simple matrix presenting job results, class sizes, LSAT/GPA 25/75, etc... would be far more useful for students.

[Flanker1067]

We also need better job placement data. Using the NLJ250 presents alot of problems because 250 is alot of firms, and placement into them doesn't necessarily represent the placement power of a school. If a school (Yale, bastards) have many people self select away from biglaw but the ones who go all end up in V5 firms, we should know that.

Also thinking NYU, who seems to have a good reach into the very top firms although placement % was on par with many others this year. Maybe people there figured that if they couldn't get the biglaw jobs they wanted, they would go PI because of good LRAP (this is arbitrary speculation, I have no idea what really happens)

[Renzo]

why not quote the actual letter? It was very insightful

Because creating a big huge wall of text when a link will suffice is obnoxious.

[Grizz]

We also need better job placement data. Using the NLJ250 presents alot of problems because 250 is alot of firms, and placement into them doesn't necessarily represent the placement power of a school. If a school (Yale, bastards) have many people self select away from biglaw but the ones who go all end up in V5 firms, we should know that.

Also thinking NYU, who seems to have a good reach into the very top firms although placement % was on par with many others this year. Maybe people there figured that if they couldn't get the biglaw jobs they wanted, they would go PI because of good LRAP (this is arbitrary speculation, I have no idea what really happens)

It would be nice if every school just published a list of where all the grads from their most recent class were working, a la Duke and Vanderbilt, but that's not gonna happen.

[Rawlsian]

An aside: Vanderbilt has moved up to #10 for scholarly impact of faculty. Also of note: UC Irvine is 9.