quakeroats wrote:It brings in many fewer issues than assessments scores with 30% response rates and poor question phrasing. Outside of Yale-type problems I'd say it's a pretty good measure.
I know I'm about 2.5 months late on this, but these critiques are over-hyped. First, Leiter's problem with the question-phrasing is applicable only to the Peer Assessment scores, and his problem with the response rate is applicable only to the Lawyers/Judges Assessment scores. You bundle all the criticism together and apply all of it to both in order to discredit them to a larger degree--probably because you don't like the results (Duke at 11).
Second, Leiter's question-phrasing criticism is that faculty lists are not included with surveys distributed to the "peers." But neither are placement stats. Or revenue. Or student accomplishments. Or the size of the library. If the survey was about which faculty was best, then yes, Leiter could legitimately argue that a list of a school's top professors should be included with the survey--but it's
not about which faculty is best. The survey is about which school is best, and there are so many more factors that should be taken into account. The fact that U.S. News does not provide all relevant information to the survey takers--which would frankly be impossible anyway--does not flatly discredit the results of the survey. Including cherry-picked information would actually skew the results by emphasizing only one subset of all the information that should be taken into account--which of course is exactly what Leiter wants: in his view, Chicago has a better faculty than Harvard and Stanford, and I'm sure he'd love for U.S. News to come out that way.
Third, a low response rate is not, in and of itself, a discredit to the results of the survey. A sample of 600 respondents is a very large sample, regardless of the size of the population. Assuming no biases in the survey, a sample of 600 is going to yield results with a with a very low margin of error at a high confidence level (95%).
Leiter's strongest criticism of the Lawyers/Judges survey is not response rate--it's (potential) geographical bias. Leiter claims that lawyers and judges from New York City are more likely to respond than lawyers from other regions. Again, I'm sure Leiter is concerned with this potential bias principally because Columbia is rated higher than Chicago. I'm not sure whether this bias exists, but if it does, then I feel bad for NYU students looking for jobs outside of New York: Virginia is already consistently rated higher and Michigan is already consistently rated as high or higher. If mostly New York City lawyers are responding, they must not be NYU alumni.
