hoos89 wrote:That really doesn't explain why there are people who do consistently well to excellent in classes with several different professors. I guess your theory is that they are somehow experts at figuring out which arguments the professor wants, but I would argue that they are just better at putting forth arguments in general. Also I think you might be overestimating how many issues most exams pick up on; there really is a pretty sizeable difference in terms of raw points (in most classes) between the CALI(s) and the median.
I think most of that difference is driven not by issue spotting, but by "analysis" points, which are driven by typing speed/word count, etc., as well as the subjective criteria I was talking about above.
I'm kind of surprised that this is all that controversial. My career before law school was developing reliable and valid talent and skill assessment tests. If I would have come up with something as crude, unreliable, and invalid as a law exam, I would have been fired instantly. I'm not the only one that has brought this up.http://www.nesl.edu/userfiles/file/lawr ... /crane.PDF
In 1976, the Law School Admission Council published the results of a study by Stephen P. Klein and Frederick M. Hart supporting the idea that factors other than substantive knowledge affect essay grades. One factor that correlated highly with success on law school essay examinations was legible handwriting. Another leading indicator of higher grades was length. Longer answers were viewed by law professors as better.
Law schools have an obligation to use the most accurate and internally consistent, or reliable examination methods. The essay exam is inherently capricious not only because of the number of subjective factors used in scoring that influence the student's overall grade; but also because they compare law students based on too few samples of each student's knowledge of a given domain of material to be reliable or statistically valid.
The traditional law school essay exam is mathematically unsound and unable to consistently measure the law student's proficiencies within the law school's curriculum. This is due to an inability to either accurately sample the same amount of material or to render the same number of samples of a given domain of material as an objective exam can within a comparable time period. Therefore, single-shot essay exams used to measure numerous domains of information within each larger law school subject are notoriously subjective and unreliable. Accordingly, they are also invalid for their intended purpose. This is especially true given the enormous importance placed on the results of law school essay examinations and because those results are used to compare students' performances.
The essay exam format is inherently incapable of affording law students an adequate opportunity to demonstrate proficiency in an entire subject. It is infeasible for the professor to draft an essay exam that is capable of sampling a sufficient quantity of information from various the domains of a complex subject. If the professor were to draft successfully an essay examination that was lengthy enough to contain enough questions for the examination to be considered valid, it would be impossible for the student to actually complete the examination within normal time constraints; and various physical and psychological phenomenon would hinder the students ability to perform well during the course of completing such an arduous task. Critics of essay examinations doubt that their unreliability can be lowered to a level that makes them valid.
I can't seem to find a copy of the actual Klein and Hart study anywhere online, but their analysis was done on issue-spotter essay exams that were essentially the same as the exams we see today in law school. Obviously, handwriting as a confounding variable has been eliminated due to SoftTest--but this has probably only exacerbated a different confounding variable: word count (which is largely a function of typing speed).
I'm not going to hijack this thread any further, and I've made my points clear by now. I didn't even mean to respark this contentious debate. I just though I made a fairly obvious and innocuous comment that's pretty much in line with the conventional TLS wisdom. I just don't like my view on this matter dismissed as "sour grapes." That's an ad hom attack lacking any logos-based value. I ended up getting the job I always wanted out of law school anyway, so I have no reason to be bitter. I like to speak up on this issue because it's an area where I have a lot of particular expertise and work experience--and if an 0L asks for honest advice regarding the realities of going to law school, this is the most honest advice I can think of.
The flip side of the "sour grapes" argument is that there's no doubt an inclination for people to defend a system they are so deeply invested in, and there's doubly an incentive to defend that system if it tells you you are much better than others--despite objective evidence showing that system isn't worth much in the empirical sense. It's possible that I am biased myself, but everything I've read suggests that I'm not far off.