soj wrote:True or false?
1. The LSAT is "curved" in the sense that on harder tests, you can get more questions wrong and still get the same scaled score.
True.
LSAC goes through a bunch of steps to make sure you're not penalized for getting a tougher test or rewarded for getting an easier test. First, LSAC pretests questions using the experimental sections. Ambiguous or poorly worded questions are thrown out. Questions that seem to reward poor reasoning (i.e. questions that low scorers tend to get right more often than high scorers do) are thrown out. Then they combine the remaining questions so that the test has a good mix of hard, medium, and easy questions. But some tests inevitably end up being tougher than others, and so LSAC comes up with a conversion method that takes difficulty into account. If a test has too many easy questions, then it takes more credited responses to get a 140 on that test than on a test with fewer easy questions. If a test has too many hard questions, then it doesn't take as many credited responses to get a 170 on that test than on a test with fewer hard questions.
2. The LSAT is "curved" in the sense that every test is meant to have the same distribution of scores. In other words, if all the smart people took the June test, then the June test will have a tougher curve, and it'll be harder to score higher in June than in other administrations.
False.
The LSAT standardizes scaled scores, not percentile scores. In theory, a person will always get the same score that reflects that person's reasoning abilities. If it weren't for outside factors such as test-day condition, relative strength in a certain question type over others, and random luck, someone with a "170" reasoning ability will get a 170 no matter when that person takes the LSAT. Your score has nothing to do where you stand relative to others taking the same test. You're not competing against others sitting in your testing room. If all the smart people take the test in June, it may be more difficult to get a certain percentile score in June, but no more difficult to get a certain scaled score. If test-takers are getting smarter or more prepared, then more people will achieve high scores. But from the perspective of a single test-taker, no single scaled score will be harder to obtain. The fact that people are better prepared for the LSAT these days is reflected in the fact that a 170 used to be a 98th percentile score but is now a 97th percentile score. But people who scored a 170 in 2011 are on average no better or worse reasoners than people who scored a 170 in 2001.
3. Whether a question is labelled easy, medium, or difficult depends on how well previous test-takers did when that question was used in an experimental section, so the fact that people are getting smarter and more prepared is setting the bar higher.
False.
Everything before "so the fact that" is true. Yes, how well previous test-takers did on individual questions in experimental sections partially determines whether LSAC considers a question easy or difficult, but keep in mind that LSAC has been equating the exams precisely to deal with this problem. Conversion scales are set so that a given scaled score on any exam is equivalent to the same scaled score in the previous generation of exams, which is equivalent to the same score in the generation of exams before them, and so on. As a result, a given scaled score on any exam is equivalent to the same scaled score on all the other exams. If there are more high scorers in a certain administration, that just means more high scorers to use as data points when determining the scale for the next administrations. It does not "raise the standard."
I may be failing to comprehend something here but please, humor me: I see your point that the LSAT is
designed to equate in a way that allows for a change in the preparedness of LSAT takers as a group while maintaining consistency in what level of reasoning ability earns what scaled score but I fail to see how this is possible.
I don't doubt that they've attempted this, but if my understanding of how the tests are created and calibrated is correct then I don't see how the LSAC could accomplish that.
The LSAC's source of feedback on question difficulty is test takers, as a group. So, if test takers as a group have gradually gotten better prepared but have stayed the same in reasoning ability (which is what we all suspect?) then test takers as a group have, slowly and gradually, made it appear that questions which formerly required X level of reasoning ability are marginally less difficult, leading test makers to include more difficult questions (unwittingly).
Essentially, what I don't understand, is how test makers could separate reasoning ability from preparedness when the changes are gradual and pervasive. Where does the independent verification of question difficulty occur that allows test makers to be certain that they aren't gradually increasing difficulty (inadvertently) as test takers get better prepared?
Really not trying to be dense, just curious.
