Smaller Curves & Their Effects on the LSAT Forum

[mst]

TL;DR Version: LSAT SCORE SYSTEM BLOWS. LETS CHANGE IT?!

LSAT RC Version using 8th grade grammar: As I sat outside my test room this week, in between Sections 3 and 4, my brain ran over a certain question I know I had incorrectly bubbled, trying to reinforce the answer/number combination to memory so that if I wanted to, I could fix it in section 4. Now, I know that bubbling prior sections is against the rules, and I know that it's a lot to risk, if it means getting caught by the proctor and basically being banished from law school. But I also realized something at that moment: that one question, at my performance level, would undoubtedly make the difference between my score being a 1** and a 1**+1.

All too often, people around the country realize this when considering whether to make slight changes to their scantrons, and the vast majority of them, from my experience, are willing to go the extra step for the extra point: whether that means keeping the pencil up for a few extra seconds to finish your section, filling in missed bubbles, or correcting missed answers outside of the alloted time.

Was it always like this? To some extent, sure. But taking a look at the decreasing curve amounts over the last 20 years, the importance in the upper echelons of scoring (164+) of individual points has become vastly more important in achieving certain scores. Coupled with this has been the increased importance of LSAT scores in law school admissions, and thus the narrowed ranges acceptable at the nations top programs.

For example, as demonstrated by LSN and other sites, the difference between a 169 and 170 will in many cases make a noticeable difference in the strength of your application at certain schools. And when that difference in points between a 169 and a 170 is defined by a single question, slight-cheating can become almost too temptatious to avoid.

While this obviously points to growing issues in the realm of cheating, I believe it raises more points about the structure of the LSAT scoring system and it's place in law school admissions. Generally speaking, the LSAT does represent the only real system for objectively measuring candidates against each other. But the scores that the test is producing no longer represent the distinct difference in performance that they did 10 to 20 years ago. Whereas 20 years ago the difference between two certain scores could be a couple questions, those same two scores could today could be separated by as little as 1 question.

Schools have virtually claimed certain and distinct LSAT scores as the brackets for admitted students, and performance at certain levels is now key. Whether or not it is the intention of the schools, the practical effect this has had is the following on students taking the LSAT: "One question will get me into a much better law school. If one question means I will get in as opposed to won't get in, that must mean they think that question represents the needed level of ability to get into their school." I doubt that any school really believes that their is a tangible difference in quality between somebody who got a 173 and a 174, but all evidence points to the fact that this is a critically important difference in whether or not to admit a student to certain schools.

It seems LSAT has recognized this trend and has tried to put more focus on "score ranges" when reporting scores to schools. For example, they emphasize that a 167 fits in the 166-168 range and that a 167 is not some distinct level of performance. However, as schools compete desperately for LSAT based rankings, those ranges importance seems to all but disappear.

So what's the solution here?

Do you guys believe that the time has come for a major overhaul in the way that the LSAT is scored, or at least a change in the way that the raw scores are translated to comparable scores/percentiles?

Would you agree more or less with a system that perhaps didn't distinguish between 170-175, 176-180, etc., (although this would still leave borderline areas where students would be tempted to cheat or schools would be tempted to advance into...)?

If you could propose a new system to fix this issue, what would it be? If you think that the current system is better than any alternative, why?

[Bildungsroman]

tl;dr

ITT: cheater attempts to justify cheating; fails.

[mst]

Big assumption there, but go ahead and make it? All I claimed to do was spend my precious 10 minutes of LSAT break time comprehending the risk of cheating...not a crime.

[voice of reason]

The question really is what is the standard error of the estimate. This is an arcane statistical point but it's what the issue comes down to. It is not whether the difference between a 173 and a 174 is 1 question or 2. It is whether the test produces precise measurements of the underlying ability trait, where precision is indicated by a narrow score band and imprecision is indicated by a wide one.

The problems with collapsing scores into 170-175 and 176-180 are that this would elide real differences between test-takers who would end up with the same score and that the same problem you identify would exist at the margin between the two scores. Combining scores in those ranges would make things worse.

Scores are subject to error in a way very much analogous to how surveys are subject to sampling error. The solution to this problem, all else being equal, is a longer test. Unless you can write better questions, the only way to get more precise measurement is to ask more questions. Would you rather sit for the LSAT in 3 sessions of 6 hours each spread over 3 weeks, and pay 3 times as much for it? That would be the direction to go in order to get scores where a 1 point difference is meaningful and where there is no meaningful difference in ability between people with the same scores. I don't think we want to put up with that. And I don't think schools would benefit much from it because the power of the LSAT to identify more desirable students will never be spectacular.

[romothesavior]

tl;dr

tb;dr

(tb = too boring)

[Ragged]

tl;dr, but at first I thought the thread was about the female form.

I'll read it eventually when I'm bored.

[Renzo]

tl;dr, but at first I thought the thread was about the female form.

I'll read it eventually when I'm bored.

Yep, except I won't read it. But smaller is always better, as long as we're still talking physique.

[Ragged]

tl;dr, but at first I thought the thread was about the female form.

I'll read it eventually when I'm bored.

Yep, except I won't read it. But smaller is always better, as long as we're still talking physique.

I will go ahead and disagree with you there in the words of an epic song: I like big butts and I can not lie....

[mst]

Voice, you make a good point concerning a more realistic approach to start to distinguish the quality of students, in a more accurate manner. But you acknowledge it's impractical.

I previously acknowledged that the core problem I initially pointed out would still exist if the LSAT reported broader score ranges to schools rather than scores. However, the frequency of a tester approaching the break in a score range would be less frequent, and one could assume that the student would be less enticed to cheat, and that the schools wouldn't be forced to differentiate between candidates that in all likelihood demonstrate equal ability despite a difference of 1-2 questions on the test.

With this method, the test structure wouldn't have to change, the scores from candidates who had taken the test previously would still be applicable, and law schools would feel less pressure to respond to what most could agree are unimportant LSAT differences.

[Tautology]

Schools know that the difference between a 170 and a 171 is small, but that doesn't mean it's not meaningful.

The problem is that you are looking at this from the perspective of the test taker, which, understandable as it is, is not the perspective of the schools when they're choosing who to accept. From the school's point of view, there is a population of students with 170s and a population of students with 171s. Within each population there will be some who got lucky on some questions and overachieved, some who got unlucky on a question or two and underachieved, and some who are right where they're supposed to be. On average though, the population with 171 has higher achieving students. From the school's point of view, on average you're going to get better students taking the 171s than taking the 170s. That difference is small, but small distinctions are still distinctions. The best schools get way more applicants than they can accept, and they get to make small distinctions among them when deciding who they want.

[World B. Free]

Schools know that the difference between a 170 and a 171 is small, but that doesn't mean it's not meaningful.

+1

[mst]

I'm not suggesting it's not meaningful. I believe it does have meaning to some extent. But I don't think anybody believes it has such meaning to the extent that schools will rarely accept students outside of a 3 point range, or that the cutoff line on LSATs for accepted students (non URM) is so noticeable and apparent, for the sole sake that those score differences are so vitally important. If schools really believe that 172/171 difference is that meaningful, that is sad. We all know the effect of a missed bubble, or a broken pencil. Or a sneeze that stops you from finishing that last question. If schools are at the point where they truly believe they must downgrade entire applicant pools of students with scores one point lower because the performance is going to be on average worse, that is sad.

However, there's plenty of reasons to believe this is not the case. Much evidence points to the fact that schools are generally more holistic in their reviews than that. They take into account a myriad of factors for decisions. Yet you claim that they willingly choose to have such stringent LSAT points solely for the case of their ability to predict future performance?

I don't think that's the case. And from most news articles I've read, and LSAC research papers, either do many admission committees, law professors, deans, or students. All the above had one thing in common: they believe that LSAT scores play too much of a role in decisions because schools are motivated to raise their averages in order to maintain or improve their national ranking.

[Chimica]

I think that while it is possible to bubble in sections you aren't actually doing, it just doesn't help you.

You need an amazing memory; it distracts you from the section at hand and most people would do better with full focused attention on the 35 minutes they have on the section they can read directly; it gets you one point...maybe. How much is lost from loss of focus?

So, I don't think that LSAC forces this and I don't think that the cheating actually helps.

(I may be biased. I've taught and most cheaters are stupid, lazy and eventually get caught or fail before they are officially caught-granted, we may have missed the really smart ones, but it was probabely easier to just learn the damn material )

[malfurion]

--ImageRemoved--

[malfurion]

I don't think the cheating idea is a valid premise for your argument. People who are willing to cheat are going to continue to do so, even if the scoring system changes. These are the same people that have been cheating on stuff since 3rd grade. I remember reading about studies that showed that people would continue to cheat on tests even when they were told ahead of time that their results on the test did not matter. I think it's incredibly rare that someone who has never cheated before in their lives would decide to cheat for the first time due to the LSAT scoring system.

That said, I do agree that it's ridiculous how a one-point difference in scores can have such an impact on admissions, given that the stated margin of error for the test is greater than one point (you could make a good LR flaw question out of that, actually). In my opinion, the absurd dependence on the school rankings system is more to blame for this than the actual LSAT scoring system, though.

[balzern]

I don't think the cheating idea is a valid premise for your argument. People who are willing to cheat are going to continue to do so, even if the scoring system changes. These are the same people that have been cheating on stuff since 3rd grade. I remember reading about studies that showed that people would continue to cheat on tests even when they were told ahead of time that their results on the test did not matter. I think it's incredibly rare that someone who has never cheated before in their lives would decide to cheat for the first time due to the LSAT scoring system.

That said, I do agree that it's ridiculous how a one-point difference in scores can have such an impact on admissions, given that the stated margin of error for the test is greater than one point (you could make a good LR flaw question out of that, actually). In my opinion, the absurd dependence on the school rankings system is more to blame for this than the actual LSAT scoring system, though.

+1000000000000000

[pinkzeppelin]

*Effects

[suspicious android]

Here's my solution: let people look ahead or back in their booklet, let them bubble in answers from previous sections. Start up section one, have the proctor say "you have one hour, forty-five minutes to complete the next three sections." After that, they collect your booklets and before you start section 4 & 5, you get a new booklet with only two sections. You have 70 minutes to complete that section. This would reward people who are perfect in one section and have the time management skills to make sure they get through the whole thing, which seems like it'd be a valuable skill to test. They might need to make some material a bit harder because I think this would intend to increase the scores for people who are already in the 170's, but that shouldn't be too hard to adjust.

This is a really self-serving suggestion, since I finished my two LR sections with about 5 minutes left and spent that time furious that I couldn't look back at my LG section. But I think it really could mitigate the effect of one crazy question/LG costing someone 5-6 points, which doesn't seem to me to be a sensible way to measure people's ability. Also, for people who have already hit the ceiling in one section (that is, they can almost always get -0 in significantly less than 35 minutes), the LSAT isn't measuring that increased ability compared to someone who just barely gets all the questions right in 35 minutes. This change could address that.

[schnoodle]

sweet jesus, it's EFFECTS, NOT AFFECTS. further, re-read your OP. is "tempestuous" what you were looking for there? i thought school place a lot of value on good writing. go to your local community college and take a couple english courses before thinking about retaking the lsat/applying to law school.

next time you think a word might be right and might make you sound smarter to a bunch of anonymous strangers, do us all a favor and put it away. go with a word you know.

[3|ink]

TS; AR

[nireca]

sweet jesus, it's EFFECTS, NOT AFFECTS. further, re-read your OP. is "tempestuous" what you were looking for there? i thought school place a lot of value on good writing. go to your local community college and take a couple english courses before thinking about retaking the lsat/applying to law school.

next time you think a word might be right and might make you sound smarter to a bunch of anonymous strangers, do us all a favor and put it away. go with a word you know.

woah, this dude is clearly a grammar bad ass, that or just a pedantic douche.

[mst]

Was going for temptatious, but it got spell corrected...sorry broha. And honestly, I still can't get the effects/affects thing straight, I always gloss over that mistake, but thank you for pointing that out. I will mosey on down to da local commoontitty college right away sir' so I can learns me some proppa's grammEr for the spider-webs! Afta's alls, we all know the schnoodle be gittin upset every time he see someones use a words wrong! Best not let him nears that 4th LSAT Logic game thread!!!

[sumus romani]

Just a small point, but sometimes one answer difference can be two actual point differences. Someimes in the mid 170's, there is a gap, which no one can score (so, just as an illustration, -8 is a 174 and -7 is a 176). A free sample LSAT from LSAC has this feature, as does the Oct 09 test, believe (and some other tests too).

[citrustang]

Here's my solution: let people look ahead or back in their booklet, let them bubble in answers from previous sections. Start up section one, have the proctor say "you have one hour, forty-five minutes to complete the next three sections." After that, they collect your booklets and before you start section 4 & 5, you get a new booklet with only two sections. You have 70 minutes to complete that section. This would reward people who are perfect in one section and have the time management skills to make sure they get through the whole thing, which seems like it'd be a valuable skill to test. They might need to make some material a bit harder because I think this would intend to increase the scores for people who are already in the 170's, but that shouldn't be too hard to adjust.

This is a really self-serving suggestion, since I finished my two LR sections with about 5 minutes left and spent that time furious that I couldn't look back at my LG section. But I think it really could mitigate the effect of one crazy question/LG costing someone 5-6 points, which doesn't seem to me to be a sensible way to measure people's ability. Also, for people who have already hit the ceiling in one section (that is, they can almost always get -0 in significantly less than 35 minutes), the LSAT isn't measuring that increased ability compared to someone who just barely gets all the questions right in 35 minutes. This change could address that.

This is a really interesting idea.

[mst]

Here's my solution: let people look ahead or back in their booklet, let them bubble in answers from previous sections. Start up section one, have the proctor say "you have one hour, forty-five minutes to complete the next three sections." After that, they collect your booklets and before you start section 4 & 5, you get a new booklet with only two sections. You have 70 minutes to complete that section. This would reward people who are perfect in one section and have the time management skills to make sure they get through the whole thing, which seems like it'd be a valuable skill to test. They might need to make some material a bit harder because I think this would intend to increase the scores for people who are already in the 170's, but that shouldn't be too hard to adjust.

This is a really self-serving suggestion, since I finished my two LR sections with about 5 minutes left and spent that time furious that I couldn't look back at my LG section. But I think it really could mitigate the effect of one crazy question/LG costing someone 5-6 points, which doesn't seem to me to be a sensible way to measure people's ability. Also, for people who have already hit the ceiling in one section (that is, they can almost always get -0 in significantly less than 35 minutes), the LSAT isn't measuring that increased ability compared to someone who just barely gets all the questions right in 35 minutes. This change could address that.

This is a really interesting idea.

It is a very unique idea, and it has a lot of merit. Plenty of cons though: People don't have the same 3 sections first, so those who have certain forms will be at a distinct advantage depending on their circumstances. I think that this could be avoided in some regards if they ensured everyone had the same first 3 sections, just in different order.

At the very least, this clears up peoples ability to cheat in many circumstances, and even clears up much of their motivation. The problem with score differentials still be rather meaningless yet critically important would still exist.