LSAC settles LSAT Disability Lawsuit Forum

[UnicornHunter]

As to testing accommodations fraud, I can't comment on SAT's standards for giving extra time and how they compare to the old LSAC standard. What I can say is that LSAC employs a large number of specialists who can tell whether somebody has faked a disability (for example, a student is only diagnosed with having a disability in the year they take the SAT). I feel that LSAC is unlikely to roll over and just hand out extra time like candy after this ruling. But, we will have to wait and see on this point.

Under its settlement with the government today, LSAT moves past Powell, lowering its standards for offering students up to double the time for the test. Now they will be granted the extra time if they got that accommodation on any other post-graduate test. Otherwise they must provide “documentation from a qualified medical professional and evidence of substantial limitation of a major life activity.”

Part of the issue is that LSAC just simultaneously gave up all this oversight you just described while at the same time removing the flags. Time isn't as much of an issue on the GMAT or GRE or MCAT. To the extent that the old system was overly restrictive and unfair to some people, I'm glad it's being changed. However, a lot of those protections were there for a reason. It seems like LSAC just gave in to pressure here, this does not appear to be a deliberate effort to craft a fair system of accommodations.

[ScottRiqui]

0.36 isn't a great correlation, but to say that it's worthless is a bit much. It's the single best predictor of success in law school.

Again, let's not go confusing "best we have" with "good". I'm sure there's also a "best" ice hockey team in Rwanda. With a correlation coefficient of ~0.35, your confidence intervals are either going to have to be very large, or your confidence levels will have to be very low. An LSAT score is simply not a very reliable predictor of 1L performance, despite being the best single predictor we have.

Don't know what you mean by this..

There's no multiplication going on. Or correction factors. It's literally doing the exact same thing as the regression of FYA on LSAT, just restricting the domain to the accommodated scores.

Sorry, I misunderstood your post - I thought you were talking about quantifying how the accommodations affected the score, then applying that correction factor to the accommodated scores so that they could be fairly compared to non-accommodated scores.

[cotiger]

0.36 isn't a great correlation, but to say that it's worthless is a bit much. It's the single best predictor of success in law school.

Again, let's not go confusing "best we have" with "good". I'm sure there's also a "best" ice hockey team in Rwanda. With a correlation coefficient of ~0.35, your confidence intervals are either going to have to be very large, or your confidence levels will have to be very low. An LSAT score is simply not a very reliable predictor of 1L performance, despite being the best single predictor we have.

Alright, granting this, what's your point? That's it's okay to grant a certain subgroup an X point advantage because it doesn't indicate much of a real-world difference in performance? It certainly indicates a large admissions difference in results.

I still don't see the argument for why we should prefer to use the "boosted" score as is instead of indicating to schools how that particular accommodated score tracks with law school performance (even if the difference in performance is slight), which is what the test is ostensibly meant to do.

eta: furq it is late. going to bed now.

[ScottRiqui]

If you're talking about "indicating a level of adjustment" to be applied to the accommodated score in order to hopefully turn it into something that's more predictive of law school performance, then you ARE talking about a correction factor, and any uncertainties/slop in your "level of adjustment" is just going to compound the poor predictive ability.

As an example, how are you going to know whether a tester used all of his extra time, or only a fraction of it? "De-boosting" his score by the same amount as someone who used all the extra time is as unfair as anything else in this thread.

I don't have a perfect solution - the best I've been able to come up with is re-writing the LSAT as an un-timed test, since I think the time crunch is an artificiality in the first place, and not essential to evaluating what the LSAT purports to evaluate.

If we start seeing a lot of people making questionable claims in order to get accommodations, that's one thing. But railing against the idea of accommodations themselves, or how they're viewed by adcomms (or if they know about them at all) seems a silly thing to wrap yourself around the axle about. You may as well complain that the handicapped parking space at the mall reduces your chances of finding a spot.

[cotiger]

If you're talking about "indicating a level of adjustment" to be applied to the accommodated score in order to hopefully turn it into something that's more predictive of law school performance, then you ARE talking about a correction factor, and any uncertainties/slop in your "level of adjustment" is just going to compound the poor predictive ability.

As an example, how are you going to know whether a tester used all of his extra time, or only a fraction of it? "De-boosting" his score by the same amount as someone who used all the extra time is as unfair as anything else in this thread.

I don't have a perfect solution - the best I've been able to come up with is re-writing the LSAT as an un-timed test, since I think the time crunch is an artificiality in the first place, and not essential to evaluating what the LSAT purports to evaluate.

If we start seeing a lot of people making questionable claims in order to get accommodations, that's one thing. But railing against the idea of accommodations themselves, or how they're viewed by adcomms (or if they know about them at all) seems a silly thing to wrap yourself around the axle about. You may as well complain that the handicapped parking space at the mall reduces your chances of finding a spot.

No, nothing extra would be applied to the score. There is no correction factor. No additional error is being introduced.

Here's an example. Say at Princeton Law the regression that LSAC calculates shows that a 170 LSAT predicts a 3.3 FYA. However, those who got extra time and score a 170 only average a 3.2. This is what LSAC found actually happens with the results of extended time tests (esp ADHD and learning disabilities). It's not accurate to say that those two scores indicate a similar level of ability.*

Do you recognize this as a problem? The whole reason for this lawsuit was that people with disabilities presumably are under predicted when forced to take without accommodation. The whole point is that the LSAT is supposed to represent an applicants abilities. If over estimation of abilities doesn't matter because LSAT is a rough tool, then that same argument could be used for under prediction. Understandably, people didn't accept that reasoning.

How to fix it:

• First, flag it and don't count the score towards medians. It's not the same test and doesn't predict the same thing as the standard test. Regardless of the influence of USNWR and medians, the reason schools require everyone to take to LSAT is because it provides the best (though certainly not great) estimation of how an applicant will fare in school.
• Second, indicate that that extra time 170 corresponds to a 3.2 FYA, which is what standard-time 167 LSAT students are predicted to score at that school. Also indicate the lower correlation with law school performance and thus the decreased weight the school should give to that score.

For the 15th time, I'm not railing against the idea of accommodations. I'm fine with a fairly low bar. What I'm railing against is unflagging extra time because extra time demonstrably gives an advantage beyond what is necessary to correct for the disability.

I agree that ultimately the best solution is to write a test that doesn't have these issues, but I'm assuming that that's not in the cards in the short run. We've got to figure out how to do the bet with what we've got, at least in the short run.

*To the extent that we define "ability" as meaning to well on law school exams. I understand that there's problems with that whole set up, but ultimately employers care more about how well you do on law school exams than anything else, so it is a very relevant measure.

[cotiger]

Moved from another thread:

But people in this and the other thread do seem awfully invested in the idea that the LSAT has some kind of significant meaning apart from its role in admissions.

How so in the other thread? The entire discussion is about its role in admissions, not about how much impact it actually has on grades. Honestly, that's not even really up for debate. It explains about 13% of grades for unaccommodated applicants.

If people itt are debating whether overstudying/retakers underperform what their top score predicts, the simple answer is yes. There are papers from LSAC that demonstrate this. Internet searches, people.

But part of the whole discussion in the other thread was that an accommodated LSAT doesn't accurately predict grades and therefore people who get accommodations and thus get into schools they wouldn't otherwise "deserve" to get into will do worse in law school and do worse on the bar exam. That presumes the LSAT means something about performance in school and on the bar exam. (And before you bring it up: yes, it has some predictive power. That works in the aggregate, not for any one individual person - we all know/know of plenty of people who had ordinary LSATs, killed it in LS, and transferred to schools they couldn't have got into when they were initially applying.)

I don't know what's controversial about saying what the data reveals: on average, a standard time 170 will do better in school than an extended time 170. The fact that there are other, unmeasurable factors that go into grades is irrelevant. That's what softs are for and why people with identical numbers can have vastly different admissions cycles.

Yeah, accommodated has even less predictive power. But I don't know why that means that it makes sense to represent that score as indicating a higher predicted aptitude than it actually does and on top of that represent that that test is more predictive than it actually is.

Part of whole argument for flagging is that we don't know wtf that score really means. The amount of grades that is explained drops from 13% unaccommodated to only 2% for those with extended time for ADHD. Why should we weight those equally, especially when the more random one tends to exaggerate the aptitude of the applicant?

[094320]

..

[cotiger]

Overprediction and underprediction aren't the same thing.

Just as we'd rather underconvict rather than overconvict criminally, there could be a decent argument for overpredicting rather than underpredicting, though I don't feel like making it.

I agree with this, which is part of the reason why I support a low bar for accommodations.

It doesn't affect the argument for whether it should be flagged or not, though.

[ScottRiqui]

Overprediction and underprediction aren't the same thing.

Just as we'd rather underconvict rather than overconvict criminally, there could be a decent argument for overpredicting rather than underpredicting, though I don't feel like making it.

I agree with this, which is part of the reason why I support a low bar for accommodations.

It doesn't affect the argument for whether it should be flagged or not, though.

I haven't drilled down into the actual text of the ADA, but found this in a paper about the ADA and education:

"Admissions committees may not hold standard tests taken with accommodations to be of lesser value than standardized tests taken under normal circumstances."

So that might be the answer right there - if schools aren't allowed to treat accommodated tests differently, then there's no good reason for LSAC to flag the scores, and discontinuing the practice of flagging the scores has the benefit of removing the temptation for adcomms to violate the ADA.

[cotiger]

Overprediction and underprediction aren't the same thing.

Just as we'd rather underconvict rather than overconvict criminally, there could be a decent argument for overpredicting rather than underpredicting, though I don't feel like making it.

I agree with this, which is part of the reason why I support a low bar for accommodations.

It doesn't affect the argument for whether it should be flagged or not, though.

I haven't drilled down into the actual text of the ADA, but found this in a paper about the ADA and education:

"Admissions committees may not hold standard tests taken with accommodations to be of lesser value than standardized tests taken under normal circumstances."

So that might be the answer right there - if schools aren't allowed to treat accommodated tests differently, then there's no good reason for LSAC to flag the scores, and discontinuing the practice of flagging the scores has the benefit of removing the temptation for adcomms to violate the ADA.

Good find. If that accurately summarizes the text of the ADA, then that pretty clearly makes flagging illegal.

Doesn't mean it's the best policy, though.

I guess the only real solution is to rewrite the test lol. Or I guess find some more accurate accommodations for those getting extra time. But I assume that if that was easy to do, LSAC would have already done it.

[094320]

..

[ScottRiqui]

Overprediction and underprediction aren't the same thing.

Just as we'd rather underconvict rather than overconvict criminally, there could be a decent argument for overpredicting rather than underpredicting, though I don't feel like making it.

I agree with this, which is part of the reason why I support a low bar for accommodations.

It doesn't affect the argument for whether it should be flagged or not, though.

I haven't drilled down into the actual text of the ADA, but found this in a paper about the ADA and education:

"Admissions committees may not hold standard tests taken with accommodations to be of lesser value than standardized tests taken under normal circumstances."

So that might be the answer right there - if schools aren't allowed to treat accommodated tests differently, then there's no good reason for LSAC to flag the scores, and discontinuing the practice of flagging the scores has the benefit of removing the temptation for adcomms to violate the ADA.

Good find. If that accurately summarizes the text of the ADA, then that pretty clearly makes flagging illegal.

Doesn't mean it's the best policy, though.

I'm also not sure if that prohibition would keep the ABA from leaving out accommodated scores when figuring a school's medians, since the ABA isn't involved in determining who gets accepted or rejected. So there may still be a need for flagging, but it would best be implemented in such a way that the schools don't know whose scores were flagged until shortly before they have to submit their 509 forms to the ABA, long after admissions decisions have been made.

[cotiger]

Also when it comes to any argument re: the LSAT, I'll say this.

I read research in undergrad looking into the grad school admissions process, which claims to be fairly holistic (with in-person interviews and everything). The general finding is that usually the claim of holistic is a hopeful & well-meaning lie, not because the committee itself doesn't want to be holistic (or doesn't imagine themselves to be), but just because there'a an inherent cognitive bias to anchor to quantitative values like test scores & GPA and to value those a lot higher than they realize. The committee gave their relative % importance to the different factors (GRE, GPA, research, recs, etc.), but then the researchers ran the actual data against different models of weight to different factors, and found that yeah, the committee was unaware of what they were doing.

So that sort of scarily means that law school admissions committees might actually genuinely believe their claims of holistic admission (even as LSN seems to suggest that's mostly not that true) and not even fully be aware of how much they're overvaluing the LSAT. This also makes me wary that they don't penalize flagged people even if they don't think they do. Though I'm not sure I support the current change, just something to think about.

This is why in cotiger's world, adcomms would be given the info as to what the equivalent LSAT is (and also be told that it has virtually no explanatory power, even when compared to the already low explanatory power of unaccommodated tests).

But it seems like not-flagging is mandated, so this discussion is all pretty moot.

[NYSprague]

In other words, the majority of ADHD cases are diagnosed by pediatricians and primary care physicians with no training and no stringent standard for making the diagnoses. Even when they do follow the standard, "teacher and parent rating scales" are a large part of their system. If ever there was something ripe for gaming, this is it.

Then get mad over LSAC's failure to implement safeguards against bullshit diagnoses. Don't get mad at a policy meant to prevent schools from unfairly discriminating against students by highlighting the students who accepted an accommodation.

This is what bugs me about this whole issue. None of you were complaining when LSAC's policies hurt people who weren't you. You're only getting bent out of shape now that the policy has the potential to slightly harm your chances at one single tier-layer of schools.

Me too.

Also, in New York extensive testing is required for a school to grant accommodations and a plan has to be approved by the school. You can't just get a prescription.

[LSAT Hacks (Graeme)]

Students with wealthy families tend to have more free time to focus on studying, pay for LSAT classes, and buy more study supplements. Why is this not a concern here? Wealthy parents don't just aid in "bribing" a specialist to diagnose a student with a disability (btw, the credentials of the evaluating specialist is taken into consideration when determining to give out accommodations). Should LSAC deduct a certain number of points off of a test taker's score if s/he comes from a wealthy household?

What of the ability to take the LSAT multiple times? Is it valid that somebody with 156, 169, and 175 gets to use only their 175 score? Is that really more accurate of their natural ability? I would say no. So why is the validity of the test so important to protect when it comes to accommodations? Why not insist on only using a test taker's first score?

As to testing accommodations fraud, I can't comment on SAT's standards for giving extra time and how they compare to the old LSAC standard. What I can say is that LSAC employs a large number of specialists who can tell whether somebody has faked a disability (for example, a student is only diagnosed with having a disability in the year they take the SAT). I feel that LSAC is unlikely to roll over and just hand out extra time like candy after this ruling. But, we will have to wait and see on this point.

To a great extent, this thread is filled with posters with double standards, ignorance, and (sometimes) prejudice against the mentally handicapped. Pretty disappointing.

I addressed the wealth argument earlier. Wealthy students have a leg up, currently. But only to some extent. Currently, wealthy students hit the same wall as everyone else does, unless they shift their internal thought processes. You can't shift your thought processes by paying someone to do it for you. So to a remarkable extent, the LSAT has been immune to money.

I should know. I teach the damn thing. And if someone comes to me with stacks of bills but the wrong attitude, there is NOTHING I can do for them. Whereas, on the SAT, if someone comes to me with money and time, I can boost their score by hundreds of points.

To me this comes down to the question of safeguards. If the LSAC is allowed to weed out fraud, then there shouldn't be much of an issue. I agree that there are many people out there with disabilities that require accommodation and they've been unfairly denied it until now. I've heard many horror stories.

My concern is that the agreement seems to go too far in the other direction. It mandates that LSAC automatically accept accommodations given on any other test. Prima facie this creates a gigantic loophole.

We'll see how it works out in practice.

[ScottRiqui]

But it seems like not-flagging is mandated, so this discussion is all pretty moot.

Looking around online, I was surprised to find out that prior to this settlement, the LSAT and the MCAT were the last standardized tests used for school admissions that flagged accommodated scores. All the other major ones like the GRE/SAT/ACT gave up the practice over a decade ago.

[mornincounselor]

Post removed.

[ScottRiqui]

Couldn't they just throw another question on the scantron right after the ethnicity stuff.

Did you utilize any accommodations while taking this test?

(Yes -- Explain)
No
Rather Not Answer

That way they have access to the information, and if people lie they can be subject to penalties similiar to those which may apply if a person lies about beign URM.

Who's the information for? LSAC already knows whether you're taking an accommodated test, and the schools' adcomms aren't allowed to use the fact in their admissions decisions, so why do they even need it? What you're describing is essentially equivalent to the system that's about to be discontinued, where LSAC tells the schools directly who took the test with accommodations, except that under the current system the taker doesn't have the opportunity to lie about it.

[NYSprague]

Did you guys read the consent decree? It is much more detailed and adds many more requirements than I thought. They have to create a panel, hire more experts, be more fair in requesting information and submit to independent monitoring.

This seems to completely revamp their previous system.

[ScottRiqui]

Did you guys read the consent decree? It is much more detailed and adds many more requirements than I thought. They have to create a panel, hire more experts, be more fair in requesting information and submit to independent monitoring.

Reading it now - here's a link (LinkRemoved) for anyone else who wants it.

[NYSprague]

Couldn't they just throw another question on the scantron right after the ethnicity stuff.

Did you utilize any accommodations while taking this test?

(Yes -- Explain)
No
Rather Not Answer

That way they have access to the information, and if people lie they can be subject to penalties similiar to those which may apply if a person lies about beign URM.

Who's the information for? LSAC already knows whether you're taking an accommodated test, and the schools' adcomms aren't allowed to use the fact in their admissions decisions, so why do they even need it? What you're describing is essentially equivalent to the system that's about to be discontinued, where LSAC tells the schools directly who took the test with accommodations, except that under the current system the taker doesn't have the opportunity to lie about it.

All forms of annotating are prohibited.

[NYSprague]

Did you guys read the consent decree? It is much more detailed and adds many more requirements than I thought. They have to create a panel, hire more experts, be more fair in requesting information and submit to independent monitoring.

Reading it now - here's a link (LinkRemoved) for anyone else who wants it.

Does make it seem that LSAC was doing a horrible job in complying with the law.

[ScottRiqui]

Couldn't they just throw another question on the scantron right after the ethnicity stuff.

Did you utilize any accommodations while taking this test?

(Yes -- Explain)
No
Rather Not Answer

That way they have access to the information, and if people lie they can be subject to penalties similiar to those which may apply if a person lies about beign URM.

Who's the information for? LSAC already knows whether you're taking an accommodated test, and the schools' adcomms aren't allowed to use the fact in their admissions decisions, so why do they even need it? What you're describing is essentially equivalent to the system that's about to be discontinued, where LSAC tells the schools directly who took the test with accommodations, except that under the current system the taker doesn't have the opportunity to lie about it.

All forms of annotating are prohibited.

And that, too.

[Mosie]

I write this as a disability consultant for several testing organization, but NOT the LSAC. However, procedures are quite similar across these groups and I know several of the LSAC consultants.

Here's the problem: When we are talking about certain disabling conditions (blind, motor impairment, etc.) it is quite easy to discern whether an individual is disabled, and these requests are typically approved without issue. I strongly doubt that these individuals were involved in the suit; they received what they needed. The problem lies in the LD and ADHD cases, which account for a large majority of all requests. While it is true that many of these applicants have a history of past accommodations, often these accommodations were provided either in the absence of any evaluation (more commonly the case in private schools) or subsequent to an evaluation that was either equivocal in its findings or done inadequately or incompetently. This is especially true with ADHD-- I have seen a number of people whose 'documentation' accompanying their request was nothing more than a prescription pad on which the doctor has written "Mr. X has ADHD based on my evaluation on 3/13/2010". Within my profession this is very bad practice, yet many students have received accommodations from ACT, SAT, etc., based on such incomplete information. And now DOJ has said that this is entirely sufficient, and that there shouldn't be any further consideration of the merits of the request. In other words, rubber stamp the request as "Approved".

In my opinion this will further open the floodgates for unethical applicants (and health care providers) to buy and sell diagnoses. It's being done already, but this agreement between LSAC and DOJ makes it an even more obvious approach if one wants extra time.

Virtually everyone- disabled or not- taking the LSAC benefits from having extra time. Lots of data have shown this to be true. While the DOJ is acting on behalf of disabled people who feel they are being discriminated against, this agreement creates a new aggrieved class-- those who are not disabled and do not seek any accommodations, because they are now at a disadvantage due to all of the people who will be receiving an edge by receiving unwarranted and unneeded accommodations. I really think that the next step will be for a group of such nondisabled individuals to file their own class action suit claiming that their chance at a legal education is being adversely impacted by nondisabled people gaming the system (with DOJ's approval).

[NYSprague]

I write this as a disability consultant for several testing organization, but NOT the LSAC. However, procedures are quite similar across these groups and I know several of the LSAC consultants.

Here's the problem: When we are talking about certain disabling conditions (blind, motor impairment, etc.) it is quite easy to discern whether an individual is disabled, and these requests are typically approved without issue. I strongly doubt that these individuals were involved in the suit; they received what they needed. The problem lies in the LD and ADHD cases, which account for a large majority of all requests. While it is true that many of these applicants have a history of past accommodations, often these accommodations were provided either in the absence of any evaluation (more commonly the case in private schools) or subsequent to an evaluation that was either equivocal in its findings or done inadequately or incompetently. This is especially true with ADHD-- I have seen a number of people whose 'documentation' accompanying their request was nothing more than a prescription pad on which the doctor has written "Mr. X has ADHD based on my evaluation on 3/13/2010". Within my profession this is very bad practice, yet many students have received accommodations from ACT, SAT, etc., based on such incomplete information. And now DOJ has said that this is entirely sufficient, and that there shouldn't be any further consideration of the merits of the request. In other words, rubber stamp the request as "Approved".

In my opinion this will further open the floodgates for unethical applicants (and health care providers) to buy and sell diagnoses. It's being done already, but this agreement between LSAC and DOJ makes it an even more obvious approach if one wants extra time.

Virtually everyone- disabled or not- taking the LSAC benefits from having extra time. Lots of data have shown this to be true. While the DOJ is acting on behalf of disabled people who feel they are being discriminated against, this agreement creates a new aggrieved class-- those who are not disabled and do not seek any accommodations, because they are now at a disadvantage due to all of the people who will be receiving an edge by receiving unwarranted and unneeded accommodations. I really think that the next step will be for a group of such nondisabled individuals to file their own class action suit claiming that their chance at a legal education is being adversely impacted by nondisabled people gaming the system (with DOJ's approval).

The consent agreement makes it obvious that valid claims were denied.