Isn't the curve determined by the experimental? Forum

[ajmanyjah]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

[Patriot1208]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

Yes, but I think people are extrapolating on how they did and how those in last june and february must have done.

[Remnantofisrael]

This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.

This is a quote from LSAC's Associate Director of Psychometric Research, Lynda Reese in Feb of 2010 I believe. I stole it from Steve Schwartz's blog, LSAT Blog (which I would link but I think I'm not allowed to).

THAT SAID, I thought common belief is that no one really knows exactly how it is formed. BUT that the "scale" is determined before the test is even given by combination of factors including the experimental. THEN, based on statistics of the actual test, things CAN be adjusted SLIGHTLY or questions thrown out.

The idea, though, is that you are talking about a test put together by a bunch of PhD's who understand this stuff better than almost anyone else on the planet and that they know full well what to expect in any given group from HUGE statistical analysis. Hundreds of thousands of test data, and tens of thousands applying directly to the questions on this particular test.

So what does all this mean? That if the test was hard and people screwed the pooch, the test makers probably knew this already. If they didn't, then the test maybe wasn't really any harder than usual tests of the past (in, say, the -8, -9 zone) and that, instead, there were 4-5 REALLY tough ?'s and a whole bunch of easy ones balancing it out, which would make the TLS community assume that this was a particularly hard test when perhaps it wasn't. Or maybe the test makers thought this was a -9 test, the data says it was more like a -11, and in the end we get a -10.

Or maybe there is a room full of monkeys, all given laxatives and shit-ton of bananas. When the last monkey shits, they check to see how much time passed and determine the scale based on that. That is what I am going with.

[Sandro]

So what you're saying is a -14 curve? wooooo

[d34d9823]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

[Remnantofisrael]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

Kinda.

Maybe.

I think.

./ \
/...\
..|
..|
..|
There- fixed that for ya.

[d34d9823]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

Kinda.

Maybe.

I think.

./ \
/...\
..|
..|
..|
There- fixed that for ya.

This has been stated before. If OP wanted a boring "nobody knows for sure", she wouldn't have asked.

[Remnantofisrael]

Maybe I'm on LR mode and need to shut down, but if the answer is "no one knows" then the answer isn't "yes" unless the question as to how the curve/scale is determined is:

"Is it true that no one knows how the scale is determined, other than LSAC and anyone else with that privileged knowledge that won't be sharing?"

Then yes is correct.

Also, I'm just having fun at your expense. I'm sure if you took a minute to look at my posts in just the last 30 minutes there are plenty of grammar and spelling errors. Grammer. Grammar. That word just doesn't look right with an a near the end.

[d34d9823]

Maybe I'm on LR mode and need to shut down, but if the answer is "no one knows" then the answer isn't "yes" unless the question as to how the curve/scale is determined is:

"Is it true that no one knows how the scale is determined, other than LSAC and anyone else with that privileged knowledge that won't be sharing?"

Then yes is correct.

Also, I'm just having fun at your expense. I'm sure if you took a minute to look at my posts in just the last 30 minutes there are plenty of grammar and spelling errors. Grammer. Grammar. That word just doesn't look right with an a near the end.

I wasn't saying no one knows. I was saying it has been credibly stated, but of course you can't prove it for sure.

[Remnantofisrael]

oh, so we do disagree in point of fact. Ok. Where has this been creditably stated? I mean, I'm not arguing that experimental LIKELY is the FOUNDATION of the scale, but we have absolutely no idea how significant the test-day scores are in the end all scale.

[Remnantofisrael]

and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

[LSAT Blog]

This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.

This is a quote from someone in high places at LSAC back in 2009 I believe. I stole it from a blog a while ago but don't remember where the hell it came from so apologies.

That'd be me, haha. It's all good. I've written a series of posts about how the "curve" is created (yes, tests are equated, not curved).

[Patriot1208]

This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.

This is a quote from someone in high places at LSAC back in 2009 I believe. I stole it from a blog a while ago but don't remember where the hell it came from so apologies.

That'd be me, haha. It's all good. I've written a series of posts about how the "curve" is created (yes, tests are equated, not curved).

You are high up in LSAC?

[d34d9823]

and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

Well, that's the thing. I saw it on here and thought it was credible, but at this point, I'd have to google to find it and anybody can do that.

[Remnantofisrael]

He means I stole it from his awesome blog. THANK YOU for crediting it- I hate just stealing crap like this- I'll edit back to give credit.

That said, the quote is actually Lynda Reese (from LSAC) and its from Feb, but its due Steve Schwartz is the one who found it and blogged about it. I recommend, if you are actually interested in the science/theory of it, to read his blog on it. I'm sure if I link I'll get slapped by the mods, but just google some of the text in the quote and you'll find the series of blog posts.

[SchopenhauerFTW]

.

[Remnantofisrael]

and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

Well, that's the thing. I saw it on here and thought it was credible, but at this point, I'd have to google to find it and anybody can do that.

If anyone cares for the answers, read this guys blog I mention above. He really goes into great detail. I feel like such a shill promoting it, but really, its the best detailed explanation I've seen. And it supports what you are saying, d34dluk3, but I would like to know to what extent. The blog seems to indicate that there are only two possibilities for any test scale-

1) Each question is determined to have a relatively consistent and expected rate of correct answers after the test.
2) Some question is determined to be way off, and so is thrown out.

This would seem to suggest that the scale is determined COMPLETELY before hand, but many folks seem to believe that the actual test has some weight on the scale.

I honestly do not know the answer and would love to be schooled.

[OrdinarilySkilled]

I get the equating part of it, I just don't see how the whole group of people taking the test fall perfectly into the percentiles each time without adjusting it a bit. Unless the percentiles change and the score stays the same.

[Remnantofisrael]

I think its that they know each questions relative normalized difficulty and so based on a combination they can get within a few points of equal tests from one exam to the other, but with a slight variation. For example a 4% variation in difficulty would be the difference, on two 100 question exams, between a -7 curve and a -11. So they basically know ahead of time. But other factors play a role. For example, there is no way to really know how long test takers take for each question, so perhaps the combo of questions is actually harder in a group then expected. Because of stuff like that, there is the suggestion, if I understand correctly, that there is some post-test fixing.

But who knows right?

[ajmanyjah]

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

...I know IRT and test-equating (and I know curve is wrong but it is what is used to determine the 170 here)---studied it pretty intensively in school.

The issue is, if IRT is used (basically confusion matrices) and questions are ranked in difficulty and curved individually, how can every question have the same value in the bell curve?

Anyway, here's hoping to another-14.