Isn't the curve determined by the experimental?

ajmanyjah
Posts: 263
Joined: Wed Jun 30, 2010 4:51 pm

Isn't the curve determined by the experimental?

Postby ajmanyjah » Thu Oct 14, 2010 7:55 pm

Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

User avatar
Patriot1208
Posts: 7044
Joined: Tue May 18, 2010 11:28 am

Re: Isn't the curve determined by the experimental?

Postby Patriot1208 » Thu Oct 14, 2010 7:56 pm

ajmanyjah wrote:Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)


Yes, but I think people are extrapolating on how they did and how those in last june and february must have done.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 8:05 pm

This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.


This is a quote from LSAC's Associate Director of Psychometric Research, Lynda Reese in Feb of 2010 I believe. I stole it from Steve Schwartz's blog, LSAT Blog (which I would link but I think I'm not allowed to).

THAT SAID, I thought common belief is that no one really knows exactly how it is formed. BUT that the "scale" is determined before the test is even given by combination of factors including the experimental. THEN, based on statistics of the actual test, things CAN be adjusted SLIGHTLY or questions thrown out.

The idea, though, is that you are talking about a test put together by a bunch of PhD's who understand this stuff better than almost anyone else on the planet and that they know full well what to expect in any given group from HUGE statistical analysis. Hundreds of thousands of test data, and tens of thousands applying directly to the questions on this particular test.

So what does all this mean? That if the test was hard and people screwed the pooch, the test makers probably knew this already. If they didn't, then the test maybe wasn't really any harder than usual tests of the past (in, say, the -8, -9 zone) and that, instead, there were 4-5 REALLY tough ?'s and a whole bunch of easy ones balancing it out, which would make the TLS community assume that this was a particularly hard test when perhaps it wasn't. Or maybe the test makers thought this was a -9 test, the data says it was more like a -11, and in the end we get a -10.

Or maybe there is a room full of monkeys, all given laxatives and shit-ton of bananas. When the last monkey shits, they check to see how much time passed and determine the scale based on that. That is what I am going with.
Last edited by Remnantofisrael on Thu Oct 14, 2010 9:02 pm, edited 1 time in total.

Sandro
Posts: 2526
Joined: Sat Jul 18, 2009 12:12 am

Re: Isn't the curve determined by the experimental?

Postby Sandro » Thu Oct 14, 2010 8:14 pm

So what you're saying is a -14 curve? wooooo

d34d9823
Posts: 1915
Joined: Wed Apr 14, 2010 2:52 pm

Re: Isn't the curve determined by the experimental?

Postby d34d9823 » Thu Oct 14, 2010 8:15 pm

ajmanyjah wrote:Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 8:18 pm

d34dluk3 wrote:
ajmanyjah wrote:Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

Kinda.

Maybe.

I think.



./ \
/...\
..|
..|
..|
There- fixed that for ya.

d34d9823
Posts: 1915
Joined: Wed Apr 14, 2010 2:52 pm

Re: Isn't the curve determined by the experimental?

Postby d34d9823 » Thu Oct 14, 2010 8:21 pm

Remnantofisrael wrote:
d34dluk3 wrote:
ajmanyjah wrote:Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.

Kinda.

Maybe.

I think.



./ \
/...\
..|
..|
..|
There- fixed that for ya.

:roll: This has been stated before. If OP wanted a boring "nobody knows for sure", she wouldn't have asked.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 8:24 pm

Maybe I'm on LR mode and need to shut down, but if the answer is "no one knows" then the answer isn't "yes" unless the question as to how the curve/scale is determined is:

"Is it true that no one knows how the scale is determined, other than LSAC and anyone else with that privileged knowledge that won't be sharing?"

Then yes is correct.

Also, I'm just having fun at your expense. I'm sure if you took a minute to look at my posts in just the last 30 minutes there are plenty of grammar and spelling errors. Grammer. Grammar. That word just doesn't look right with an a near the end.

d34d9823
Posts: 1915
Joined: Wed Apr 14, 2010 2:52 pm

Re: Isn't the curve determined by the experimental?

Postby d34d9823 » Thu Oct 14, 2010 8:27 pm

Remnantofisrael wrote:Maybe I'm on LR mode and need to shut down, but if the answer is "no one knows" then the answer isn't "yes" unless the question as to how the curve/scale is determined is:

"Is it true that no one knows how the scale is determined, other than LSAC and anyone else with that privileged knowledge that won't be sharing?"

Then yes is correct.

Also, I'm just having fun at your expense. I'm sure if you took a minute to look at my posts in just the last 30 minutes there are plenty of grammar and spelling errors. Grammer. Grammar. That word just doesn't look right with an a near the end.

I wasn't saying no one knows. I was saying it has been credibly stated, but of course you can't prove it for sure.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 8:30 pm

oh, so we do disagree in point of fact. Ok. Where has this been creditably stated? I mean, I'm not arguing that experimental LIKELY is the FOUNDATION of the scale, but we have absolutely no idea how significant the test-day scores are in the end all scale.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 8:31 pm

and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

User avatar
LSAT Blog
Posts: 1262
Joined: Mon Dec 07, 2009 9:24 pm

Re: Isn't the curve determined by the experimental?

Postby LSAT Blog » Thu Oct 14, 2010 8:39 pm

Remnantofisrael wrote:This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.


This is a quote from someone in high places at LSAC back in 2009 I believe. I stole it from a blog a while ago but don't remember where the hell it came from so apologies.



That'd be me, haha. It's all good. I've written a series of posts about how the "curve" is created (yes, tests are equated, not curved).

User avatar
Patriot1208
Posts: 7044
Joined: Tue May 18, 2010 11:28 am

Re: Isn't the curve determined by the experimental?

Postby Patriot1208 » Thu Oct 14, 2010 8:42 pm

LSAT Blog wrote:
Remnantofisrael wrote:This might get deep, BUT the LSAT isn't really on a Curve at all.

[T]he LSAT is not graded to a curve...Rather, for every form of the LSAT, a statistical process called test equating is carried out to adjust for minor differences in difficulty between different forms of the test. Specifically, the item response theory (IRT) true score equating method is applied to convert raw scores (the number correct) for each administration to a common 120 to 180 scale. A detailed description of this methodology can be found in...Applications of Item Response Theory to Practical Testing Problems...The equating process assures that a particular LSAT scaled score reflects the same level of ability regardless of the ability level of others who tested on the same day or any slight differences in difficulty between different forms of the test. That is, the equating process assures that LSAT scores are comparable, regardless of the administration at which they are earned.


This is a quote from someone in high places at LSAC back in 2009 I believe. I stole it from a blog a while ago but don't remember where the hell it came from so apologies.



That'd be me, haha. It's all good. I've written a series of posts about how the "curve" is created (yes, tests are equated, not curved).


You are high up in LSAC?

d34d9823
Posts: 1915
Joined: Wed Apr 14, 2010 2:52 pm

Re: Isn't the curve determined by the experimental?

Postby d34d9823 » Thu Oct 14, 2010 8:56 pm

Remnantofisrael wrote:and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

Well, that's the thing. I saw it on here and thought it was credible, but at this point, I'd have to google to find it and anybody can do that.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 9:00 pm

He means I stole it from his awesome blog. THANK YOU for crediting it- I hate just stealing crap like this- I'll edit back to give credit.

That said, the quote is actually Lynda Reese (from LSAC) and its from Feb, but its due Steve Schwartz is the one who found it and blogged about it. I recommend, if you are actually interested in the science/theory of it, to read his blog on it. I'm sure if I link I'll get slapped by the mods, but just google some of the text in the quote and you'll find the series of blog posts.

SchopenhauerFTW
Posts: 1793
Joined: Sun Oct 10, 2010 10:22 pm

Re: .

Postby SchopenhauerFTW » Thu Oct 14, 2010 9:07 pm

.
Last edited by SchopenhauerFTW on Mon Dec 10, 2012 2:34 pm, edited 1 time in total.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 9:08 pm

d34dluk3 wrote:
Remnantofisrael wrote:and by the way- your 1600 posts means you could be totally right and I'm completely wrong- I'm fully willing to concede that, but I'd like to know where you are getting your info so I can change what I'm communicating if, in fact, I am wrong.

Well, that's the thing. I saw it on here and thought it was credible, but at this point, I'd have to google to find it and anybody can do that.


If anyone cares for the answers, read this guys blog I mention above. He really goes into great detail. I feel like such a shill promoting it, but really, its the best detailed explanation I've seen. And it supports what you are saying, d34dluk3, but I would like to know to what extent. The blog seems to indicate that there are only two possibilities for any test scale-

1) Each question is determined to have a relatively consistent and expected rate of correct answers after the test.
2) Some question is determined to be way off, and so is thrown out.

This would seem to suggest that the scale is determined COMPLETELY before hand, but many folks seem to believe that the actual test has some weight on the scale.

I honestly do not know the answer and would love to be schooled.

User avatar
OrdinarilySkilled
Posts: 266
Joined: Sun Jun 20, 2010 10:22 am

Re: Isn't the curve determined by the experimental?

Postby OrdinarilySkilled » Thu Oct 14, 2010 9:17 pm

I get the equating part of it, I just don't see how the whole group of people taking the test fall perfectly into the percentiles each time without adjusting it a bit. Unless the percentiles change and the score stays the same.

User avatar
Remnantofisrael
Posts: 335
Joined: Wed May 26, 2010 4:38 pm

Re: Isn't the curve determined by the experimental?

Postby Remnantofisrael » Thu Oct 14, 2010 9:28 pm

I think its that they know each questions relative normalized difficulty and so based on a combination they can get within a few points of equal tests from one exam to the other, but with a slight variation. For example a 4% variation in difficulty would be the difference, on two 100 question exams, between a -7 curve and a -11. So they basically know ahead of time. But other factors play a role. For example, there is no way to really know how long test takers take for each question, so perhaps the combo of questions is actually harder in a group then expected. Because of stuff like that, there is the suggestion, if I understand correctly, that there is some post-test fixing.

But who knows right?

ajmanyjah
Posts: 263
Joined: Wed Jun 30, 2010 4:51 pm

Re: Isn't the curve determined by the experimental?

Postby ajmanyjah » Fri Oct 15, 2010 3:16 pm

d34dluk3 wrote:
ajmanyjah wrote:Everyone is talking about the curve being good or bad because people did well or badly on the test. Isn't it determined by the performance on the experimental section (when it was one?)

It's equating, not a curve, but yes.


...I know IRT and test-equating (and I know curve is wrong but it is what is used to determine the 170 here)---studied it pretty intensively in school.

The issue is, if IRT is used (basically confusion matrices) and questions are ranked in difficulty and curved individually, how can every question have the same value in the bell curve?

Anyway, here's hoping to another-14.




Return to “LSAT Prep and Discussion Forum”

Who is online

Users browsing this forum: cherrygalore, mintme, Tazewell and 8 guests