Does LSAC adjust their experimentals after administering...

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Does LSAC adjust their experimentals after administering...

Postby crestor » Wed Sep 25, 2013 4:37 pm

Do you guys think LSAC adjusts their experimentals after administering them as experimentals on official tests? I sat for june and had experimental lg. I subsequently cancelled this test because I bombed the real lg. however, you can usually tell when you do good on lg and the experimental lg was in my opinion much easier than the real lg.

My thought process is that LSAC will not put on an experimental that was an experimental onto the test as a real section within at least a period of one year. Why? Because someone like me for instance who sat down for June would have a tremendous advantage if LSAC put the experimental lg on june as their real section on October. This would be unfair to other test takers because I have seen these questions before and would have a head start SOMEWHAT in making deductions and the diagram.. I recall in the june 2013 waiting thread that someone who said they sat for december 2011 had that rc as an experimental.

Nonetheless, do you guys think LSAC makes adjustments to a section after administering it as an experimental or do you guys think LSAC keeps it as is?

Thanks.

User avatar
PourMeTea
Posts: 6508
Joined: Sun Jun 17, 2012 8:32 am

Post removed.

Postby PourMeTea » Wed Sep 25, 2013 4:38 pm

Post removed.
Last edited by PourMeTea on Fri May 08, 2015 12:21 am, edited 1 time in total.

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Re: Does LSAC adjust their experimentals after administering...

Postby crestor » Wed Sep 25, 2013 4:39 pm

PourMeTea wrote:
crestor wrote:Do you guys think LSAC adjusts their experimentals after administering them as experimentals on official tests? I sat for june and had experimental lg. I subsequently cancelled this test because I bombed the real lg. however, you can usually tell when you do good on lg and the experimental lg was in my opinion much easier than the real lg.

My thought process is that LSAC will not put on an experimental that was an experimental onto the test as a real section within at least a period of one year. Why? Because someone like me for instance who sat down for June would have a tremendous advantage if LSAC put the experimental lg on june as their real section on October. This would be unfair to other test takers because I have seen these questions before and would have a head start SOMEWHAT in making deductions and the diagram.. I recall in the june 2013 waiting thread that someone who said they sat for december 2011 had that rc as an experimental.

Nonetheless, do you guys think LSAC makes adjustments to a section after administering it as an experimental or do you guys think LSAC keeps it as is?

Thanks.


They make adjustments to avoid --ITEM REMOVED FROM SCORING--


Thanks! I wish I could be a fly on the wall when that discussion occurs among the head test makers.

bp shinners
Posts: 3091
Joined: Wed Mar 16, 2011 7:05 pm

Re: Does LSAC adjust their experimentals after administering...

Postby bp shinners » Wed Sep 25, 2013 7:07 pm

crestor wrote:
PourMeTea wrote:
crestor wrote:Do you guys think LSAC adjusts their experimentals after administering them as experimentals on official tests? I sat for june and had experimental lg. I subsequently cancelled this test because I bombed the real lg. however, you can usually tell when you do good on lg and the experimental lg was in my opinion much easier than the real lg.

My thought process is that LSAC will not put on an experimental that was an experimental onto the test as a real section within at least a period of one year. Why? Because someone like me for instance who sat down for June would have a tremendous advantage if LSAC put the experimental lg on june as their real section on October. This would be unfair to other test takers because I have seen these questions before and would have a head start SOMEWHAT in making deductions and the diagram.. I recall in the june 2013 waiting thread that someone who said they sat for december 2011 had that rc as an experimental.

Nonetheless, do you guys think LSAC makes adjustments to a section after administering it as an experimental or do you guys think LSAC keeps it as is?

Thanks.


They make adjustments to avoid --ITEM REMOVED FROM SCORING--


Thanks! I wish I could be a fly on the wall when that discussion occurs among the head test makers.


They've generally tested them a lot before putting them on the test, with those experimentals mainly to set the score conversions when the experimentals actually show up (or, at least, that's my understanding). So while they will adjust for anomalous results, they don't want to tinker at that point as it would mean they would have to retest the section to set the conversion.

User avatar
Clearly
Posts: 4166
Joined: Sat Feb 11, 2012 4:09 pm

Re: Does LSAC adjust their experimentals after administering...

Postby Clearly » Wed Sep 25, 2013 7:25 pm

My question is how long do they wait? Granted I'm not looking for them, but I've heard 0 cases of people seeing those sections on retake, which means either they wait a long long time, or they are switching between Sabbath and non Sabbath tests because you can't switch back and forth. This seems important to me with the relative lack of high scores applying. Granted we don't know if the 170s took, then opted not to apply, but I assume at least some actual drop on scores for the administered tests, which means eventually that experimental data will do screwy things to the difficulty of the test, or the percentiles will shift... when the last year's experimental becomes real is when we'll see if I'm correct.

User avatar
JMKeynes
Posts: 27
Joined: Thu Jun 06, 2013 7:35 am

Re: Does LSAC adjust their experimentals after administering...

Postby JMKeynes » Wed Sep 25, 2013 7:30 pm

I sat for June 2009 (infamous dinos test) and I swear I've seen some LR questions that were in my experimental that day in prepping for the October exam this year. I specifically remember a question about QWERTY keyboards that I've run across in my recent prep. Obviously I can't confirm if this was the actual question because I'm sure keyboards have shown up in several LR sections but this one just seemed distinctly familiar.

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Re: Does LSAC adjust their experimentals after administering...

Postby crestor » Wed Sep 25, 2013 7:33 pm

JMKeynes wrote:I sat for June 2009 (infamous dinos test) and I swear I've seen some LR questions that were in my experimental that day in prepping for the October exam this year. I specifically remember a question about QWERTY keyboards that I've run across in my recent prep. Obviously I can't confirm if this was the actual question because I'm sure keyboards have shown up in several LR sections but this one just seemed distinctly familiar.


http://www.manhattanlsat.com/forums/q3- ... t6480.html

http://www.manhattanlsat.com/forums/q21 ... t1576.html

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Re: Does LSAC adjust their experimentals after administering...

Postby crestor » Wed Sep 25, 2013 7:36 pm

Clearly wrote:My question is how long do they wait? Granted I'm not looking for them, but I've heard 0 cases of people seeing those sections on retake, which means either they wait a long long time, or they are switching between Sabbath and non Sabbath tests because you can't switch back and forth. This seems important to me with the relative lack of high scores applying. Granted we don't know if the 170s took, then opted not to apply, but I assume at least some actual drop on scores for the administered tests, which means eventually that experimental data will do screwy things to the difficulty of the test, or the percentiles will shift... when the last year's experimental becomes real is when we'll see if I'm correct.



if anything like this occurs, this bodes especially well to people taking it (2015, 2016) if less qualified test takers are taking it now compared to 5 years ago.

User avatar
Jeffort
Posts: 1896
Joined: Wed Jun 18, 2008 4:43 pm

Re: Does LSAC adjust their experimentals after administering...

Postby Jeffort » Wed Sep 25, 2013 11:54 pm

According to the academic article "Developing and assembling the LSAT", every question appears in an administered experimental section at least twice before it appears in a scored section.

The article talks about two main different types of experimental sections. One type are designed to be 'as is' operational sections just getting their last experimental run to get final ratings for the individual questions as well as for the section as a whole in order to calibrate it for putting it together with other sections to form a complete test form. These are questions at the end of the pre-testing process in the final stage before they become an operational section.

The other main type is for questions earlier in the pre-testing process to get performance data about the questions individually, but not necessarily with much concern as to how they operate within that particular section with the other questions in terms of the overall section difficulty and distribution of skills tested and Q types. They are not intended to be used as is sections, but mainly just to see how individual questions perform to get baseline data about each item that is used later to construct a hopefully fully balanced section in terms of overall difficulty as well as proper distribution of questions of each difficulty level, distribution of Q types, concepts/skills tested, etc. The purpose is to get initial item specific performance data under test conditions in section format.

These questions are re-packaged later into another section with different questions as a pre-operational section for final pre-testing as a section that is intended to be a fully balanced section with proper distributions of everything according to test specs (the first type described above). These are probably the type of experimental section that people get when they report that their exp section was way easier or harder than usual and/or weird in some way with balance, Q type distribution or whatever.



The article is interesting for LSAT nerds but not useful for preparing for the test. It gives a lot of technical info about what goes into test development, but far from a complete picture of the process. Interesting nonetheless.

http://www.math.washington.edu/~billey/ ... Week.3.pdf

IDK for sure, but from the article and other things I've read and know about LSAT test development, and from my understanding of basic statistical methods, I don't think the significant drop in test takers and corresponding drop in number of achieved 170+ score in the last three years is going to make any significant difference to the difficulty ratings assigned to questions and sections pre-tested during the period as Clearly suggested may occur and certainly don't think it is going to artificially skew the scales/difficulty ratings of upcoming test forms.

The types of hard core statistical analysis and psychometric processes involved in creating the test are very advanced and I'd bet have measures built in to account and equate for both qualitative and quantity differences in test taker volumes and populations from year to year.

One big aspect of the assembly process is a large element of random selection from the item pool (notice the parts in the article that talk about a Monte Carlo approach) that likely has specifications to insure that each full test form is composed of items that were pretested at various different times relative to each other so that you don't end up with a test form composed of sections that were all pre-operationally final experimental tested within the same/overlapping short time span which could result in the experimental data being skewed by what could be a less representative than typical sample of the population of test takers that took those exp sections. That seems like an obvious type of potential statistical sampling bias the system would be designed to prevent from happening even when there are unusual temporary shifts in the test taker population from test to test, cycle to cycle. Making sure to have representative cross sections, sampling and all that jazz is the foundation of any good statistically designed thing and I doubt the LSAC psychometricians forgot the foundational basics of good statistical analysis and predictions.

I'm sure part of the assembly process is designed to insure that any particular final test form ends up having questions that were pretested at various different times over a span covering several years. Meaning that some questions on a new test may have been written and begun the pre-testing process many many years ago while others in the same test/section may be really new items, along with others of medium age, etc. LSAC keeps a large item bank available at any given time sufficient to produce enough full test forms for several testing cycles even if they completely stop writing new items. It mentions how many items they had in the item bank ready to go as of the time that article was written and it was huge!

User avatar
thisiswater
Posts: 995
Joined: Wed May 05, 2010 1:51 am

Re: Does LSAC adjust their experimentals after administering...

Postby thisiswater » Thu Sep 26, 2013 10:57 am

I sat December 2009 and PT 63 G4 - Balls in boxes was definitely on my experimental. So that's about an 18 month turnaround there

User avatar
jingosaur
Posts: 2218
Joined: Fri Jan 04, 2013 10:33 am

Re: Does LSAC adjust their experimentals after administering...

Postby jingosaur » Thu Sep 26, 2013 11:36 am

I could definitely see the 2 runs thing being true. My December 2012 experimental was just like a real RC section and my June 2013 LG section was the sketchiest thing I've ever seen on any LSAT. I have recurring nightmares about getting to test day with there being a different section type and that pretty much came true in June 2013.

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Re: Does LSAC adjust their experimentals after administering...

Postby crestor » Thu Sep 26, 2013 1:35 pm

jingosaur wrote:I could definitely see the 2 runs thing being true. My December 2012 experimental was just like a real RC section and my June 2013 LG section was the sketchiest thing I've ever seen on any LSAT. I have recurring nightmares about getting to test day with there being a different section type and that pretty much came true in June 2013.


are you talking about the petri dishes para legal june lg? or the experimental lg?

User avatar
jingosaur
Posts: 2218
Joined: Fri Jan 04, 2013 10:33 am

Re: Does LSAC adjust their experimentals after administering...

Postby jingosaur » Thu Sep 26, 2013 1:48 pm

crestor wrote:
jingosaur wrote:I could definitely see the 2 runs thing being true. My December 2012 experimental was just like a real RC section and my June 2013 LG section was the sketchiest thing I've ever seen on any LSAT. I have recurring nightmares about getting to test day with there being a different section type and that pretty much came true in June 2013.


are you talking about the petri dishes para legal june lg? or the experimental lg?


My experimental LG was really weird.

The real one was IMO one of the easier LG sections.

User avatar
crestor
Posts: 324
Joined: Thu Jan 10, 2013 5:37 am

Re: Does LSAC adjust their experimentals after administering...

Postby crestor » Thu Sep 26, 2013 4:38 pm

jingosaur wrote:
crestor wrote:
jingosaur wrote:I could definitely see the 2 runs thing being true. My December 2012 experimental was just like a real RC section and my June 2013 LG section was the sketchiest thing I've ever seen on any LSAT. I have recurring nightmares about getting to test day with there being a different section type and that pretty much came true in June 2013.


are you talking about the petri dishes para legal june lg? or the experimental lg?


My experimental LG was really weird.

The real one was IMO one of the easier LG sections.


I loved the experimental. Was done with 5 minutes left and unsure on 3 questions max. Lol




Return to “LSAT Prep and Discussion Forum”

Who is online

Users browsing this forum: somedeadman and 11 guests