How have people generally performed during practice tests? Forum

[Lyov Myshkin]

In our most ambitious data visualization to date, we've spliced our data together so that you can explore it across several dimensions. Basically, we've taken tens of thousands of records and restructured it so that you can see how people have tended to perform based on section, question number, and PT (and combinations of those factors).

You can check out the chart here (you may have to give it a few seconds to load). (LinkRemoved)

--------
a small caveat - Our sample is a bit skewed since the people who tend to use our site score significantly better than the average lsat test taker. At time of writing (7-31-12), our average score hovers at above 164, which means that our average user is actually scoring around the 90% percentile. We think this is due to the fact that people who use our service are likely a self-selecting sample, and probably even a bit more biased than even the TLS community, since our service is a bit more specialized.
--------

Our most significant observation (and you may want to skip this part if you would like to explore the dataset without tainting it with our inferences, i.e. with a clean slate, so to speak):


We think that this dataset gives support to the common wisdom to do 'the first 10 in 10' or '15 in 15'.

Why?

• The difficulty of a question appears to be inversely proportional to question number. In other words, as you move through a section, questions tend to get progressively more difficult. The implications of this is that the amount of time that a person should be willing to expend on a given question in order to get it correct should also tend to increase progressively through a section.

If you have a unique interpretation of the data, please share!

[TopHatToad]

Great analysis! I do have one tweak to your interpretation, however. My understanding (based on personal experience and word-of-mouth but no data) is that LR question difficulty peaks slightly before the end, roughly in the late teens to early twenties. For LSAC, this would present a "roadblock" to test takers who tackled all the questions in order, rather than selecting their questions more judiciously. It may be that this data was obscured in your analysis by people who don't quite have time to reach the end, instead just bubbling blindly. What do you think?

[Lyov Myshkin]

Great analysis! I do have one tweak to your interpretation, however. My understanding (based on personal experience and word-of-mouth but no data) is that LR question difficulty peaks slightly before the end, roughly in the late teens to early twenties. For LSAC, this would present a "roadblock" to test takers who tackled all the questions in order, rather than selecting their questions more judiciously. It may be that this data was obscured in your analysis by people who don't quite have time to reach the end, instead just bubbling blindly. What do you think?

It's funny, I totally remember that idea being communicated to me when I was studying for the test, but upon looking at the data, I'm no longer so certain of it. If you look at the slope, it's a pretty linearly sloped.

The linear slope could be due to the fact that

(1) questions merely get progressively more difficult

or..

(2) questions get more difficult but then easier but the easier questions tend to be missed more because of bad time management.

or..

(3) something else which I'm not considering.

To be honest, the simplicity of (1) appeals to me.. :/

[NoodleyOne]

I would think the time crunch at the end plays a significant factor. Most people don't finish LR sections with time left on the clock, so those last couple of questions are probably often guesses.

[TopHatToad]

Great analysis! I do have one tweak to your interpretation, however. My understanding (based on personal experience and word-of-mouth but no data) is that LR question difficulty peaks slightly before the end, roughly in the late teens to early twenties. For LSAC, this would present a "roadblock" to test takers who tackled all the questions in order, rather than selecting their questions more judiciously. It may be that this data was obscured in your analysis by people who don't quite have time to reach the end, instead just bubbling blindly. What do you think?

It's funny, I totally remember that idea being communicated to me when I was studying for the test, but upon looking at the data, I'm no longer so certain of it. If you look at the slope, it's a pretty linearly sloped.

The linear slope could be due to the fact that

(1) questions merely get progressively more difficult

or..

(2) questions get more difficult but then easier but the easier questions tend to be missed more because of bad time management.

or..

(3) something else which I'm not considering.

To be honest, the simplicity of (1) appeals to me.. :/

Hey, I like simplicity as much as the next guy, Occam is my man. It just seems to me that you're not accounting for people who don't finish all questions, and therefore skew the % correct at the end of each section. Even for a sample pool averaging 164, it's likely a significant factor.

[Lyov Myshkin]

It just seems to me that you're not accounting for people who don't finish all questions, and therefore skew the % correct at the end of each section. Even for a sample pool averaging 164, it's likely a significant factor.

I concede that that's a valid interpretation. But it gets tricky also to think about what 'difficulty' really means.. if 'difficulty' merely means percentage that one is likely to get an answer correct, then in virtue of the fact that some significant fraction of people don't finish all the questions, the difficulty of later questions has increased. Statistically, anyway..

[TopHatToad]

I concede that that's a valid interpretation. But it gets tricky also to think about what 'difficulty' really means.. if 'difficulty' merely means percentage that one is likely to get an answer correct, then in virtue of the fact that some significant fraction of people don't finish all the questions, the difficulty of later questions has increased. Statistically, anyway..

Fair enough. It's testable, but I doubt anyone would go through the trouble to test it. For all you lurkers out there, I'm an LSAT teacher as well, and my general advice to students is as follows:

For LR, tackle the questions in order until you run into a couple really tough time-killers in a row (roughly in the 17-22 range). That's your indicator that you've hit the hardest questions. From there, jump straight to the last question and work your way backwards until you're only left with that toughest section. That way, you've already hit all of the easier problems first, and you can devote your final minutes to the toughies without any fear of the clock.

[cc.celina]

This seems like a reasonable interpretation. I agree that it can sometimes seem like difficulty peaks near the end and then gets a little easier at the very end, but either way, it supports the 'first 15 in 15' wisdom.

I am more posting just to thank you for sharing with us. I don't currently tutor, but when I try to help around the boards I link many posters to LSAT QA because it was integral to my prep, and a few posters have told me they're using it now. What a great resource. Thank you!!

[Lyov Myshkin]

This seems like a reasonable interpretation. I agree that it can sometimes seem like difficulty peaks near the end and then gets a little easier at the very end, but either way, it supports the 'first 15 in 15' wisdom.

I am more posting just to thank you for sharing with us. I don't currently tutor, but when I try to help around the boards I link many posters to LSAT QA because it was integral to my prep, and a few posters have told me they're using it now. What a great resource. Thank you!!

thanks! It's always great to hear people getting utility from this. Personally, I think it's only going to get better.

I would like to implement this cross-section type visualization into the user analytics area, which I think would be pretty cool. Imagine for a second, being able to sort through all your tests like this, click on a question type tag, filter all your results, filter by pt or when you took the test... could be pretty enlightening