"Proof" that LSAT difficulty level has been stable over time

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

"Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 8:44 pm

I put proof in quotes for a reason. This is not proof. But it is kind of interesting, to me at least.

Background: I am taking PT 39-69 in random order. The reasons for this are not relevant, though I'm happy to discuss why I did it this way if anyone is interested. But the random order is important for my little analysis, because it has allowed me to track my scores over time and my scores over successive LSATs separately. In other words, if you take the PTs in order, then your scores on the later PTs (the 60s) are "infected" by the fact that you had studied more when you took them. By contrast, if you take them in random order, where some of the 60s are way before the 40s, you have a cleaner look at the PTs and how they changed over time. For example, if you do better on the 60s than the 40s, you can infer that the 60s are easier (for you at least) than the 40s, because you have essentially controlled for getting better at the test over time. Hopefully this makes sense.

Analysis: In Excel, I created a couple of simple line graphs to track my performance. Within each graph I inserted a linear trendline -- basically a straight line that captures the overall trend of the data (upward, downward, neither, etc.). Graph 1 tracks my scores over time -- i.e., the PTs as I did them (52, 44, 63, 59, 39, etc.). Graph 2 tracks my scores over successive LSATs (39, 40, 41...).

Findings: In Graph 1, the trendline is exactly what one would hope to see: upward slope. In truth, I wish it were a bit steeper, but the line moves from 166 to 171 over time. (Diag was 158.) But the interesting thing is Graph 2. The trendline is perfectly -- and I mean perfectly -- flat. In other words, when the tests are sorted from 39 to 69, there is no change in my performance. Had the 40s been "harder," one would expect to see the line slope downward. Had the 40s been "easier," it would slope upward. But it is dead flat. Like literally zero slope. Yes, there is variation along the line (168, 173, etc.), but that's to be expected in any data like this. The fact that the trendline is flat is the key thing.

Interpretation: Keeping in mind that this is really not very scientific (one dude, 25 tests), I do think it's quite striking that LSAC is able to write tests that hold from year to year. We all recognize that there have been changes in format and substance, but whatever psychometrics they are using to ensure that the scores mean the same thing from year to year is pretty impressive.

Caveat: My brain is pretty fried from studying, so apologies if I am missing something really basic that calls into question my interpretation. Also I know that 99% of you are probably rolling your eyes if you've even made it this far. That's cool. It's useless and nerdy. I admit that.

Pancakes12
Posts: 269
Joined: Tue Jul 02, 2013 10:13 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby Pancakes12 » Tue Aug 06, 2013 9:01 pm

I don't think there's any issue with your analysis, besides the fact that it's a small sample (which you recognize). The difficulty across tests may be different depending on the person. For me, the older tests were MUCH easier at one point. My average score differences were once greater than 10 points for older and newer tests.

User avatar
ScottRiqui
Posts: 3640
Joined: Mon Nov 29, 2010 8:09 pm

Re: "Proof" that LSAT difficulty level has been stable over time

Postby ScottRiqui » Tue Aug 06, 2013 9:03 pm

There are variations in the difficulty of the various tests - significant ones. That's what the "curve" is for. Off the top of my head, I remember seeing 170 curves ranging from at least -9 to -16 over the years.

The curve for a particular test is determined before anyone takes the test, so I will say that LSAC does an admirable job of making the scaled scores comparable from test to test, such that (for example), a 168 is always right around a 96th-percentile score. But there have definitely been "harder" and "easier" tests.

I think you'd get more useful information if, instead of plotting your scaled scores, you plotted the percentage of questions on each PT you got right. That would give you some spikes/troughs that would show the relatively difficulties of the past tests.

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 9:06 pm

Right but this is accounting for the curve. I'm saying that, curve included, the tests perform roughly the same over time. It's just a really well engineered exam that allows law schools to know that a 170 in 2013 is basically the same brain as a 170 in 1993. That's all I'm saying.

Pancakes12
Posts: 269
Joined: Tue Jul 02, 2013 10:13 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby Pancakes12 » Tue Aug 06, 2013 9:06 pm

ScottRiqui wrote:There are variations in the difficulty of the various tests - significant ones. That's what the "curve" is for. Off the top of my head, I remember seeing 170 curves ranging from at least -9 to -16 over the years.

The curve for a particular test is determined before anyone takes the test, so I will say that LSAC does an admirable job of making the scaled scores comparable from test to test, such that (for example), a 168 is always right around a 96th-percentile score. But there have definitely been "harder" and "easier" tests.


I think when the OP refers to difficulty, he means the difficulty of achieving a certain score, not the difficulty of the test's questions.

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 9:17 pm

jlb251 wrote:
I think when the OP refers to difficulty, he means the difficulty of achieving a certain score, not the difficulty of the test's questions.


This.

We've all talked about this ad nauseum, but in my view the (more important) question of whether the LSAT has gotten more difficult over time generally gets overshadowed by the (less important) question of whether particular sections have gotten more difficult over time. I think that LSAC has done a pretty amazing job of balancing the borrow and give, so that while RC has gotten harder, LG has gotten easier -- or whatever specific balancing act they have actually achieved.

User avatar
ScottRiqui
Posts: 3640
Joined: Mon Nov 29, 2010 8:09 pm

Re: "Proof" that LSAT difficulty level has been stable over time

Postby ScottRiqui » Tue Aug 06, 2013 9:18 pm

vuthy wrote:Right but this is accounting for the curve. I'm saying that, curve included, the tests perform roughly the same over time. It's just a really well engineered exam that allows law schools to know that a 170 in 2013 is basically the same brain as a 170 in 1993. That's all I'm saying.


That makes sense, then. You can also look here for a historical comparison of scaled scores and the corresponding percentiles. For any give scaled score over the 11-year period shown in the chart, the percentile that score would place you in doesn't vary by much more than 1-1.5%, which is impressive, considering that the test only has a 60-point range, and that scores are all whole numbers, so there's not much granularity in scoring.

Pancakes12
Posts: 269
Joined: Tue Jul 02, 2013 10:13 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby Pancakes12 » Tue Aug 06, 2013 9:20 pm

vuthy wrote:
jlb251 wrote:
I think when the OP refers to difficulty, he means the difficulty of achieving a certain score, not the difficulty of the test's questions.


This.

We've all talked about this ad nauseum, but in my view the (more important) question of whether the LSAT has gotten more difficult over time generally gets overshadowed by the (less important) question of whether particular sections have gotten more difficult over time. I think that LSAC has done a pretty amazing job of balancing the borrow and give, so that while RC has gotten harder, LG has gotten easier -- or whatever specific balancing act they have actually achieved.


At least for me personally, I really have to disagree. I scored much better on the old tests. I would hit 170s (180 on one occasion) while I was still in mid-160s on the newer tests.

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 9:21 pm


That makes sense, then. You can also look here for a historical comparison of scaled scores and the corresponding percentiles. For any give scaled score over the 11-year period shown in the chart, the percentile that score would place you in doesn't vary by much more than 1-1.5%, which is impressive, considering that the test only has a 60-point range, and that scores are all whole numbers, so there's not much granularity in scoring.


Yeah that has always impressed and surprised me to a certain degree, given that the dramatic increase in the availability of high-quality study materials over the past few years should, in theory, have raised the overall ability of the test takers. But they've somehow kept those percentiles in roughly the same place without -- if you agree w/ my theory at least -- making the test significantly harder. I guess reshaping the questions is what makes that possible.

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 9:26 pm

jlb251 wrote:
At least for me personally, I really have to disagree. I scored much better on the old tests. I would hit 170s (180 on one occasion) while I was still in mid-160s on the newer tests.


And you did them in order? That would be a really surprising albeit interesting set of results. You'd expect that the increased knowledge of the test would offset the increased level of difficulty. A 15 point drop as you improve your skills would really seem to suggest that the test had gotten almost incomprehensibly more difficult.

User avatar
ScottRiqui
Posts: 3640
Joined: Mon Nov 29, 2010 8:09 pm

Re: "Proof" that LSAT difficulty level has been stable over time

Postby ScottRiqui » Tue Aug 06, 2013 9:29 pm

vuthy wrote:

That makes sense, then. You can also look here for a historical comparison of scaled scores and the corresponding percentiles. For any give scaled score over the 11-year period shown in the chart, the percentile that score would place you in doesn't vary by much more than 1-1.5%, which is impressive, considering that the test only has a 60-point range, and that scores are all whole numbers, so there's not much granularity in scoring.


Yeah that has always impressed and surprised me to a certain degree, given that the dramatic increase in the availability of high-quality study materials over the past few years should, in theory, have raised the overall ability of the test takers. But they've somehow kept those percentiles in roughly the same place without -- if you agree w/ my theory at least -- making the test significantly harder. I guess reshaping the questions is what makes that possible.


Looking more closely at the page I linked to, it looks like achieving a particular scaled score *has* become slightly easier, at least in certain score bands. Back in 1996, a 158 would put you in the 77.7 percentile, whereas a 158 is now a 73.7 percentile score.

Also, I think curves have gotten smaller and more stable compared to 15-20 years ago, which makes me think the raw difficulty of the test has gone down and steadied out as well.
Last edited by ScottRiqui on Tue Aug 06, 2013 9:32 pm, edited 1 time in total.

Pancakes12
Posts: 269
Joined: Tue Jul 02, 2013 10:13 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby Pancakes12 » Tue Aug 06, 2013 9:30 pm

vuthy wrote:
jlb251 wrote:
At least for me personally, I really have to disagree. I scored much better on the old tests. I would hit 170s (180 on one occasion) while I was still in mid-160s on the newer tests.


And you did them in order? That would be a really surprising albeit interesting set of results. You'd expect that the increased knowledge of the test would offset the increased level of difficulty. A 15 point drop as you improve your skills would really seem to suggest that the test had gotten almost incomprehensibly more difficult.


Mostly in order. Like I said, this "difficulty gap" may differ by person. But yes, as I ascended to more recent tests my scores starting dropping and I had to do a lot of work in LR and RC. It was like a whole new test to me. But I also enjoyed the new tests more once I learned them. I don't know if this is the right word, but they were just more "fun."

User avatar
vuthy
Posts: 379
Joined: Wed Jun 12, 2013 8:55 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby vuthy » Tue Aug 06, 2013 9:36 pm

ScottRiqui wrote:Looking more closely at the page I linked to, it looks like achieving a particular scaled score *has* become slightly easier, at least in certain score bands. Back in 1996, a 158 would put you in the 77.7 percentile, whereas a 158 is now a 73.7 percentile score.

Also, I think curves have gotten smaller and more stable compared to 15-20 years ago, which makes me think the raw difficulty of the test has gone down and steadied out as well.


But don't things tighten up considerably -- i.e., the percentile has been more stable -- at the higher elevations, like 170+?

User avatar
ScottRiqui
Posts: 3640
Joined: Mon Nov 29, 2010 8:09 pm

Re: "Proof" that LSAT difficulty level has been stable over time

Postby ScottRiqui » Tue Aug 06, 2013 9:44 pm

vuthy wrote:
ScottRiqui wrote:Looking more closely at the page I linked to, it looks like achieving a particular scaled score *has* become slightly easier, at least in certain score bands. Back in 1996, a 158 would put you in the 77.7 percentile, whereas a 158 is now a 73.7 percentile score.

Also, I think curves have gotten smaller and more stable compared to 15-20 years ago, which makes me think the raw difficulty of the test has gone down and steadied out as well.


But don't things tighten up considerably -- i.e., the percentile has been more stable -- at the higher elevations, like 170+?


Yes, due to the thin upper "tail" in the LSAT score distribution. The entire range from 170-180 only spans 2-3% of the test-takers, so percentile differences in that range over time are going to be necessarily small.

The "rising tide" of more/better prep material doesn't lift all boats equally; It has a much better chance of turning a 50th percentile scorer into a 55th percentile scorer than it does of turning a 94th percentile scorer into a 99th percentile scorer.

I really wish that chart went back ~10 years further, to the beginning of the "modern" LSAT era.

User avatar
clay7676
Posts: 116
Joined: Sun Feb 10, 2013 1:44 am

Re: "Proof" that LSAT difficulty level has been stable over time

Postby clay7676 » Wed Aug 07, 2013 1:31 am

This table compares all the raw scores from 1991-2013 and how they correspond with the actual score each year.

http://www.cambridgelsat.com/resources/ ... on-charts/

my analysis: I tracked a couple including 164, 166, 178, etc. And it appears that as far as raw score conversion to actual score goes, there is and can be significant fluctuation, ex: 164 score there is a raw score of 85 for June 2007, and for October 1997 76 (which is pretty significant), but obviously this is a curve.
The point being, if you look at the scores the fluctuations differ from year to year and there doesn't appear to be any serious difference from the earliest year to the latest on the chart. And while material may have changed, I don't personally see a lot more difficulty from just recently taking a 2013 PT at my local university, scoring only two points lower than what I did on two PT's from before 2000. I doubt they would make it significantly harder, but it makes sense to tweak it, otherwise certain tricks and niches would catch on from generation to generation and make it easier to master. Moreover, I feel like (this might sound insulting) people in our generation MAY not be as focused and serious about higher education as previous generations (personal experience, certainly not a fact). And this may compensate coincidentally (maybe not?) with the wide availability of materials.

User avatar
KevinP
Posts: 1324
Joined: Sat Sep 26, 2009 8:56 pm

Re: "Proof" that LSAT difficulty level has been stable over time

Postby KevinP » Wed Aug 07, 2013 4:52 am

vuthy wrote:Interpretation: Keeping in mind that this is really not very scientific (one dude, 25 tests), I do think it's quite striking that LSAC is able to write tests that hold from year to year. We all recognize that there have been changes in format and substance, but whatever psychometrics they are using to ensure that the scores mean the same thing from year to year is pretty impressive.

LSAC uses Item Response Theory (https://en.wikipedia.org/wiki/Item_response_theory).
Even if you had a representative sample size and you ran the same analysis, it would still be statistically meaningless. To show the scores are properly equated, you not only have to show identical scores between tests but also identical distribution functions (which require transformations). See: http://www.lsac.org/lsacresources/resea ... -04-03.pdf

ScottRiqui wrote:Looking more closely at the page I linked to, it looks like achieving a particular scaled score *has* become slightly easier, at least in certain score bands. Back in 1996, a 158 would put you in the 77.7 percentile, whereas a 158 is now a 73.7 percentile score.

Since you had quotes around "curve" in your other post, I assume you understand that the test isn't actually curved (https://www.youtube.com/watch?feature=p ... k&t=13m10s).
The percentile changes are expected since the tests are equated. The percentile changes show that the population of test takers has gotten better, not that the test has gotten easier. If you are interested in how it's actually done: http://www.lsac.org/lsacresources/resea ... -05-01.pdf

vuthy wrote:But don't things tighten up considerably -- i.e., the percentile has been more stable -- at the higher elevations, like 170+?

It has been more stable because the score chart follows an approximately normally distributed curve, which is required for many statistical tests. That is why you can miss a couple questions near 150 and not have a score change, but only miss one near 170 and have a score change.




Return to “LSAT Prep and Discussion Forum”

Who is online

Users browsing this forum: No registered users and 2 guests