How many "difficult" questions in LR

lsatim
Posts: 14
Joined: Mon Oct 01, 2012 4:48 pm

How many "difficult" questions in LR

Postby lsatim » Thu Oct 11, 2012 11:58 am

Ok, so this question is slightly ridiculous but I'm curious. I know both Cambridge and Kaplan and I assume others break down the difficulty of question types in ranges like 1-4 and I think Kaplan does it by stars. My question is, what is the relative distribution of difficult questions per LR section? In other words, how many very difficult questions, somewhat difficult, easy etc. (read: 4 for Cambridge) can we expect per section?

I know that the general school of thought is that middle questions tend to be more difficult and the easier ones tend to be at the very end and beginning 11 or so. Obviously, the range is relative (between PTs) and certainly different companies argue certain questions are more difficult.

Just wondering is someone has actually figured this out.
Thanks

ws81086n
Posts: 318
Joined: Sun Aug 26, 2012 9:47 pm

Re: How many "difficult" questions in LR

Postby ws81086n » Thu Oct 11, 2012 12:29 pm

The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.

User avatar
CyanIdes Of March
Posts: 743
Joined: Thu Aug 16, 2012 1:57 pm

Re: How many "difficult" questions in LR

Postby CyanIdes Of March » Thu Oct 11, 2012 12:45 pm

ws81086n wrote:The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.


I'm usually pretty good at LR (~-2 to -4 average both LR sections together) but I found SP B LR on the difficult side. Guess it just depends on skill set.

User avatar
Cerebro
Posts: 239
Joined: Thu Aug 09, 2012 9:22 pm

Re: How many "difficult" questions in LR

Postby Cerebro » Thu Oct 11, 2012 12:47 pm

What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.

ws81086n
Posts: 318
Joined: Sun Aug 26, 2012 9:47 pm

Re: How many "difficult" questions in LR

Postby ws81086n » Thu Oct 11, 2012 12:53 pm

CyanIdes Of March wrote:
ws81086n wrote:The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.


I'm usually pretty good at LR (~-2 to -4 average both LR sections together) but I found SP B LR on the difficult side. Guess it just depends on skill set.


That's odd. I'm in the 4-6 range, and I think I was a bit worse when I took B. I actually just calculated the difficulty average for B's LR sections, and it came out to 2.68 and 3.3, respectively. 2.68 seems very low. And I think my assessment of the wrong answers holds for both sections, though more so in the case of the lower-difficulty section. Willing to bet you were just having an off day, since you crush LR otherwise.

Swimp
Posts: 493
Joined: Sat May 26, 2012 9:32 am

Re: How many "difficult" questions in LR

Postby Swimp » Thu Oct 11, 2012 12:56 pm

Speaking entirely based on my subjective experience, I'd say there are around four difficult questions per section. I generally get through LR with 5-10 minutes to spare and go back to questions I marked as tricky, and usually there are around four of them.

ws81086n
Posts: 318
Joined: Sun Aug 26, 2012 9:47 pm

Re: How many "difficult" questions in LR

Postby ws81086n » Thu Oct 11, 2012 12:59 pm

Cerebro wrote:What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.


I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.

User avatar
NoodleyOne
Posts: 2358
Joined: Fri May 25, 2012 7:32 pm

Re: How many "difficult" questions in LR

Postby NoodleyOne » Thu Oct 11, 2012 1:10 pm

LSATQA is also a pretty good resource for this. They rank difficulty on percentage of answers that were incorrect. It's probably a more organic system than an arbitrary difficulty rating, since the most difficult ones are the ones most test takers got incorrect. If you look at a test by section you'll see the relative difficulty.

Theopliske8711
Posts: 2213
Joined: Mon Oct 01, 2012 10:21 am

Re: How many "difficult" questions in LR

Postby Theopliske8711 » Thu Oct 11, 2012 1:16 pm

I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.

lsatim
Posts: 14
Joined: Mon Oct 01, 2012 4:48 pm

Re: How many "difficult" questions in LR

Postby lsatim » Thu Oct 11, 2012 1:23 pm

Thanks. I realize that it can't be objective. I more just want to understand the number of very difficult questions for timing purposes. I'm definitely going to check out the Super Preps and LSATQA - I didn't know they had that feature. I really should input more of my PT's into something that tracks them more efficiently than just Excel.

ws81086n
Posts: 318
Joined: Sun Aug 26, 2012 9:47 pm

Re: How many "difficult" questions in LR

Postby ws81086n » Thu Oct 11, 2012 1:24 pm

Theopliske8711 wrote:I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.


Yeah, that's def what they're based on, but as far as I know this is not readily available for each test, right? LSATQA's data seems like it would skew less difficult, since the sample of test-takers that use it is liable to be better than average...

Theopliske8711
Posts: 2213
Joined: Mon Oct 01, 2012 10:21 am

Re: How many "difficult" questions in LR

Postby Theopliske8711 » Thu Oct 11, 2012 1:44 pm

ws81086n wrote:
Theopliske8711 wrote:I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.


Yeah, that's def what they're based on, but as far as I know this is not readily available for each test, right? LSATQA's data seems like it would skew less difficult, since the sample of test-takers that use it is liable to be better than average...


Can we reasonable make that assumption? I don't know much about LSATQA to say. Do they have average LSAT numbers for the site compiled? If its in the high 150s or 160s, then it would be telling.

User avatar
arcanecircle
Posts: 128
Joined: Sun Sep 16, 2012 12:33 am

Re: How many "difficult" questions in LR

Postby arcanecircle » Thu Oct 11, 2012 3:20 pm

--LinkRemoved--

The site is relatively new, and of the 150ish people registered, most are already in the 160+ range. Among other things, this would skew the difficulty rating of easier problems.

User avatar
Cerebro
Posts: 239
Joined: Thu Aug 09, 2012 9:22 pm

Re: How many "difficult" questions in LR

Postby Cerebro » Thu Oct 11, 2012 3:37 pm

ws81086n wrote:
Cerebro wrote:What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.


I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.



Does LSAC publish % correct numbers for all of the PTs? I assume these data may be available somewhere, since LRB frequently mentions the percentage of test takers who selected each of the answer choices. If these data are available, where can I get them?


I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.

User avatar
NoodleyOne
Posts: 2358
Joined: Fri May 25, 2012 7:32 pm

Re: How many "difficult" questions in LR

Postby NoodleyOne » Thu Oct 11, 2012 4:30 pm

I actually prefer the "skewed" data of a site like LsatQA. Who cares if a bunch of 140s keep getting a question wrong? Finding which questions skilled test takers are missing is much more telling of actual difficulty in my opinion.

ws81086n
Posts: 318
Joined: Sun Aug 26, 2012 9:47 pm

Re: How many "difficult" questions in LR

Postby ws81086n » Thu Oct 11, 2012 4:38 pm

Cerebro wrote:
ws81086n wrote:
Cerebro wrote:What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.


I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.



Does LSAC publish % correct numbers for all of the PTs? I assume these data may be available somewhere, since LRB frequently mentions the percentage of test takers who selected each of the answer choices. If these data are available, where can I get them?


I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.


That's a good point. Would love to get a hold of the data if it's available.

User avatar
Lyov Myshkin
Posts: 232
Joined: Sat Jun 28, 2008 11:28 pm

Re: How many "difficult" questions in LR

Postby Lyov Myshkin » Fri Oct 12, 2012 4:22 am

Theopliske8711 wrote:Can we reasonable make that assumption? I don't know much about LSATQA to say. Do they have average LSAT numbers for the site compiled? If its in the high 150s or 160s, then it would be telling.


I can actually just tell you.

Users: ~300
Inputted LSATs: ~1500
Average: ~165

arcanecircle wrote:--LinkRemoved--

The site is relatively new, and of the 150ish people registered, most are already in the 160+ range. Among other things, this would skew the difficulty rating of easier problems.


Personally, I think that the fact that our user base is pretty self-selecting actually skews all of the difficulty ratings on the problems, not just the ratings on the easier ones. But I think it skews all the difficulty ratings in the same direction, which is the important point since that can be adjusted for. Unless there was evidence against it, I would assume that high scorers get more easy questions right as well as hard questions right. I am making a statistical assumption here but given evidence to the contrary I'm not sure why that assumption would be incorrect or anything.

Cerebro wrote:I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.


That sounds reasonable. I can just explain how we get those numbers, and I'll just leave it to you to come to your own conclusions about it.

NoodleyOne, you were surprisingly accurate about our ranking of question difficulty. We do, in fact, rank difficulty of questions based on user data, i.e. the percentages of correct vs incorrect answers. We adjust the relative difficulty ratings so that they scale from the easiest question on the section to the hardest question in the section so, yeah, I also like to think the rankings do evolve over time, getting more accurate as more data is fed into it. Organic describes it well, I think.

Anyhow, I hope this helps. Glad to see that people like the tool.

Nat Sherman
Posts: 68
Joined: Thu Aug 16, 2012 9:12 pm

Re: How many "difficult" questions in LR

Postby Nat Sherman » Fri Oct 12, 2012 6:03 am

NoodleyOne wrote:LSATQA is also a pretty good resource for this. They rank difficulty on percentage of answers that were incorrect. It's probably a more organic system than an arbitrary difficulty rating, since the most difficult ones are the ones most test takers got incorrect. If you look at a test by section you'll see the relative difficulty.

I find the answer break down really interesting. I only looked over five of the PTs I did on lsatqa, however every answer seemed very clear cut in how lsac tried to get you to answer the wrong answer based on the responses. Almost all the questions had at least 2 and most the times 3 answers that no one ever picked, which helped tremendously in seeing the answers that they were trying to get you to go for.

User avatar
hallbd16
Posts: 136
Joined: Tue Aug 14, 2012 12:40 pm

Re: How many "difficult" questions in LR

Postby hallbd16 » Fri Oct 12, 2012 8:55 am

So I planning on trying out LSAT QA on PTs moving forward, but I would love to know why (if) it is more valuable than the excel spreadsheet provided by 3link: viewtopic.php?f=6&t=152117

So far I reckon:
-One breaks down the question type for you (which has advantages and potential disadvantage of less time analyzing your own answers)
-One can provide comparative question difficulty based on other registered users data input
-Why else would you use one of them over the other, or do you use both?




Return to “LSAT Prep and Discussion Forum”

Who is online

Users browsing this forum: Instrumental, Yahoo [Bot] and 9 guests