How many "difficult" questions in LR Forum

[lsatim]

Ok, so this question is slightly ridiculous but I'm curious. I know both Cambridge and Kaplan and I assume others break down the difficulty of question types in ranges like 1-4 and I think Kaplan does it by stars. My question is, what is the relative distribution of difficult questions per LR section? In other words, how many very difficult questions, somewhat difficult, easy etc. (read: 4 for Cambridge) can we expect per section?

I know that the general school of thought is that middle questions tend to be more difficult and the easier ones tend to be at the very end and beginning 11 or so. Obviously, the range is relative (between PTs) and certainly different companies argue certain questions are more difficult.

Just wondering is someone has actually figured this out.
Thanks

[ws81086n]

The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.

[CyanIdes Of March]

The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.

I'm usually pretty good at LR (~-2 to -4 average both LR sections together) but I found SP B LR on the difficult side. Guess it just depends on skill set.

[Cerebro]

What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.

[ws81086n]

The only data available on this is in the Super Prep test book. After each answer explanation, a difficulty of 1-5 is given. I would caution you against drawing too firm a conclusion from SP though, since it is a bit different from the test nowadays. In particular, I found SP's LR to be pathetically easy on the only test I've taken from that book, Test B. Wrong answer choices were extremely easy to eliminate.

I'm usually pretty good at LR (~-2 to -4 average both LR sections together) but I found SP B LR on the difficult side. Guess it just depends on skill set.

That's odd. I'm in the 4-6 range, and I think I was a bit worse when I took B. I actually just calculated the difficulty average for B's LR sections, and it came out to 2.68 and 3.3, respectively. 2.68 seems very low. And I think my assessment of the wrong answers holds for both sections, though more so in the case of the lower-difficulty section. Willing to bet you were just having an off day, since you crush LR otherwise.

[Swimp]

Speaking entirely based on my subjective experience, I'd say there are around four difficult questions per section. I generally get through LR with 5-10 minutes to spare and go back to questions I marked as tricky, and usually there are around four of them.

[ws81086n]

What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.

I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.

[NoodleyOne]

LSATQA is also a pretty good resource for this. They rank difficulty on percentage of answers that were incorrect. It's probably a more organic system than an arbitrary difficulty rating, since the most difficult ones are the ones most test takers got incorrect. If you look at a test by section you'll see the relative difficulty.

[Theopliske8711]

I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.

[lsatim]

Thanks. I realize that it can't be objective. I more just want to understand the number of very difficult questions for timing purposes. I'm definitely going to check out the Super Preps and LSATQA - I didn't know they had that feature. I really should input more of my PT's into something that tracks them more efficiently than just Excel.

[ws81086n]

I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.

Yeah, that's def what they're based on, but as far as I know this is not readily available for each test, right? LSATQA's data seems like it would skew less difficult, since the sample of test-takers that use it is liable to be better than average...

[Theopliske8711]

I think most ratings are based off of numbers correct; unless LSAC is designating difficulty via measure of logic twisting.

Yeah, that's def what they're based on, but as far as I know this is not readily available for each test, right? LSATQA's data seems like it would skew less difficult, since the sample of test-takers that use it is liable to be better than average...

Can we reasonable make that assumption? I don't know much about LSATQA to say. Do they have average LSAT numbers for the site compiled? If its in the high 150s or 160s, then it would be telling.

[arcanecircle]

--LinkRemoved--

The site is relatively new, and of the 150ish people registered, most are already in the 160+ range. Among other things, this would skew the difficulty rating of easier problems.

[Cerebro]

What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.

I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.

Does LSAC publish % correct numbers for all of the PTs? I assume these data may be available somewhere, since LRB frequently mentions the percentage of test takers who selected each of the answer choices. If these data are available, where can I get them?


I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.

[NoodleyOne]

I actually prefer the "skewed" data of a site like LsatQA. Who cares if a bunch of 140s keep getting a question wrong? Finding which questions skilled test takers are missing is much more telling of actual difficulty in my opinion.

[ws81086n]

What objective criteria are used to determine the difficulty rating of a LR question in the Cambridge LSAT groupings? I'm asking, because I've found many "difficult" (level 4) questions that are extraordinarily easy to answer, but some of the "easy" (level 2) questions sometimes required a little more time (maybe I was tired?).

In other words, what makes a LR question difficult or easy?

ETA: Also, is there a resource that rates the "difficulty" of the newer PrepTests? As far as I know, Cambridge LSAT only rates the difficulty of questions from PTs 1-38, and I believe the Kaplan M&P book only goes up to PT 49 or so.

I would caution against using data from anything other than LSAC. With that said, I did think the LSAC difficulty ratings for some questions were a bit strange. Obviously it is a bit skill-set dependent. As for criteria, I don't think you can find the specific answer you're seeking, but complexity of stimulus and attractiveness of wrong answer choices are two obvious determinants that come to mind.

Does LSAC publish % correct numbers for all of the PTs? I assume these data may be available somewhere, since LRB frequently mentions the percentage of test takers who selected each of the answer choices. If these data are available, where can I get them?


I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.

That's a good point. Would love to get a hold of the data if it's available.

[Lyov Myshkin]

Can we reasonable make that assumption? I don't know much about LSATQA to say. Do they have average LSAT numbers for the site compiled? If its in the high 150s or 160s, then it would be telling.

I can actually just tell you.

Users: ~300
Inputted LSATs: ~1500
Average: ~165

--LinkRemoved--

The site is relatively new, and of the 150ish people registered, most are already in the 160+ range. Among other things, this would skew the difficulty rating of easier problems.

Personally, I think that the fact that our user base is pretty self-selecting actually skews all of the difficulty ratings on the problems, not just the ratings on the easier ones. But I think it skews all the difficulty ratings in the same direction, which is the important point since that can be adjusted for. Unless there was evidence against it, I would assume that high scorers get more easy questions right as well as hard questions right. I am making a statistical assumption here but given evidence to the contrary I'm not sure why that assumption would be incorrect or anything.

I'm a little skeptical about data from LSATQA, because I don't know how representative they are, or even whether the people reporting their answers were answering the questions timed or untimed.

That sounds reasonable. I can just explain how we get those numbers, and I'll just leave it to you to come to your own conclusions about it.

NoodleyOne, you were surprisingly accurate about our ranking of question difficulty. We do, in fact, rank difficulty of questions based on user data, i.e. the percentages of correct vs incorrect answers. We adjust the relative difficulty ratings so that they scale from the easiest question on the section to the hardest question in the section so, yeah, I also like to think the rankings do evolve over time, getting more accurate as more data is fed into it. Organic describes it well, I think.

Anyhow, I hope this helps. Glad to see that people like the tool.

[Nat Sherman]

LSATQA is also a pretty good resource for this. They rank difficulty on percentage of answers that were incorrect. It's probably a more organic system than an arbitrary difficulty rating, since the most difficult ones are the ones most test takers got incorrect. If you look at a test by section you'll see the relative difficulty.

I find the answer break down really interesting. I only looked over five of the PTs I did on lsatqa, however every answer seemed very clear cut in how lsac tried to get you to answer the wrong answer based on the responses. Almost all the questions had at least 2 and most the times 3 answers that no one ever picked, which helped tremendously in seeing the answers that they were trying to get you to go for.

[hallbd16]

So I planning on trying out LSAT QA on PTs moving forward, but I would love to know why (if) it is more valuable than the excel spreadsheet provided by 3link: http://www.top-law-schools.com/forums/v ... 6&t=152117

So far I reckon:
-One breaks down the question type for you (which has advantages and potential disadvantage of less time analyzing your own answers)
-One can provide comparative question difficulty based on other registered users data input
-Why else would you use one of them over the other, or do you use both?