Top Law Schools

This is something of a question about the theory of writing standardized tests and your opinion as to the value of the reading comprehension sections. I find the way standardized tests––at least in America, pretty much the only place where a multiple choice response to a passage could affect the entire course of your life and career (that's stunningly messed up)––bizarre and inconsistent. If you've taken a few tests, or looked at others, you will notice the each test can have a very different style of reading-based question––SAT, ACT, GMAT, GRE, MCAT, and, of course, LSAT. In my view, LSAT reading passages and questions are, if difficult, overall reasonable in quality. I would contrast that with the passages on the GRE revised General Test, which has become incredibly stupid. They assess your reading abilities by giving you a 4 to 5 sentence PARAGRAPH and ask you extremely tricky, and in my view often valueless, questions that depend mostly on whether or not you can find a detail, spot a fact, or select a sentence in the passage that performs some function. It often involves essentially no critical thinking, just mechanical fetching. Sorry, I didn't mean for the GRE rant, but if you've taken the GRE, you will know what I'm talking about: They have sacrificed intelligence for computer-based clicking skills that have nothing to do with the reading you ever do in graduate school (I am currently in a Ph.D. program). I can see why the predictive validity studies of the GRE pale in comparison to those of the LSAT.

Anyway, enough of about the limits of the GRE. I am wondering if people know anything about the theory of writing these questions and the selection of passages. I imagine that it is more challenging to come up with those questions and assess their difficulty than one would expect. It must be rather challenging to find a passage that suits the exam: not too tendentious, not too emotional, not too content-specific such that people who happen to have some expertise in the material could potentially be greatly advantaged, but also complex. Who are these people who come up with these questions? Are they just sitting around reading pages and pages of boring texts that aren't too famous that the chances of their having already been read? Are pages and pages of potential reading questions fed through a computer that eliminates passages that appear too literary or too emotional or too intricate (the way computers can grade essays on the basis of key words or sentence structures found in the text)? Who assesses the quality of the question, and the quality of the answer? (Often questions about the "main point" can be strangely worded and rather unexpected, and finding the right answer mostly depends on process of elimination.) Does the fact that many, if not most, reading comprehension questions can be solved by process of elimination speak to a real skill, or is it purely a test-taking skill, since most of the time we don't have sets of potential answers to be eliminated in the texts that we read in graduate study? Is there a theory on what makes the ideal reading comprehension assessment? Is there a sense in which the preparation for this kind of test is supposed to carry over into future studies? I have no doubt that a great deal of thought goes into the development of these assessments. At the same time, I imagine that a great deal of insight could be gleaned on the part of test takers if they knew a little bit more about how these questions are created and the passages are sifted through and selected for the exam. These questions are among the most difficult to imitate for test-prep companies that develop their own non-official LSAT materials, which suggests how specific to the test developers is the process of developing and standardizing the assessment.

Give me some of your thoughts on the reading comprehension part of the LSAT or other tests. Did the studying for RC teach you something about reading in general, or did it simply teach you about reading for the LSAT? Did your major or personal reading habits influence the way you studied? Were you frustrated more with the trickiness of LSAT reading comprehension questions, or with timing, or with the passages themselves? Or were you impressed with the consistency of the reading part of the test, a consistency that perhaps suggests a comprehensive theory underlying the development of the assessment? Do you think you could go out and find a passage and come up with similar questions and answer choices on your own? I sometimes find that reverse-engineering approach to test studying valuable (come up with questions on your own, and you really know how these things work.) I myself would have a difficult time finding a suitable passage and imitating the types of questions that are asked. If I were to try to do so, I imagine you would immediately be able to find the difference between my fake passage and a real LSAT passage. I imagine that's the case with other test takers, who perhaps could come up with fake LR questions that could "blend in" with real LR questions.

Sorry this was long. But perhaps it could start a conversation about reading practices, and how those practices can be successfully captured in a 35 minute, 27-question, multiple-choice assessment. Foreigners find these kinds of tests Americans love strange, and I think for good reason.

Here's a link to an article written by LSAC (and published in an academic journal) that's very specific to many of the questions you asked/your curiosity about how LSAC designs/develops the reading comprehension sections and questions asked about the passages.

The article is not only a very interesting read from an inquisitive theoretical perspective, it's also a very informative and insightful article for test takers preparing for the LSAT since it describes in detail the various different 'reading skills' that the various different question types are designed to test, including descriptions of the hierarchy of difficulty of the various different reading skills and skill levels different questions and question types are designed to measure.

Use of Bloom’s Taxonomy in Developing Reading Comprehension Specifications:
http://www.testpublishers.org/assets/do ... l%2013.pdf

Here's a link to another LSAC scholarly article written for publication in an academic journal that gives a pretty detailed overview of the psychometrics and processes LSAC uses to write and pre-test test questions and to construct and assemble properly equated 'operational test forms' (meaning the four scored sections that comprise an official administered LSAT)

Developing and Assembling the Law School Admission Test:
http://www.math.washington.edu/~billey/ ... Week.3.pdf

I must warn/advise anybody that sees this that both articles are very sophisticated and not easy to read and fully comprehend without spending a lot of time studying/thinking about them and that a lot of the stuff in the second article is really hard to understand without a background in psychometrics and sophisticated statistical analytical methods.

The second article about developing and assembling the LSAT doesn't contain much if any information that can benefit/help people better prepare for the LSAT to improve their score, but is an interesting read for anybody curious about how each different LSAT test-form is constructed and equated to be standardized and comparable to all other LSAT's/PT's.

However, the first article about how they develop reading comprehension sections and questions contains a lot of information that can be very useful for helping improve ones reading comp performance if applied properly to adjust ones approach and reading style/priorities in order to be best prepared to handle the different question types asked. It really gives a good thorough 'behind the scenes' description of the methods. rhyme and reasons behind the seeming madness of the different challenging questions and question types asked about passages that can be very helpful for improving RC performance even though the article wasn't written to be a 'prep guide'.

wow! thank you !!! this is just what i was looking for!!!

conticuere wrote: Give me some of your thoughts on the reading comprehension part of the LSAT or other tests. Did the studying for RC teach you something about reading in general, or did it simply teach you about reading for the LSAT? Did your major or personal reading habits influence the way you studied? Were you frustrated more with the trickiness of LSAT reading comprehension questions, or with timing, or with the passages themselves? Or were you impressed with the consistency of the reading part of the test, a consistency that perhaps suggests a comprehensive theory underlying the development of the assessment? Do you think you could go out and find a passage and come up with similar questions and answer choices on your own? I sometimes find that reverse-engineering approach to test studying valuable (come up with questions on your own, and you really know how these things work.) I myself would have a difficult time finding a suitable passage and imitating the types of questions that are asked. If I were to try to do so, I imagine you would immediately be able to find the difference between my fake passage and a real LSAT passage. I imagine that's the case with other test takers, who perhaps could come up with fake LR questions that could "blend in" with real LR questions.

Sorry this was long. But perhaps it could start a conversation about reading practices, and how those practices can be successfully captured in a 35 minute, 27-question, multiple-choice assessment. Foreigners find these kinds of tests Americans love strange, and I think for good reason.

Very interesting questions. This is a topic I don't see brought up on this board very often. Those two articles could do a much better job of answering some of your more theoretical questions than I ever could, but I'll share some of my perspective as someone who's taught the LSAT for over 2 years.

I studied literature in college, focusing largely on modernist and post-modern works. So I spent a lot of time grappling with books that are highly symbolic/allegorical, deliberately obscure, and encouraging of multiple interpretations. When I started studying for the LSAT, RC was my weakest section, largely because I was used to maintaining an open mind about what a piece could be implying. There were plenty of times where it seemed to me that multiple answer choices could be correct, depending on how you chose to interpret the intentions of the author. I had to learn to approach LSAT passages differently than I might approach a novel - anything that wasn't backed up by strict textual evidence had to go, and each passage had to be reduced to one primary purpose, rather a series of possible ones.

So studying for the RC definitely did teach me something about reading. I think that the "trickiness" of RC questions was something that became easier to decode as I practiced more and more and learned to look out for certain phrases and implications. Timing was also something that required practice. The passages themselves became easier to reduce as I read more and more of them. So practice and an effective approach trumped the exam. But I definitely am impressed with the consistency of this section of the test. I feel like I might be able to modify a piece to turn it into an RC passage, but it would be pretty damn hard to pass it off as a real LSAT passage without a lot of practice in that regard.

Thanks Blueprint Mithun for your input. I too studied literature in college, and am in a Ph.D. program in literature. I am almost offended when I get RC questions wrong, because it's pretty much lifeblood. But I suppose literary texts are rather different from..."Intellectual property law states that..." or "Two studies of the evolution of brain size in primates suggest that..." I suppose it's a matter of mindset.

BY the way, I assume you teach for Blueprint? I am following their online course right now: I think it's excellent. It starts with difficult things right away (conditional logic---more than just explaining necessary/sufficient but diagramming conditional sentences: diagramming instantly turned conditional "match the flaw/parallel reasoning" questions into something more easily handled.)

Haha, I know what you mean by being almost offended. I couldn't believe that I wasn't acing Reading Comp, and I think I was initially resistant to changing my approach due to overconfidence. "I read Ulysses, you can't teach me anything, Mr.Prep-Guide author..." The way I see it, thinking about RC passages as arguments, and then learning to break those arguments down into their most important aspects is what mastering that section is really all about.

Yes, I do teach for them. I'm glad you're liking the online course! Conditional logic is one of the keys to the test, which is why we teach it in the first few lessons. Feel free to PM me if you come across any questions about the course/content down the line.

This is a great thread. The input has been very interesting so far.

I have been teaching test prep for more than 10 years. I also feel offended when I miss an RC question, but RC continues to be my weakest section on the LSAT.

I agree with Blueprint Mithum that the RC passages should be read like arguments. In fact, I encourage my students to think of RC passages as a series of linked arguments (i.e., paragraphs), and my students' job is to identify the conclusion of each paragraph individually as they would for each argument on the LR section.

I have some experience writing simulated exam questions for prep materials, and I'll have to admit that it is much more difficult than I would have imagined. Adapting the text and drafting questions are not that difficult. Writing correct answers is also relatively easy. But I struggled to come up with clever incorrect answers that were neither obviously incorrect nor ambiguous to the point of making them arguably correct.

I'm going to read the articles linked above when I have more time.

Jeffort: Do you know of any similar articles pertaining to the bar exam? Or the MBE in particular? You seem to have a good handle on the research behind the LSAT, and I'm trying to expand my expertise to bar prep. If you can share any papers or thoughts on that front, I would appreciate it.