Do retakes matter? Forum

[LSATSCORES2012]

I know TLS wisdom suggests that retakes don't negatively affect one's application, and last night I figured out how to access retaker information on LSN and thought I'd test that hypothesis for the T14.

Edited: Analysis from elterrible and squeebz, stats geniuses - thanks to both of you for doing this

WL included, 2009-2010, probit+logit regression

Schools w/ No Statistical Difference Between Retakers and Non-Retakers - Penn, GULC, Northwestern, UVA, Yale, Michigan, and Duke

Schools Where Retakers Faired Worse - Columbia, Berkeley, Cornell, NYU, Harvard, Stanford, and Chicago

2006-2013, WL excluded, bad LSN data points excluded, logit regression

Quick and dirty:

Controlling for LSAT, GPA, URM, ED (where applicable), and how early the app was sent in, these are the results:

Boalt - Retakers are 19% less likely to be admitted.
Cornell - Retakers are 26% less likely to be admitted.
Harvard - Retakers are 31% less likely to be admitted.
NYU - Retakers are 62% less likely to be admitted (not a typo).
Stanford - Retakers are 40% less likely to be admitted.

It doesn't matter a bit at the rest of the T14.

All the usual LSN-related caveats apply to this.

Edited: Instead of listing my conclusions, I'm providing images for each of the T14 which help you to reach your own conclusions on the significances of any difference. In these graphs, red indicates that the acceptance rate for retakers is lower; yellow indicates no difference; and green indicates that retakers have a greater acceptance rate. Note that yellow in the lower part of the graph is possibly attributable to the fact that no one in that range is getting in. (data 2008-2012)

--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--
--ImageRemoved--

[Dr. Dre]

[NoWorries]

Damn son. Impressive work.

[062914123]

.

[Vexed]

For your categories of schools that consider or probably consider, is it in favor or against retakers? Is there anyway to control for the location of the higher score (IE, if you end up scoring lower on a single retake, does that lend to a poorer cycle at those schools than someone who scores higher the second time around?)

[jrsbaseball5]

Excellent work as always. Though when I saw 2 tailed t-test I cringed a little at the repressed memories

[rebexness]

I would disagree about Harvard.

[sublime]

..

[bowser]

I have no idea how your statistics work. But FWIW your list is very counter-TLS wisdom (which is generally, the higher up the rankings you go the more likely a school is to consider retakes, and only HYS really even care a little bit).

[rebexness]

It also depends on the number and range of retakes.

For example, a 158, 172 will be looked at differently than a 165,16,170

At least, I would guess so.

[Balthy]

I have no idea how your statistics work. But FWIW your list is very counter-TLS wisdom (which is generally, the higher up the rankings you go the more likely a school is to consider retakes, and only HYS really even care a little bit).

I agree. Also, NU explicitly states on their website that they only consider your top score. If their official policy was that they consider all your scores in a "holistic" manner, I could believe that this was bullshit and they really only care about keeping their medians up (and so only care about your top score). However, NU stating themselves that they only consider top score makes me think something is wrong with OPs placement of NU.

[LSATSCORES2012]

So, as I said in the first post, there are a lot of potential flaws. What I've done is reported what the numbers say, but numbers can be misleading. For Northwestern, for instance, about 70% of the data points that I sampled showed that non-retakers, or retakers without a very low first score or retake score, had a better chance of being admitted than those who retook and ended up with their highest LSAT score being three points or more greater than their lowest.

Could it be random statistical fluctuation? Perhaps, but that's what the t-test was designed to exclude (though it can't ever exclude it entirely, just say there's a very small chance that the difference is due to random chance as opposed to another factor).

Then again, that other factor can be many things. I provided an example for Chicago, but other reasons could also apply to Northwestern. Say, for example, it is the case (hypothetically) that applicants who have been out of school for a few years are slightly less likely to retake the LSAT, instead performing well on their first try (maybe because they don't have the time to retake or because of the skills they have developed in the workforce or for whatever reason). That would explain Northwestern's results, too, because Northwestern is a school that is known for preferring candidates with work experience, and if the retakes are disproportionally being done by candidates without work experience then it will seem as though they are discriminating against retakes.

Of course, this is one possibility. Another is that Northwestern is lying, though that seems unlikely because it's not something that seems like it would benefit them.

One more thing I'd like to add: I don't think any of this should discourage anyone from retaking... I mostly posted it to put more information out there and question this piece of TLS wisdom. I definitely think retaking is good in most situations (the exception being if you've already scored very high).

[LSATSCORES2012]

For your categories of schools that consider or probably consider, is it in favor or against retakers? Is there anyway to control for the location of the higher score (IE, if you end up scoring lower on a single retake, does that lend to a poorer cycle at those schools than someone who scores higher the second time around?)

They we all in favor of non-retakers, except where I specified otherwise. (I think Berkeley, Yale, and UVA were the only ones in favor of retakers, and those ever so slightly (results were not statistically significant)).

I'll see what I can do about your second question...

[LSATSCORES2012]

To eliminate the complicated statistics (and because I think an ANOVA test might have actually been more appropriate), these numbers are the percentage of the data points I looked at where the "non-retakers" had a greater acceptance rate than the retakers (note that this does not discriminate between a 1% difference and a 50% difference).

32%: Yale
40%: Berkeley
45%: Stanford
49%: UVA
53%: Harvard
55%: Michigan
55%: Chicago
58%: Penn
60%: Georgetown
65%: Columbia
70%: Northwestern
72%: Duke
79%: Cornell
83%: NYU

Also, keep in mind that this data is from 2009-2013, and IIRC NYU only recently changed their policy on how they treat retakers.

[sinfiery]

Wow, did not expect to see those results. Awesome, thanks for doing this.

[Balthy]

Didn't want to detract from this at all, just my initial thought after seeing NU. Still great info, especially if understood with the caveats you detailed, and a great contribution. Thanks!

[LSATSCORES2012]

Didn't want to detract from this at all, just my initial thought after seeing NU. Still great info, especially if understood with the caveats you detailed, and a great contribution. Thanks!

No, I didn't take it that way... sorry if I came off as overly defensive. I definitely think you raise a great point. It IS puzzling why NU says this one thing, when you would expect them to try to publicly push in the other direction, and at first glance they actually do seem to do it the other way.

I think it's interesting, so thank you for pointing that out, largely because it helps to show just how misleading the data can be. Surely, I would say, it's not the case that Northwestern is actually looking at the average score or considering both scores when they say otherwise and they would benefit from not saying otherwise. So there must be some other factor contributing to that difference.

What REALLY surprises me is how Yale seems to prefer people who have taken the LSAT more than once.

[BigZuck]

Fascinating.

In my case, Duke, Cornell, Northwestern, and Michigan almost certainly did not care about my other scores and only looked at my highest. And Berkeley almost certainly did not care about the fact that I retook up into their 75th (nor the fact that I was a reapplicant. Seems like they could care less about my dedication. Or they just thought I wasn't pretty enough).

Nice work!

[supernma]

I think the consensus seems to be that policies on retakes are fairly school-specific (NYU averages multiple scores, Columbia takes only the highest of multiple scores, etc). Even so, it's best to only take once if at all possible. Retakes can definitely be warranted depending on the circumstances though.

[sublime]

..

[squeebz]

To eliminate the complicated statistics (and because I think an ANOVA test might have actually been more appropriate), these numbers are the percentage of the data points I looked at where the "non-retakers" had a greater acceptance rate than the retakers (note that this does not discriminate between a 1% difference and a 50% difference).

32%: Yale
40%: Berkeley
45%: Stanford
49%: UVA
53%: Harvard
55%: Michigan
55%: Chicago
58%: Penn
60%: Georgetown
65%: Columbia
70%: Northwestern
72%: Duke
79%: Cornell
83%: NYU

Also, keep in mind that this data is from 2009-2013, and IIRC NYU only recently changed their policy on how they treat retakers.

Did you consider trying a probit regression? You could regress gpa, highest LSAT, and an indicator variable on retakers on outcome. You could then look at the sign and significance level of the indicator variable. That way you could avoid all the messy within .05 GPA stuff and calculating acceptance rates.

Just some thoughts from a fellow stats nerd.

[LSATSCORES2012]

This was done for each school in the T-14 considering applicants between 2009 and 2013

It is possible also, with the decline in applicants, that more schools are forced to not care and cannot be as picky as they were a few years ago.

This is really bizarre...

When I change to just the 2011-2012 cycle, nearly EVERY T14 school seems to very heavily prefer retakers - even when controlling for LSAT.

I've checked for errors in my code quite a few times because this is so bizarre, and even hand checked one of the results, and just can't find any problems.

[LSATSCORES2012]

To eliminate the complicated statistics (and because I think an ANOVA test might have actually been more appropriate), these numbers are the percentage of the data points I looked at where the "non-retakers" had a greater acceptance rate than the retakers (note that this does not discriminate between a 1% difference and a 50% difference).

32%: Yale
40%: Berkeley
45%: Stanford
49%: UVA
53%: Harvard
55%: Michigan
55%: Chicago
58%: Penn
60%: Georgetown
65%: Columbia
70%: Northwestern
72%: Duke
79%: Cornell
83%: NYU

Also, keep in mind that this data is from 2009-2013, and IIRC NYU only recently changed their policy on how they treat retakers.

Did you consider trying a probit regression? You could regress gpa, highest LSAT, and an indicator variable on retakers on outcome. You could then look at the sign and significance level of the indicator variable. That way you could avoid all the messy within .05 GPA stuff and calculating acceptance rates.

Just some thoughts from a fellow stats nerd.

I don't even know what that means, lol.

Could I send you the raw data and you can perform your own analysis?

[Balthy]

Didn't want to detract from this at all, just my initial thought after seeing NU. Still great info, especially if understood with the caveats you detailed, and a great contribution. Thanks!

No, I didn't take it that way... sorry if I came off as overly defensive. I definitely think you raise a great point. It IS puzzling why NU says this one thing, when you would expect them to try to publicly push in the other direction, and at first glance they actually do seem to do it the other way.

I think it's interesting, so thank you for pointing that out, largely because it helps to show just how misleading the data can be. Surely, I would say, it's not the case that Northwestern is actually looking at the average score or considering both scores when they say otherwise and they would benefit from not saying otherwise. So there must be some other factor contributing to that difference.

What REALLY surprises me is how Yale seems to prefer people who have taken the LSAT more than once.

Awww, I feel like we belong in a Canadian law school

[squeebz]

To eliminate the complicated statistics (and because I think an ANOVA test might have actually been more appropriate), these numbers are the percentage of the data points I looked at where the "non-retakers" had a greater acceptance rate than the retakers (note that this does not discriminate between a 1% difference and a 50% difference).

32%: Yale
40%: Berkeley
45%: Stanford
49%: UVA
53%: Harvard
55%: Michigan
55%: Chicago
58%: Penn
60%: Georgetown
65%: Columbia
70%: Northwestern
72%: Duke
79%: Cornell
83%: NYU

Also, keep in mind that this data is from 2009-2013, and IIRC NYU only recently changed their policy on how they treat retakers.

Did you consider trying a probit regression? You could regress gpa, highest LSAT, and an indicator variable on retakers on outcome. You could then look at the sign and significance level of the indicator variable. That way you could avoid all the messy within .05 GPA stuff and calculating acceptance rates.

Just some thoughts from a fellow stats nerd.

I don't even know what that means, lol.

Could I send you the raw data and you can perform your own analysis?

Probit regression is a type of regression things that occur in binary outcomes (admit/reject). The math to calculate the coefficients is pretty intense, but the code is fairly straightforward to run in R or Stata.

PM and I'll try to give it a run. My coding skills are a tad rusty though.

Just an FYI when then p-value is really high it doesn't mean schools prefer retakes. It just means that there is no difference on admissions decisions between retaking and not.