Do retakes matter? Forum

[sublime]

..

[LSATSCORES2012]

This was done for each school in the T-14 considering applicants between 2009 and 2013

It is possible also, with the decline in applicants, that more schools are forced to not care and cannot be as picky as they were a few years ago.

This is really bizarre...

When I change to just the 2011-2012 cycle, nearly EVERY T14 school seems to very heavily prefer retakers - even when controlling for LSAT.

I've checked for errors in my code quite a few times because this is so bizarre, and even hand checked one of the results, and just can't find any problems.

That is really strange. Although I hope it is true as a potential retaker!

Any possible explanations that you can think of? If you just controlled for LSAT, maybe the GPA is causing something, as the higher your GPA, the more advantageous and perhaps, more likely, a person is to retake? I have no idea.

Possible explanation: I f'ed something up.

Curious to see what squeebz finds...

[LSATSCORES2012]

Okay, the problem was that I forgot to apply the sample size limits that I had earlier. When I apply them the numbers start to look sane (but similar to the full 2009-2013 numbers). Interested to hopefully have squeebz (and maybe elterrible78?) weigh in. (If anyone else would like the raw data PM me and I'll send it over.)

[LSATSCORES2012]

(Irrelevant) interesting discovery just now...

about 4% of LSN apps seem to be showing the wrong LSAT score.

This user, for example, seems to have reported receiving a 120 and a 169, but his profile shows 144 (I bet it was actually 144 -> 169). (Discovered by elterrible78, I should add.)

ETA: and elterrible78 - a mad genius - is going to come in and correct my numbers now

[LSATSCORES2012]

Updated my original post with pretty graphs that you can interpret. I think we'll also be graced by elterrible78 presenting his genius-level analysis (he's freaking amazing...) and perhaps squeebz (who seems to know much more about stats than I )

From the graphs, NYU is the only school that looks to me like it clearly and methodically considers previous scores. Elterrible78 is saying that his analysis shows that they care quite a lot as well.

[rebexness]

Have you done much considering how many people skew their posted data so that they don't get outed?

[LSATSCORES2012]

Have you done much considering how many people skew their posted data so that they don't get outed?

I've thought about it before but I don't think there's much of a way to correct for it so I kind of pretend it doesn't happen... besides, I think most of the skewing should cancel out.

[elterrible78]

Have you done much considering how many people skew their posted data so that they don't get outed?

I've thought about it before but I don't think there's much of a way to correct for it so I kind of pretend it doesn't happen... besides, I think most of the skewing should cancel out.

When I did this, I eliminated anything that was obviously insane from the data. As far as people who skew it, as long as they're not skewing it too much, it's not going to be a big deal with this sample size, especially considering that the skewers are probably a pretty small minority.

Quick and dirty:

Controlling for LSAT, GPA, URM, ED (where applicable), and how early the app was sent in, these are the results:

Boalt - Retakers are 19% less likely to be admitted.
Cornell - Retakers are 26% less likely to be admitted.
Harvard - Retakers are 31% less likely to be admitted.
NYU - Retakers are 62% less likely to be admitted (not a typo).
Stanford - Retakers are 40% less likely to be admitted.

It doesn't matter a bit at the rest of the T14.

All the usual LSN-related caveats apply to this.

[rebexness]

I still think the nature of the retake has more to do with it that can be accounted for.

[elterrible78]

I still think the nature of the retake has more to do with it that can be accounted for.

The nature of the retake is clearly going to play a role for an individual student, there's no doubt about this. But a lot of these retakers are applying across the T14, so if it was all just about the nature of retakes, you wouldn't see massive differences in the way schools deal with retakers (with retaker simply meaning anyone who has taken more than 1 LSAT). It's abundantly clear from the data that some schools penalize retakers, period, and some couldn't care less.

[sublime]

..

[elterrible78]

Have you done much considering how many people skew their posted data so that they don't get outed?

I've thought about it before but I don't think there's much of a way to correct for it so I kind of pretend it doesn't happen... besides, I think most of the skewing should cancel out.

When I did this, I eliminated anything that was obviously insane from the data. As far as people who skew it, as long as they're not skewing it too much, it's not going to be a big deal with this sample size, especially considering that the skewers are probably a pretty small minority.

Quick and dirty:

Controlling for LSAT, GPA, URM, ED (where applicable), and how early the app was sent in, these are the results:

Boalt - Retakers are 19% less likely to be admitted.
Cornell - Retakers are 26% less likely to be admitted.
Harvard - Retakers are 31% less likely to be admitted.
NYU - Retakers are 62% less likely to be admitted (not a typo).
Stanford - Retakers are 40% less likely to be admitted.

It doesn't matter a bit at the rest of the T14.

All the usual LSN-related caveats apply to this.

Fuck.

Do you think that those lower likelihoods are mostly from borderline applicants and as long as you hit above median you should be fine as long as your gpa is ok, or...

Above median GPA?

[elterrible78]

It also depends on the number and range of retakes.

For example, a 158, 172 will be looked at differently than a 165,16,170

At least, I would guess so.

Actually, just looking at the retakers at NYU, since that was the most extreme case, the "spread" between the highest and lowest GPA makes no difference whatsoever, statistically. That's surprising to me, I guess. I'll do it for Stanford, Boalt, etc., sometime when it isn't 3 a.m. and I have to work the next day.

[sublime]

..

[squeebz]

I ran my probit model, and I got results that more closely resemble standard TLS wisdom.

Schools w/ No Statistical Difference Between Retakers and Non-Retakes - Yale, Stanford, Columbia, Penn, Berkeley, Michigan, Duke, and Cornell.

Schools Where Retakes Faired Worse - Harvard, Chicago, NYU, and UVA

Schools Where Retakers Faired Better - Northwestern and GULC

Again, a lot of LSATSCORES2012 caveats still apply. I didn't check the accuracy of the underlying data so the errors pointed out by elterrible78 are not corrected. I just personally think a probit regression is more appropriate for testing this type of data.

For schools where the retakers faired worse, an increase in LSAT score of a point or two often offset the disadvantage of retaking. For example, the probability of getting accepted at UVA is higher if a person retook and scored a point higher than if they applied with the lower score. So retaking until geting a score you're satisfied with still appears to best practices.

[Clearly]

What do we know about big score differences? I have a 12 point differential between my scores.

[sublime]

..

[Clearly]

What do we know about big score differences? I have a 12 point differential between my scores.

Congrats though!

Thanks. I'd certainly rather have the jump than not, but I do worry about the effect it'll have on my cycle.

[elterrible78]

I ran my probit model, and I got results that more closely resemble standard TLS wisdom.

Schools w/ No Statistical Difference Between Retakers and Non-Retakes - Yale, Stanford, Columbia, Penn, Berkeley, Michigan, Duke, and Cornell.

Schools Where Retakes Faired Worse - Harvard, Chicago, NYU, and UVA

Schools Where Retakers Faired Better - Northwestern and GULC

Again, a lot of LSATSCORES2012 caveats still apply. I didn't check the accuracy of the underlying data so the errors pointed out by elterrible78 are not corrected. I just personally think a probit regression is more appropriate for testing this type of data.

For schools where the retakers faired worse, an increase in LSAT score of a point or two often offset the disadvantage of retaking. For example, the probability of getting accepted at UVA is higher if a person retook and scored a point higher than if they applied with the lower score. So retaking until geting a score you're satisfied with still appears to best practices.

Weird. What was your actual model for this, and did you look only at the cycles since the '06/'07? Also, if you got the dataset from LSATSCORES2012, did you get rid of the 9000 or so bad datapoints? I'm curious, because although I originally ran logistic regressions (just because I prefer dealing with log odds as results), but when I do probit I get exactly the same results I get before, in terms of what is statistically significant and the sign of the coefficient (although the coefficients themselves did vary marginally).

EDIT: From Wikipedia (the authoritative source on everything), which explains exactly why I used logit instead, ha!

Logit versus probit [edit]
The Probit model has been around longer than the logit model. They behave similarly, except that the logistic distribution tends to be slightly flatter tailed. One of the reasons the logit model was formulated was that the probit model was computationally difficult due to the requirement of numerically calculating integrals. Modern computing however has made this computation fairly simple. The coefficients obtained from the logit and probit model are fairly close. However, the odds ratio is easier to interpret in the logit model.
Practical reasons for choosing the probit model over the logistic model would be:
There is a strong belief that the underlying distribution is normal
The actual event is not a binary outcome (e.g., bankruptcy status) but a proportion (e.g., proportion of population at different debt levels).

[elterrible78]

What do we know about big score differences? I have a 12 point differential between my scores.

Hey...I have only done this for NYU so far (will do the rest later this afternoon), but at least for them, the difference between your low and high didn't seem to really matter at all, surprisingly. It's must more about your highest score.

[squeebz]

I ran my probit model, and I got results that more closely resemble standard TLS wisdom.

Schools w/ No Statistical Difference Between Retakers and Non-Retakes - Yale, Stanford, Columbia, Penn, Berkeley, Michigan, Duke, and Cornell.

Schools Where Retakes Faired Worse - Harvard, Chicago, NYU, and UVA

Schools Where Retakers Faired Better - Northwestern and GULC

Again, a lot of LSATSCORES2012 caveats still apply. I didn't check the accuracy of the underlying data so the errors pointed out by elterrible78 are not corrected. I just personally think a probit regression is more appropriate for testing this type of data.

For schools where the retakers faired worse, an increase in LSAT score of a point or two often offset the disadvantage of retaking. For example, the probability of getting accepted at UVA is higher if a person retook and scored a point higher than if they applied with the lower score. So retaking until geting a score you're satisfied with still appears to best practices.

Weird. What was your actual model for this, and did you look only at the cycles since the '06/'07? Also, if you got the dataset from LSATSCORES2012, did you get rid of the 9000 or so bad datapoints? I'm curious, because although I originally ran logistic regressions (just because I prefer dealing with log odds as results), but when I do probit I get exactly the same results I get before, in terms of what is statistically significant and the sign of the coefficient (although the coefficients themselves did vary marginally).

EDIT: From Wikipedia (the authoritative source on everything), which explains exactly why I used logit instead, ha!

Logit versus probit [edit]
The Probit model has been around longer than the logit model. They behave similarly, except that the logistic distribution tends to be slightly flatter tailed. One of the reasons the logit model was formulated was that the probit model was computationally difficult due to the requirement of numerically calculating integrals. Modern computing however has made this computation fairly simple. The coefficients obtained from the logit and probit model are fairly close. However, the odds ratio is easier to interpret in the logit model.
Practical reasons for choosing the probit model over the logistic model would be:
There is a strong belief that the underlying distribution is normal
The actual event is not a binary outcome (e.g., bankruptcy status) but a proportion (e.g., proportion of population at different debt levels).

In a dataset this size you're not going to see much difference between logit and probit. I just had to derive the MLE estimates for the probit model, so I think my gut reaction is to always do that one. I'll rerun, but I doubt things will change significantly.

I took LSAT2012SCORES data and imported it into R. Converted GPA and LSAT to numeric and the rest to character variables. I dropped all observations where status was equal to Pending. I think created a indicator variable called Decision which was equal to 1 if the person was accepted or waitlisted and then accepted and 0 for all other observations. I also created an indicator variable called retaker for if the person retook the exam or not (if lsat2 and lsat3 were both zero). Then for each school I subsetted the data to that school and the regressed GPA, highest LSAT, and retaker on decision. I used the sign and significance of the retaker variable to determine the effect of retaking on decision.

[BigZuck]

ITT: nerds

Love it though, thanks for the contributions guys

[squeebz]

Weird. What was your actual model for this, and did you look only at the cycles since the '06/'07? Also, if you got the dataset from LSATSCORES2012, did you get rid of the 9000 or so bad datapoints? I'm curious, because although I originally ran logistic regressions (just because I prefer dealing with log odds as results), but when I do probit I get exactly the same results I get before, in terms of what is statistically significant and the sign of the coefficient (although the coefficients themselves did vary marginally).

Good catch. I was going back waaaaay too far with the data. I didn't even think to check how far back the data went. When I restricted the data to the 09-10 cycle and after, I got results more in line with yours. Also, I added a logit regression to my loop and the results didn't change.

Schools w/ No Statistical Difference Between Retakers and Non-Retakers - Penn, GULC, Northwestern, UVA, Yale, Michigan, and Duke

Schools Where Retakers Faired Worse - Columbia, Berkeley, Cornell, NYU, Harvard, Stanford, and Chicago

[LSATSCORES2012]

Weird. What was your actual model for this, and did you look only at the cycles since the '06/'07? Also, if you got the dataset from LSATSCORES2012, did you get rid of the 9000 or so bad datapoints? I'm curious, because although I originally ran logistic regressions (just because I prefer dealing with log odds as results), but when I do probit I get exactly the same results I get before, in terms of what is statistically significant and the sign of the coefficient (although the coefficients themselves did vary marginally).

Good catch. I was going back waaaaay too far with the data. I didn't even think to check how far back the data went. When I restricted the data to the 09-10 cycle and after, I got results more in line with yours. Also, I added a logit regression to my loop and the results didn't change.

Schools w/ No Statistical Difference Between Retakers and Non-Retakers - Penn, GULC, Northwestern, UVA, Yale, Michigan, and Duke

Schools Where Retakers Faired Worse - Columbia, Berkeley, Cornell, NYU, Harvard, Stanford, and Chicago

Also, elterrible ignored waitlists, and the data points he's referring to are the one's where LSN was reporting the wrong LSAT score or for which no score was reported.

Glad to see your analyses both agree, will quote both in OP. Thanks both of you!

[elterrible78]

In a dataset this size you're not going to see much difference between logit and probit. I just had to derive the MLE estimates for the probit model, so I think my gut reaction is to always do that one. I'll rerun, but I doubt things will change significantly.

I took LSAT2012SCORES data and imported it into R. Converted GPA and LSAT to numeric and the rest to character variables. I dropped all observations where status was equal to Pending. I think created a indicator variable called Decision which was equal to 1 if the person was accepted or waitlisted and then accepted and 0 for all other observations. I also created an indicator variable called retaker for if the person retook the exam or not (if lsat2 and lsat3 were both zero). Then for each school I subsetted the data to that school and the regressed GPA, highest LSAT, and retaker on decision. I used the sign and significance of the retaker variable to determine the effect of retaking on decision.

Yeah, as LSATSCORES said, I dropped WLs as well, because there's just too much uncertainty involved there. I did the same exact thing re: retaker and decision, subsetted data to the school, but I also included URM and the month the app was submitted in the model - and ED, if applicable-, which add some explanatory power to the model. I bet if you did that, we'd come out just about the same. Good to know there are more than a couple of us stats nerds around here, as well as people who are interested in what the bark of a tree is made out of