Law School Predictor: The Thread"

(Applications Advice, Letters of Recommendation . . . )
User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 1:37 pm

..

User avatar
Bronte
Posts: 2128
Joined: Sun Jan 04, 2009 10:44 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby Bronte » Wed Jul 29, 2009 2:34 pm

YCrevolution wrote:Would you all be willing to forego having the % +/-, % YOU, % MID. fields if it meant having a feature that would say something like "X% of applicants matriculated with a score at/below yours" (this was discussed a few posts ago by PrayingforHYS, Bronte, and thesealocust)?


My $0.02:

I think % +/- is actually a very important field, considering that medians drive the rankings and are thus very important to adcomms. However, %YOU, %MID, FROM 25, FROM MID, and FROM 75 could probably go, considering that the latter three are in "index units," which are fairly arbitrary, especially to the average user. You're probably using %YOU and %MID to arrive at the % +/- calculations, but you could just do % +/- directly, something like:

Code: Select all

= ( YOUR INDEX - MED INDEX ) / PERFECT INDEX


Edit: clarity

User avatar
buddyblack
Posts: 60
Joined: Tue Jun 30, 2009 8:28 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby buddyblack » Wed Jul 29, 2009 3:09 pm

Bronte wrote:[However, %YOU, %MID, FROM 25, FROM MID, and FROM 75 could probably go, considering that the latter three are in "index units," which are fairly arbitrary, especially to the average user.


I agree with you on your point about clarity/user interface, but I would rather keep the FROM 25th/median/75th index scores and lose the percentages because then you can quickly see where your index falls in that range rather than just it's relation to the median.

And YC, a % of applicants matriculated might be interesting, but I always feel like a "% applicants matriculated at a score below yours" is sort of useless because a 2.2/144 and a 3.5/167 are lumped together.

User avatar
Bronte
Posts: 2128
Joined: Sun Jan 04, 2009 10:44 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby Bronte » Wed Jul 29, 2009 3:37 pm

buddyblack wrote:I agree with you on your point about clarity/user interface, but I would rather keep the FROM 25th/median/75th index scores and lose the percentages because then you can quickly see where your index falls in that range rather than just it's relation to the median.


Again, just IMO, percent over/under median is the most important factor in a predictor, like LSP, that uses matriculation percentiles as its database. Remember, 25th and 75th don't matter nearly as much as median to the schools themselves. Further, one can still easily see which quartile his index score falls in by comparing it to the 25/50/75 index scores.

YC, maybe it's worth a poll?

buddyblack wrote:And YC, a % of applicants matriculated might be interesting, but I always feel like a "% applicants matriculated at a score below yours" is sort of useless because a 2.2/144 and a 3.5/167 are lumped together.


This actually kind of runs contrary to what you're saying above, because a percentage matriculated is equivalent to an exact percentile. In other words, it shows you exactly (albeit via estimation/interpolation) where your index falls in the range. For example, 70% would be in the third quartile, closer to the 75th percentile.

Not trying to monopolize the discussion, just wanted to get those ideas out.

User avatar
gunners
Posts: 11
Joined: Thu May 28, 2009 11:14 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby gunners » Wed Jul 29, 2009 4:55 pm

CyLaw wrote:Yeah, I was going to write a scrapper to get all the data and convert it to a usable format for YCrevolution this weekend. Speaking of, YCrevolution what format do you prefer, I can do whatever, but I guessed that the one you would want was a csv file with username, lsat, gpa, law school, result. Correct? I can also give it to you as a database or whatever format you need.



I actually did some XML parsing into Excel as part of a research project this summer. I used java. Let me know if I can be of assistance. You too, YCrevolution.

pissantvache
Posts: 115
Joined: Thu Jul 23, 2009 2:37 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby pissantvache » Wed Jul 29, 2009 5:19 pm

YC, another way to cut down on the calculation time would be to create different applications for each tier (say, top 50, 51-100, 101-150, etc). Either that, or, alternatively, a landing page that would allow one to choose one's schools could be nice. As it is, I think that people usually have a sense of what schools they should look at, and it's really not valuable for me to be looking at a sea of auto-admits far below where I would ever consider applying and a sea of auto-rejects far above that point.

Bronte wrote:My $0.02:

I think % +/- is actually a very important field, considering that medians drive the rankings and are thus very important to adcomms. However, %YOU, %MID, FROM 25, FROM MID, and FROM 75 could probably go, considering that the latter three are in "index units," which are fairly arbitrary, especially to the average user. You're probably using %YOU and %MID to arrive at the % +/- calculations, but you could just do % +/- directly, something like:

Code:
= ( YOUR INDEX - MED INDEX ) / PERFECT INDEX


I wholeheartedly agree with Bronte on this point.

CyLaw
Posts: 1557
Joined: Thu Jul 17, 2008 1:59 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby CyLaw » Wed Jul 29, 2009 5:42 pm

gunners wrote:I actually did some XML parsing into Excel as part of a research project this summer. I used java. Let me know if I can be of assistance. You too, YCrevolution.


Thanks for the offer, but I actually finished it during lunch today, cause I was bored. :wink:

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 6:10 pm

..

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 6:22 pm

..

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 11:20 pm

..

CyLaw
Posts: 1557
Joined: Thu Jul 17, 2008 1:59 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby CyLaw » Wed Jul 29, 2009 11:23 pm

YCrevolution wrote:If anybody has some easy-to-implement suggestions for LSP (in addition to those noted above), now would be a good time to tell me, so I can try and incorporate them into a new version before law school starts.


Can you make it auto-email the various admission committees so they are aware that they should admit me according to LSP

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 11:28 pm

..

CyLaw
Posts: 1557
Joined: Thu Jul 17, 2008 1:59 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby CyLaw » Wed Jul 29, 2009 11:34 pm

YCrevolution wrote:I'll have it hack into their databases and auto-admit you everywhere you apply. If you could tell me how much scholarship money you'd like, I'll put that in, too.


For some reason, I don't believe you. 8)

I think that what you have now plus the previous suggestions are plenty. Last thing you want is too much feature creep with LSP, with it doing more then really needed for the general populace. It will just make it a pain when you want to maintain or evolve the software.

User avatar
gahthelaw
Posts: 728
Joined: Sun May 17, 2009 11:30 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby gahthelaw » Wed Jul 29, 2009 11:40 pm

I don't know if this has been mentioned and dismissed (a quick glance around the thread isnt turning up too much), but is there any way to edit for multiple LSAT takers? I'm not sure what the best way to do this would be except for dredging up a list of who takes the highest score, who definitely averages, and then some number crunching just based on stats (i absolutely defer to people who are better than i at this for a good way).

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby YCrevolution » Wed Jul 29, 2009 11:53 pm

..

User avatar
gahthelaw
Posts: 728
Joined: Sun May 17, 2009 11:30 pm

Re: Law School Predictor: v2.4 with Preliminary Accuracy Data

Postby gahthelaw » Thu Jul 30, 2009 12:09 am

YCrevolution wrote:
gahthelaw wrote:I don't know if this has been mentioned and dismissed (a quick glance around the thread isnt turning up too much), but is there any way to edit for multiple LSAT takers? I'm not sure what the best way to do this would be except for dredging up a list of who takes the highest score, who definitely averages, and then some number crunching just based on stats (i absolutely defer to people who are better than i at this for a good way).

The implementation wouldn't be that hard, but then somebody would still have to dredge up (and I'd have to enter) what schools average/take the highest score. I'll put it as a longer-term goal.

As a general rule, plug your highest LSAT into LSP, even for schools that claim to average. I've been told by other TLS users that it's more accurate if you do that.


Thanks for the info. I've been using my higher one (mostly to make myself feel better).

I actually have a spreadsheet somewhere around listing who (in the T20) averages, takes higher, or is ambiguous along with the actual wording from the schools and I think the numbers of a few confirmed multiple lsat takers from LSN for each school (yes, I geeked out a little about this). It's on my work computer but if you eventually ever want the information let me know and I'll update it a bit, format it in maybe a more helpful way, and send it over.

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: Accuracy Data Now Available

Postby YCrevolution » Thu Jul 30, 2009 8:58 pm

..

User avatar
SilverE2
Posts: 931
Joined: Wed Aug 20, 2008 10:04 pm

Re: Law School Predictor: Accuracy Data Now Available

Postby SilverE2 » Thu Jul 30, 2009 9:05 pm

YCrevolution wrote:Ladies and gentlemen, I give you the accuracy statistics for the Law School Predictor.

These results are based off of more than 33,500 LSN-listed law school admission decisions from the 2008-09 cycle for all ABA schools listed on both LSN and LSP. The version of LSP tested was Version 2.4 (current as of July 30, 2009). A big thanks goes out to CyLaw for compiling the data from LSN; I couldn't have done it without CyLaw.

The admit rate was calculated by taking the number of admitted applicants divided by total applicants for each respective prediciton category (total applicants includes admitted, waitlisted, and rejected applicants, but does not include those who status was listed as "pending"). The prediction categories used in testing accuracy are the same as LSP's prediction engine with the exception of early prediction (which was not tested); predictions are adjusted to account for splitter-ness, weak GPAs, and URM status.

Target admit rates, based on how LSP renders predictions:
When LSP said Admit: >= 87%
When LSP said Strong Consider: ~ 69%
When LSP said Consider: ~ 50%
When LSP said Weak Consider: ~ 31%
When LSP said Deny: <= 13%

Based on the decisions processed so far:
When LSP said Admit, 8402 instances: 87.0%, +0.0%
When LSP said Strong Consider, 4842 instances: 78.0%, +9.0%
When LSP said Consider, 10066 instances: 50.6%, +0.6%
When LSP said Weak Consider, 3646 instances: 22.8%, -8.2%
When LSP said Deny, 6754 instances: 10.0%, -3.0%

Difference between actual admit rate and predicted admit rate: (prediction categories weighted evenly)
Average rate: -0.3%
Median rate: -3.0%

Summary:
Prediction categories that were right on the money: Admit, Consider
Prediction categories that were reasonably close: Deny
Prediction categories that were somewhat off: Strong Consider and Weak Consider

A closer look...
A possible explanation for the Strong/Weak Consider results is that the chance of being admitted (based on admission index scores and LSP adjustments) should look like a parabolic curve (since it should be a normal distribution), and, based on these accuracy results, a curve with a high and narrow peak in the center (the roughly median applicant).

At any rate, I'll do some more number-crunching before schools starts, especially in regard to URM applicants and splitters, and compile a more official report that'll go up on LSP.


Was this done for part time as well?

*edit*

And Early Decision? Did you test it with that as well? Thanks!

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: Accuracy Data Now Available

Postby YCrevolution » Thu Jul 30, 2009 9:07 pm

..

User avatar
SilverE2
Posts: 931
Joined: Wed Aug 20, 2008 10:04 pm

Re: Law School Predictor: Accuracy Data Now Available

Postby SilverE2 » Thu Jul 30, 2009 9:12 pm

YCrevolution wrote:
SilverE2 wrote:
YCrevolution wrote:Ladies and gentlemen, I give you the accuracy statistics for the Law School Predictor.

These results are based off of more than 33,500 LSN-listed law school admission decisions from the 2008-09 cycle for all ABA schools listed on both LSN and LSP. The version of LSP tested was Version 2.4 (current as of July 30, 2009). A big thanks goes out to CyLaw for compiling the data from LSN; I couldn't have done it without CyLaw.

The admit rate was calculated by taking the number of admitted applicants divided by total applicants for each respective prediciton category (total applicants includes admitted, waitlisted, and rejected applicants, but does not include those who status was listed as "pending"). The prediction categories used in testing accuracy are the same as LSP's prediction engine with the exception of early prediction (which was not tested); predictions are adjusted to account for splitter-ness, weak GPAs, and URM status.

Target admit rates, based on how LSP renders predictions:
When LSP said Admit: >= 87%
When LSP said Strong Consider: ~ 69%
When LSP said Consider: ~ 50%
When LSP said Weak Consider: ~ 31%
When LSP said Deny: <= 13%

Based on the decisions processed so far:
When LSP said Admit, 8402 instances: 87.0%, +0.0%
When LSP said Strong Consider, 4842 instances: 78.0%, +9.0%
When LSP said Consider, 10066 instances: 50.6%, +0.6%
When LSP said Weak Consider, 3646 instances: 22.8%, -8.2%
When LSP said Deny, 6754 instances: 10.0%, -3.0%

Difference between actual admit rate and predicted admit rate: (prediction categories weighted evenly)
Average rate: -0.3%
Median rate: -3.0%

Summary:
Prediction categories that were right on the money: Admit, Consider
Prediction categories that were reasonably close: Deny
Prediction categories that were somewhat off: Strong Consider and Weak Consider

A closer look...
A possible explanation for the Strong/Weak Consider results is that the chance of being admitted (based on admission index scores and LSP adjustments) should look like a parabolic curve (since it should be a normal distribution), and, based on these accuracy results, a curve with a high and narrow peak in the center (the roughly median applicant).

At any rate, I'll do some more number-crunching before schools starts, especially in regard to URM applicants and splitters, and compile a more official report that'll go up on LSP.


Was this done for part time as well?

The LSN data included everyone (FT and PT), although I only tested the LSN data against the LSP FT predictor (I only had access to the things you'd see on a LSN graph; so no way to differentiate between ED, PT, scholarships, etc.).


Gotcha. Thanks!

User avatar
OperaSoprano
Posts: 4410
Joined: Tue Nov 25, 2008 1:54 am

Re: Law School Predictor: Accuracy Data Now Available

Postby OperaSoprano » Thu Jul 30, 2009 9:26 pm

I might be misreading the data, but it appears that the people who've self selected to report their numbers have done much better than they "should have." Are we all that much savvier than the average law school applicant?

In particular, I'm thinking of applicants who want to attend schools in their strong consider range. In my experience, if someone is able to demonstrate that he or she really wants in at a strong consider school, that person will very likely be admitted. This is what happened in my own cycle, and I've seen it happen over and over again.

Is there any way to account for "persistence" factors like visiting, interviewing, Why X letters, and LOCIs? IMO these can often trump softs, and at the end of the cycle, when WL decisions need to be made quickly, they can occasionally even trump numbers. Deadatheist's recent admission to Hastings (160/3.63, non URM) is a perfect example. She worked for this, probably even harder than I worked for Fordham, though LSP called her a weak consider. I believe her persistence must have been a contributing factor in her WL admission.

I'm not sure how you could quantify this, but do you think it's credited?

I mean, I'm sure you remember the dramatic mess my cycle was, and I was decently above the median.

User avatar
YCrevolution
Posts: 4714
Joined: Sat Oct 25, 2008 1:25 am

Re: Law School Predictor: Accuracy Data Now Available

Postby YCrevolution » Thu Jul 30, 2009 9:46 pm

..

User avatar
jackassjim
Posts: 185
Joined: Sun Dec 07, 2008 10:58 pm

Re: Law School Predictor: Accuracy Data Now Available

Postby jackassjim » Thu Jul 30, 2009 10:32 pm

I know nothing about forecasting, but I can't help but feel that your results are just too good to be true. I'm not sure if it's standard procedure to test the predictive validity of a model on the very data that was used to compute. It just all seems "over-fitted" to my taste. You would probably need to first compute a model using data for a specific cycle, and then test to see if it is helpful in predicting admissions results in the second cycle. (Please tell me if I'm way off here).In essence, this would mean that you can't test the predictive power of your model using the same data you used to build it.

What you can do, however, is check how much of the variance your different models can explain. That would require you to report a statistic like a pseudo-R2. Maybe the average pseudo-R2 for all the models you calculated for each school?

In any case, I was wondering: What kind of model is this anyway? A simple logit or probit? Are you interacting variables, or is it simply additive?

CyLaw
Posts: 1557
Joined: Thu Jul 17, 2008 1:59 pm

Re: Law School Predictor: Accuracy Data Now Available

Postby CyLaw » Thu Jul 30, 2009 10:34 pm

I don't believe that the data model for LSP was built with the LSN data, as YC did not have that data when he created the model. The LSN data was only recently used to test the LSP Model. I believe.

User avatar
jackassjim
Posts: 185
Joined: Sun Dec 07, 2008 10:58 pm

Re: Law School Predictor: Accuracy Data Now Available

Postby jackassjim » Thu Jul 30, 2009 10:39 pm

CyLaw wrote:I don't believe that the data model for LSP was built with the LSN data, as YC did not have that data when he created the model. The LSN data was only recently used to test the LSP Model. I believe.


Good catch. I really dropped the ball on that one.




Return to “Law School Admissions Forum”

Who is online

Users browsing this forum: echonov, urbanist11, vivehodie, Yahoo [Bot] and 6 guests