LST Score Reports (Asking TLSers to test beta product) Forum

[jenesaislaw]

How would you explain the results of a state like California to an uninitiated applicant?

Working through the specifics mentally now. A big reason we've released a beta is to see how people use it and what questions they ask.

Generally, you start by seeing who places in the state by merely being on the report. If a school is not on the report, and you want to work in California, then you probably do not want to attend that school. (One exception is if a school did not provide enough public data to make it on. Very few of these schools, though. 15 schools or so are missing from 1 state report.)

Then you need to think about what you want out of your degree. Are you interested in a big firm? Small law? Public service? Starting a legal career at all?

What schools can meet those needs? What is it going to cost me to attend there? Can I get in?

[Tiago Splitter]

I see. I think a big problem is the default to the overall employment score and our natural desire to order everything nicely into a ranking.

[jenesaislaw]

I see. I think a big problem is the default to the overall employment score and our natural desire to order everything nicely into a ranking.

Indeed. May be worthwhile to make the default sort be by the percentage in the state. The downside there, of course, is making it seem like being local is more valuable than helping people begin a legal career.

[Mroberts3]

Also, your response to the Mercer comment is a non-response. Almost 50% of Mercer graduates go to firms of sizes 2-10 people large... common sense comes from what we know to be good employment outcomes vs. bad employment outcomes.

Right, my point about Mercer was that some jobs are objectively harder to get than others. I don't want to do biglaw, but the point is that people that do it (and clerkships/prestigious public interest) have better opportunities a priori. Why? Because big firms are undeniably more selective than small firms. Just look at the GPA cutoffs at any school's OCI.

If all big firm jobs vanished tomorrow and all big firm lawyers just retired (leaving only new grads in the employment market) what do you think would happen? Would Mercer still place all of its students into small firms while USC/UCLA students starve? No, USC/UCLA students would take those jobs and Mercer students would starve. The fact is that people choose to take biglaw over small law because it pays better and has better exit options. (This hypothetical would probably make more sense if we used Loyola LA for geographical reasons, but whatever).

Not all jobs are like this. I happen to know that local district attorneys offices and the JAG corps don't really care about school rank or grades as much (to my chagrin).

I get what you are trying to do, but just because I disagree doesn't mean I'm just regurgitating the US News spiel. The "best" job for anyone is one that pays their loans and makes them happy. But because we are talking about people as such and not one person, it doesn't make sense to treat easier to get, lower paying jobs as fungible with more difficult positions.

[dingbat]

begin a legal career.

I love the assumption that a bunch of grads banding together to start a small firm (partnership) falls under "begin a legal career" while a solo practitioner does not. How many of those small firms don't make it a year? Compare that to those who decide they'd rather get a non-JD required job that pays better.

Basically, you say you're not ranking, yet the schools are clearly ranked (by employment score)

You're stuck with a two-pronged problem:
1) On one hand, a lot of people are lazy, so they'll go to the website, click their state, and assume the top school is the best and the bottom the worst, end of story.
2) On the other hand, if you got someone who's more knowledgeable, they'll look at the ranking, realize that it doesn't make a lot of sense (because of the way it's presented), figure there's something circumspect about your site and your credibility is shot.

I doubt many people are going to look for the rationale and explanation if it's not listed front and center - I'd wager most of the people on this site who are beta-testing for you are fairly knowledgeable and have done a chunk of research, and we're getting confused - what do you think an uninitiated person with limited exposure will think?

You need to highlight in big letters the main issue (ranks JD employment only) and key issues that need further exploration. I suggest bullet points (just a few, such as "type of employment") to draw the reader's attention before they even get to the rankinglist

[SuperCerealBrah]

begin a legal career.

I love the assumption that a bunch of grads banding together to start a small firm (partnership) falls under "begin a legal career" while a solo practitioner does not. How many of those small firms don't make it a year? Compare that to those who decide they'd rather get a non-JD required job that pays better.

Basically, you say you're not ranking, yet the schools are clearly ranked (by employment score)

You're stuck with a two-pronged problem:
1) On one hand, a lot of people are lazy, so they'll go to the website, click their state, and assume the top school is the best and the bottom the worst, end of story.
2) On the other hand, if you got someone who's more knowledgeable, they'll look at the ranking, realize that it doesn't make a lot of sense (because of the way it's presented), figure there's something circumspect about your site and your credibility is shot.

I doubt many people are going to look for the rationale and explanation if it's not listed front and center - I'd wager most of the people on this site who are beta-testing for you are fairly knowledgeable and have done a chunk of research, and we're getting confused - what do you think an uninitiated person with limited exposure will think?

You need to highlight in big letters the main issue (ranks JD employment only) and key issues that need further exploration. I suggest bullet points (just a few, such as "type of employment") to draw the reader's attention before they even get to the rankinglist

You are also making quite a leap by just assuming all the 2-10 firms people are grads banding together after graduating. There is absolutely no data on this and you are just going by what you think is likely rather than any hard data suggesting this is the case. Not to mention that the huge majority of lawyers in this country work in firms around this size. There very well may be grads starting a firm together and being put into the 2-10 category. However, you can't just jump to "well, obviously this is the case for them" without any real data to back up your claims.

[jenesaislaw]

You are also making quite a leap by just assuming all the 2-10 firms people are grads banding together after graduating. There is absolutely no data on this and you are just going by what you think is likely rather than any hard data suggesting this is the case. Not to mention that the huge majority of lawyers in this country work in firms around this size. There very well may be grads starting a firm together and being put into the 2-10 category. However, you can't just jump to "well, obviously this is the case for them" without any real data to back up your claims.

This is a great post. Especially the bolded and the underlined.

And just as an example, we actually can figure out the maximum number of people in 2-10 attorney firms when we have a school's full NALP report because of the job source chart. Among the options for job source on the survey is "started own firm or business." On the NALP report, this is clumped in the "All Other" category.

For Akron, --LinkRemoved--, there were 33 people in firms of 50 or fewer attorneys (including the 6 solos). Akron collected data about all of these grads, including the 21 in 2-10 firms. Because there were 0 unknown source people for the 50 or fewer category, and 8 in the "All Other" category (which includes the 6 solos who by definition started their own practice), then a maximum of 2 graduates working for firms of 2-10 attorneys banded together. There are other possibilities for "All Other" so it is just a maximum.

This is just one school of course, but it at least supports the notion that not all 2-10 attorney firm jobs are people banding together. Happy to consider your evidence, though, dingbat.

[dingbat]

I love the assumption that a bunch of grads banding together to start a small firm (partnership) falls under "begin a legal career" while a solo practitioner does not. How many of those small firms don't make it a year? Compare that to those who decide they'd rather get a non-JD required job that pays better.

You are also making quite a leap by just assuming all the 2-10 firms people are grads banding together after graduating. There is absolutely no data on this and you are just going by what you think is likely rather than any hard data suggesting this is the case. Not to mention that the huge majority of lawyers in this country work in firms around this size. There very well may be grads starting a firm together and being put into the 2-10 category. However, you can't just jump to "well, obviously this is the case for them" without any real data to back up your claims.

Actually, I'm not assuming they all are, I'm just saying that some are. Obviously, it's more market specific, as there are proportionally a lot more 2-10 firms in Akron than in New York City.
However, having read jenesaislaw's note above, I'll amend to just say it'll be useful if the report can list (or at least factor in) the "all other" category

[jenesaislaw]

it'll be useful if the report can list (or at least factor in) the "all other" category

Not enough data this year. However, one thing we're looking into is another reporting rule change where the "solo" category becomes a "started own practice" category. So far there's been no pushback in my initial conversations. That of course means little.

[jenesaislaw]

Bump.

[manofjustice]

Create what is called a "job unit" measured as but what else on which society runs, money.



step 1:

a) List job types
b) List markets
c) Impute likely average salary for each job type/market pair*

We could use NALP sallary reports to do c). Here is a suggested methadology. Divide schools into tier/market pair, take the highest-reporting sallary data from each tier and each market. Use those for respective school-tiers and markets.


step 2:

For each school-
Multiply the imputed average salary of each job type for each market by the number of graduates in each job type/market pair.** ***
Sum products.


step 3:

Divide by total # of graduates. Reveal "job score."


*Pretend graduates with A3Cs earn NYC BigLaw.

**One job type/market pair may have no salary data because it doesn't exist, e.g. 500+ firm outside a primary market. Therefore, redistribute the school's portion of graduates in each job type/market pair, splitting the graduates in the job type/market pairs without salary data between the job type/market pairs for that job type with salary data.

***Synthesize the n of each pair if the school does not break it out by multiplying the number of graduates reported in each job type by the portion (subject to the above adjustment) of all graduates in each market.



This gets to exactly the issue brought up in the Touro/Fordham comparison and practically tests our intuition that LSAT median and peer-assessment has a positive effect on the overall characterization of ordinary employment outcomes beyond the employment score. All variables in the model are governed by trends, limiting their potential error.

That is all.

[dingbat]

Create what is called a "job unit" measured as but what else on which society runs, money.



step 1:

a) List job types
b) List markets
c) Impute likely average salary for each job type/market pair*

We could use NALP sallary reports to do c). Here is a suggested methadology. Divide schools into tier/market pair, take the highest-reporting sallary data from each tier and each market. Use those for respective school-tiers and markets.


step 2:

For each school-
Multiply the imputed average salary of each job type for each market by the number of graduates in each job type/market pair.** ***
Sum products.


step 3:

Divide by total # of graduates. Reveal "job score."


*Pretend graduates with A3Cs earn NYC BigLaw.

**One job type/market pair may have no salary data because it doesn't exist, e.g. 500+ firm outside a primary market. Therefore, redistribute the school's portion of graduates in each job type/market pair, splitting the graduates in the job type/market pairs without salary data between the job type/market pairs for that job type with salary data.

***Synthesize the n of each pair if the school does not break it out by multiplying the number of graduates reported in each job type by the portion (subject to the above adjustment) of all graduates in each market.



This gets to exactly the issue brought up in the Touro/Fordham comparison and practically tests our intuition that LSAT median and peer-assessment has a positive effect on the overall characterization of ordinary employment outcomes beyond the employment score. All variables in the model are governed by trends, limiting their potential error.

That is all.

I've actually got a model that does this, if anyone wants

[Samara]

I really like this a lot. [babble] Yes, there are considerations beyond what the employment score measures, but this website provides all the necessary information (to the extent that it can be provided) to make an informed decision. If people decide to rely too heavily on one statistic, that's hardly LST's fault.

That said, I think it would be useful to create some sort of employment score that is weighted by salary. Yes, some people may want small law and it may make sense to choose Touro over Fordham, but half the reason we care about employment is the ability to pay down loans/earning power. Thus, from an expected value perspective, Fordham is a better "investment" than Touro. On the other hand, even if Fordham has a better expected value, the employment score drives home the point that probably no one should pay sticker at either school. So, maybe in the end it doesn't matter? [/babble]

If it were up to me, I would make the threshold for placement in a state 10% instead of 5%. 5% is just a handful of students and could easily be statistical noise. 10% seems to cross the barrier of substantial placement and would solve the "UVa problem." Perhaps this is only relevant for CA though. Looking at Indiana, that would strike out Notre Dame. Interestingly, Illinois is the only state that Notre Dame placed at least 10% into, but that is probably an unusual example. Anyway, just a thought.

[jenesaislaw]

I really like this a lot. [babble] Yes, there are considerations beyond what the employment score measures, but this website provides all the necessary information (to the extent that it can be provided) to make an informed decision. If people decide to rely too heavily on one statistic, that's hardly LST's fault.

That said, I think it would be useful to create some sort of employment score that is weighted by salary. Yes, some people may want small law and it may make sense to choose Touro over Fordham, but half the reason we care about employment is the ability to pay down loans/earning power. Thus, from an expected value perspective, Fordham is a better "investment" than Touro. On the other hand, even if Fordham has a better expected value, the employment score drives home the point that probably no one should pay sticker at either school. So, maybe in the end it doesn't matter? [/babble]

If it were up to me, I would make the threshold for placement in a state 10% instead of 5%. 5% is just a handful of students and could easily be statistical noise. 10% seems to cross the barrier of substantial placement and would solve the "UVa problem." Perhaps this is only relevant for CA though. Looking at Indiana, that would strike out Notre Dame. Interestingly, Illinois is the only state that Notre Dame placed at least 10% into, but that is probably an unusual example. Anyway, just a thought.

Thanks Samara. We are looking into the weighted salary thing, actually. We haven't found a credible way to do it, yet, but are open to ideas. As I've told people on TLS a handful of times, it's one thing to post a formula on a message board and another to stake a nonprofit's reputation on it. We are perhaps too careful in the way we conduct our business.

The 5% threshold will make more sense when we have more years of data. For now, we're using just one. After two or three years, the 5% will be sufficiently meaningful.

[jenesaislaw]

[This] practically tests our intuition that LSAT median and peer-assessment has a positive effect on the overall characterization of ordinary employment outcomes beyond the employment score.

a) What school did you end up deciding on?
b) The intuition isn't right beyond the very top schools -- check out the LST Guide on the value of the U.S. News rankings. There's an unbelievable relationship between LSAT median and rank, and an unbelievable lack of a relationship between employment outcomes (any measure) and rank.
c) Happy to work with you on your job unit idea to see what we can make of it. I am skeptical, but that's the best place to start when producing something sound.

[Samara]

I really like this a lot. [babble] Yes, there are considerations beyond what the employment score measures, but this website provides all the necessary information (to the extent that it can be provided) to make an informed decision. If people decide to rely too heavily on one statistic, that's hardly LST's fault.

That said, I think it would be useful to create some sort of employment score that is weighted by salary. Yes, some people may want small law and it may make sense to choose Touro over Fordham, but half the reason we care about employment is the ability to pay down loans/earning power. Thus, from an expected value perspective, Fordham is a better "investment" than Touro. On the other hand, even if Fordham has a better expected value, the employment score drives home the point that probably no one should pay sticker at either school. So, maybe in the end it doesn't matter? [/babble]

If it were up to me, I would make the threshold for placement in a state 10% instead of 5%. 5% is just a handful of students and could easily be statistical noise. 10% seems to cross the barrier of substantial placement and would solve the "UVa problem." Perhaps this is only relevant for CA though. Looking at Indiana, that would strike out Notre Dame. Interestingly, Illinois is the only state that Notre Dame placed at least 10% into, but that is probably an unusual example. Anyway, just a thought.

Thanks Samara. We are looking into the weighted salary thing, actually. We haven't found a credible way to do it, yet, but are open to ideas. As I've told people on TLS a handful of times, it's one thing to post a formula on a message board and another to stake a nonprofit's reputation on it. We are perhaps too careful in the way we conduct our business.

The 5% threshold will make more sense when we have more years of data. For now, we're using just one. After two or three years, the 5% will be sufficiently meaningful.

haha, exactly. You guys have a reputation to uphold, so I think it's smart to be conservative and only expand your analysis where the data can support such expansion.

From what (little) I've seen, only the top schools match salary data to job type, so creating a salary weighted score is likely prohibitively difficult at this point. Any formulation will either fail to properly account for low-pay, high-presitge positions like PI, gov't and clerking, or be a morass of near-arbitrary speculation. The only thing I can think of is to use the percentage of graduates on IBR as a proxy for the likelihood that you will get a job to pay down your loans. That's the main concern of including salary anyway. But I don't know if that data is publicly available.

Will you be using a rolling average for the geographical placement percentage? That would make sense and filter out a lot of statistical noise. Personally, I don't think you should consider a school that places less than 10% in a state if you want to work in that state, due to ties, diversity, etc., but it's not your job to make that decision for people, haha. A rolling average of 5% is just as good of a cutoff.

[jenesaislaw]

From what (little) I've seen, only the top schools match salary data to job type, so creating a salary weighted score is likely prohibitively difficult at this point. Any formulation will either fail to properly account for low-pay, high-presitge positions like PI, gov't and clerking, or be a morass of near-arbitrary speculation. The only thing I can think of is to use the percentage of graduates on IBR as a proxy for the likelihood that you will get a job to pay down your loans. That's the main concern of including salary anyway. But I don't know if that data is publicly available.

Actually, I think the top schools are doing a really, really poor job with disclosure in general. The schools doing best are the public schools. Pretty interesting.

As for the IBR info, it is not publicly available. Senator Boxer pressed the Department of Education for it, but the Dept. of Education IG was totally unhelpful.

What I think might be most interesting is an estimate of what % of people qualify for IBR. The risk is that you have people make decisions based on having IBR as a default -- which is bad for a number of reasons, at the micro and macro level. To do this we would need a good way of estimating the distribution of loan disbursements, as opposed to just the average. The data schools supply to U.S. News may provide an avenue for this. If anybody has any experience with estimating distributions, I'd love to talk.

Will you be using a rolling average for the geographical placement percentage? That would make sense and filter out a lot of statistical noise. Personally, I don't think you should consider a school that places less than 10% in a state if you want to work in that state, due to ties, diversity, etc., but it's not your job to make that decision for people, haha. A rolling average of 5% is just as good of a cutoff.

You hit the nail on the head. We didn't want to make the entire judgment for people. We were in favor of giving more and letting people pare back. It's much easier to remove schools from the equation than add.

[Nickg415]

.