Reliability of LST Data for comparing schools

[Law Sauce]

So Law School Transparency, which I am sure many of you already know of, has tons of great employment data about different schools.

--LinkRemoved--

My question is: How reliable is the LST data for comparing schools? What difficulties are there in interpreting the data (this data is from 2008)? And, what is the best way to use and understand it?

Obviously, I would love it if observationalist weighed in.

[sethc]

I'm not sure I understand your question. Are you asking about in comparison to data that LSAC provides to registered users, such as the "ABA Info" PDFs for each school?

I'm not familiar with that website, so thanks for pushing it my way. I'll have a look at it while we both figure out what you're asking & the answer to it, because at first glance I'm curious too.

The data is 2008, you say? Well, that's going on 3 years and they seem to be "updating" the other facets of the website.. so I'd be weary of leaning on any sort of statistical stuff they post, especially so if there's no original source.

As per their "LST's Origins" tab:

-Law School Transparency is a Tennessee non-profit that was originally founded by two Vanderbilt law students (one of whom has since graduated) in July 2009. However, the co-founders — Patrick Lynch (Class of 2010) and Kyle McEntee (Class of 2011) — began the initiative well before making the decision to incorporate.

- Motivated by Vanderbilt’s more comprehensive disclosure of job outcomes, the two began encouraging prospective students to contact other schools and ask for the same information.

-they decided to research the systemic underlying problems as to why schools weren’t already releasing this information. LST would then derive its own standard, with Vanderbilt’s list as a base, and justify the new standard on a scholarly level. After spending 8 months working on a white paper, LST finally began its campaign to drum up support.

-This critical exploration has become the basis for undercutting the criticism that “prospectives should have known better.” Lynch and McEntee’s research shows that they should not have.

-Separate from requesting voluntary participation in a new reporting standard, LST has also sought involvement with both the ABA and U.S. News, both of which are considering making changes to the information they request and publish. You can check out our coverage of the ABA initiatives here, and we hope to report very soon on developments at U.S. News.



I realize most of this is out of context, so you should read it for yourself. But, from those excerpts I, myself, would be very wary of any statistic they published as if it were fact.. unless there was some primary source that I could investigate further. I do like the entire concept, notion, and underlying principles of that website though. I'm going to read into it further.

[jenesaislaw]

I've been working on an explanation for how to use our data, and hope to finish it in the next few days. I'll surely update this thread when we post it, and probably even include the text here.

A few notes in the meantime:

- It is based on what schools reported to U.S. News for the Class of 2008. This is the most up-to-date, comparable salary information available because the ABA does not currently consider salary information to be "basic consumer information." Thankfully, the ABA's position is changing. You may be able to compare Class of 2009 salary information among some schools, but it would have to be from school websites. The Class of 2010 salary information will not exist until February 15, 2011, and is unlikely to appear on the schools' websites until at least June when they receive their individualized NALP reports, or even later (some schools still have Class of 2008 salary information on their websites even though it is not their most current).

- The data, in our opinions, are reliable raw data. However, the data as presented on U.S. News' website are not clear and that is what we've aimed to remedy. In real terms, when somebody says the median private sector salary of graduates known to be employed full time at 9 months is $160,000, we believe it is a true statement. However, it is very misleading and almost impossible to compare between schools. This is because schools differ on the key percentages: graduates known to be employed, graduates known to be pursuing full-time degrees, graduates known to be unemployed and not seeking work, graduates known to be employed in private practice or business & industry, and the percentage of that last group that schools knew the salaries of.

- The aim of the charts is to show how different those median (and 25th/75th) salary figures are from each other. Compare NYU and NYLS. Same median, much different meaning, and it's not obvious from first glance at the salary quartiles.

- I think you can compare schools, but shouldn't rely on the figures. The market has changed a lot, but you can still tell something about the schools from prior years. School reputations don't change quickly, and their locations don't either except in a few extraordinary situations.

- The main takeaway is the giant chart that shows how much we know about the graduating class. What this means for your expected salary from a particular school, as a prospective 2014/2015 graduate, is up in the air.

[sethc]

Oh cool! You guys are on TLS - I didn't know that, but was kind of hoping so. Don't take any offense by what I said, I'm just always overly critical when I think about what statistics mean & where they came from - plus, I mentioned I hadn't even heard of the site, anyway.

As an aside - can I give a personal suggestion? I know you guys have limited time & resources for this thing, but if you can ever expand enough to focus on MORE than just employment & salary $$, I think it'd really bolster your viewership & the overall substance of your site. I'll illustrate using me as an example. I was a low LSAT-above median GPA kind of guy when searching for law schools.. so employment/income wasn't a monster factor for me. Sure it's pretty clear-cut if a person will be accepted based on LSAT & UGPA depending on where you apply, but it's not so simple in my opinion. I think there's more going on behind-the-scenes of admissions that should schools should be held accountable for. I hate to sound so conspiracy-theoryish, but that's a product of being a low-LSAT guy. In other words, I felt very deserving of being able to go to law school better than the low-tier options I was faced with.. but the primary inhibitor was my LSAT score, and I just have a personal opinion that a student's enrollment should not essentially be determined by a test score. GPA I have no qualms with, nor the "softs" when applying.. it's just the weight given to LSAT that bothers me.

[jenesaislaw]

First, don't worry about it, everybody should look at statistics with a skeptical eye. Second, I appreciate your suggestions, but I don't actually think that would be appropriate for our organization, at least not as currently conceived.

The decision to attend professional school is, at least in part, an outcome-based decision. People most often attend law school to be lawyers. There are other perfectly valid reasons (politics, fun, etc), but the immediate job and career that follows from it are huge. In this sense, prospective law students are investing in their degrees and thus are consumers of law degrees. The investment is a huge one, including time, tuition, and opportunity cost, and should be made on an informed basis. Moreover, it's in everybody's interests, not just prospectives' interests, that this investment be made on an informed basis. It's also the right thing to do from those taking people's money with certain promises. (Importantly, this is not an entitlement issue. Law students are not entitled to anything other than knowing what they're getting themselves into.)

Our position, as you pointed out above, is that prospectives do not usually make informed decisions. This doesn't mean they make bad/wrong decisions, but those with positives outcomes (in some sense) lucked out through reliance on faulty statistics. This was so unacceptable to us that we decided to try to reform disclosure policies on a number of fronts, with the ultimate goal of prospectives making more-informed decisions.

So you can see why I don't think it fits our mission. We can't and won't cure all ails in legal education. Not yet at least

In my personal opinion, totally separate from Law School Transparency, I think schools should be free to make their own determinations as to what students they want to enroll. If this means slaving to the LSAT, I don't care. If it's problematic for not producing competent attorneys, the market will react accordingly. "Correcting" how schools weigh different factors is simply fundamentally different because there's no systematically bad information out there. Perhaps if law schools cost $10,000 to apply to, I'd feel differently, because then the fee would be more like an investment instead of a deterrent or administrative cost.

[sethc]

First, don't worry about it, everybody should look at statistics with a skeptical eye. Second, I appreciate your suggestions, but I don't actually think that would be appropriate for our organization, at least not as currently conceived.

The decision to attend professional school is, at least in part, an outcome-based decision. People most often attend law school to be lawyers. There are other perfectly valid reasons (politics, fun, etc), but the immediate job and career that follows from it are huge. In this sense, prospective law students are investing in their degrees and thus are consumers of law degrees. The investment is a huge one, including time, tuition, and opportunity cost, and should be made on an informed basis. Moreover, it's in everybody's interests, not just prospectives' interests, that this investment be made on an informed basis. It's also the right thing to do from those taking people's money with certain promises. (Importantly, this is not an entitlement issue. Law students are not entitled to anything other than knowing what they're getting themselves into.)

Our position, as you pointed out above, is that prospectives do not usually make informed decisions. This doesn't mean they make bad/wrong decisions, but those with positives outcomes (in some sense) lucked out through reliance on faulty statistics. This was so unacceptable to us that we decided to try to reform disclosure policies on a number of fronts, with the ultimate goal of prospectives making more-informed decisions.

So you can see why I don't think it fits our mission. We can't and won't cure all ails in legal education. Not yet at least

In my personal opinion, totally separate from Law School Transparency, I think schools should be free to make their own determinations as to what students they want to enroll. If this means slaving to the LSAT, I don't care. If it's problematic for not producing competent attorneys, the market will react accordingly. "Correcting" how schools weigh different factors is simply fundamentally different because there's no systematically bad information out there. Perhaps if law schools cost $10,000 to apply to, I'd feel differently, because then the fee would be more like an investment instead of a deterrent or administrative cost.

Thanks for the response. I understand and appreciate it. It's your site - so you're in control, not me. I guess I just found this as an opportune time to vent more than anything hopefully it at least SOUNDED like it fit hehe. I admire the goal & mission of your site judging by what you said, keep at it. I wish I had seen/used it when applying, honestly. However, I do plan to transfer so I certainly will be checking it out in the coming months.

On a personal note, I agree with your opinion somewhat. I think schools actually *are* free to take who they want, it's just that rankings have become a huge dick-measuring contest among schools, thanks in large part to the U.S. News stuff. So, now all schools want to do is beat each other to death to get 5-10 rankings closer to HYS etc. Like you seem to allude to, there's no truly one-size-fits-all way of determining quality-of-education. You might could make a case for some of the T14/25s, but that's it IMO. At any rate, I liked your post - good stuff. Keep it up.

[Law Sauce]

Thanks jenesaislaw for your answers and explanations. I guess I already was familiar with your goals and purpose (and really support them of course, I liked you on facebook ), but the real question was how useful is what you have right now to me as a 0L comparing schools and trying to make an informed decision on where to go in the fall/balancing debt load etc.

It sounds like the answer to that is: well we really just don't know and the legal market is changing so much and usnews is not releasing much of the really important information.

I guess my follow-up then is: from your vantage point, which clearly is better than mine, how is the legal market changing? Is this a large, permanent change (a rethinking of the business and hiring model of firms) which would lessen the value of comparing the relatively small differences between schools who three years ago had similar levels of prestige (perhaps now we may being to see other factors play larger roles than in the past, like geographic proximity, firms hire from schools close in proximity, or debt load, maybe firms will like to hire debt peons who will not be able to ever leave the firm, etc.)? Or it is more the case that prestige levels will likely only be reinforced, making comparing prestige and employment data from the past all the more useful as a predictive tool for the future of the legal hiring market? Or do you think we will begin to see the hierarchy of schools begin to be shaken up, with schools that provide better education, as opposed to simple historical prestige, rise to the top of employment statistics?

On another note, did you attend vandy? If so, what you do think of its current position in the legal hiring market? How does it compare to t14 schools above it and schools like wustl below it?

[Law Sauce]

Is your avatar phil keaggy? I can't quite tell...

[jenesaislaw]

Thanks jenesaislaw for your answers and explanations. I guess I already was familiar with your goals and purpose (and really support them of course, I liked you on facebook ), but the real question was how useful is what you have right now to me as a 0L comparing schools and trying to make an informed decision on where to go in the fall/balancing debt load etc.

It sounds like the answer to that is: well we really just don't know and the legal market is changing so much and usnews is not releasing much of the really important information.

I guess my follow-up then is: from your vantage point, which clearly is better than mine, how is the legal market changing? Is this a large, permanent change (a rethinking of the business and hiring model of firms) which would lessen the value of comparing the relatively small differences between schools who three years ago had similar levels of prestige (perhaps now we may being to see other factors play larger roles than in the past, like geographic proximity, firms hire from schools close in proximity, or debt load, maybe firms will like to hire debt peons who will not be able to ever leave the firm, etc.)? Or it is more the case that prestige levels will likely only be reinforced, making comparing prestige and employment data from the past all the more useful as a predictive tool for the future of the legal hiring market? Or do you think we will begin to see the hierarchy of schools begin to be shaken up, with schools that provide better education, as opposed to simple historical prestige, rise to the top of employment statistics?

On another note, did you attend vandy? If so, what you do think of its current position in the legal hiring market? How does it compare to t14 schools above it and schools like wustl below it?

The most useful information to glean is that you need to be very careful when reading salary information. Pay careful attention to all of the factors I've outlined above, and ask questions when the answers are not available. When a school refuses to answer, let us know (lawschooltransparency@gmail.com) and we will investigate.

As far as whether the model is changing, I really haven't a clue. It seems to be, which would spell lower salaries, but it's really hard to know. See if you can get a 5-10 minute conversation with Bill Henderson from Indiana; he is much more willing to make predictions about this stuff.

I do think that using relative prestige as a predictive tool will remain useful. How pronounced the differences will be is hard to say; but it must be important that some schools perform better during this climate than others.

Anyhow, I currently attend Vandy. I'm a 3L and I think we've done pretty well, both in clerkships and biglaw. Not nearly to the level of years past, but I think we're right in there with Georgetown and the like. I guess we'll see in a few months when more information seeps out.

As far as my avatar goes, it's a random picture that I found when searching "french guy beret."

[Law Sauce]

Thanks jenesaislaw for your answers and explanations. I guess I already was familiar with your goals and purpose (and really support them of course, I liked you on facebook ), but the real question was how useful is what you have right now to me as a 0L comparing schools and trying to make an informed decision on where to go in the fall/balancing debt load etc.

It sounds like the answer to that is: well we really just don't know and the legal market is changing so much and usnews is not releasing much of the really important information.

I guess my follow-up then is: from your vantage point, which clearly is better than mine, how is the legal market changing? Is this a large, permanent change (a rethinking of the business and hiring model of firms) which would lessen the value of comparing the relatively small differences between schools who three years ago had similar levels of prestige (perhaps now we may being to see other factors play larger roles than in the past, like geographic proximity, firms hire from schools close in proximity, or debt load, maybe firms will like to hire debt peons who will not be able to ever leave the firm, etc.)? Or it is more the case that prestige levels will likely only be reinforced, making comparing prestige and employment data from the past all the more useful as a predictive tool for the future of the legal hiring market? Or do you think we will begin to see the hierarchy of schools begin to be shaken up, with schools that provide better education, as opposed to simple historical prestige, rise to the top of employment statistics?

On another note, did you attend vandy? If so, what you do think of its current position in the legal hiring market? How does it compare to t14 schools above it and schools like wustl below it?

The most useful information to glean is that you need to be very careful when reading salary information. Pay careful attention to all of the factors I've outlined above, and ask questions when the answers are not available. When a school refuses to answer, let us know (lawschooltransparency@gmail.com) and we will investigate.

As far as whether the model is changing, I really haven't a clue. It seems to be, which would spell lower salaries, but it's really hard to know. See if you can get a 5-10 minute conversation with Bill Henderson from Indiana; he is much more willing to make predictions about this stuff.

I do think that using relative prestige as a predictive tool will remain useful. How pronounced the differences will be is hard to say; but it must be important that some schools perform better during this climate than others.

Anyhow, I currently attend Vandy. I'm a 3L and I think we've done pretty well, both in clerkships and biglaw. Not nearly to the level of years past, but I think we're right in there with Georgetown and the like. I guess we'll see in a few months when more information seeps out.

As far as my avatar goes, it's a random picture that I found when searching "french guy beret."

haha awesome. thanks for all the info

[observationalist]

A few more suggestions about contacting the schools (once you're accepted somewhere, so that adcomms won't attach it to your file):

1) Email the Career Services Dean asking for complete employer lists for the Class of 2009 and Class of 2010. At a minimum, the lists should include the employer name and city for each graduate, not just those in private practice. Also ask for the 25th/median/75th starting salaries for private sector grads, along with how many private sector grads actually provided starting salaries to the school. [This percentage will usually be a fraction of all the graduates who entered private practice, and can be as low as 10-15% at some schools.] Feel free to ask for anything else along these lines that you think you want to know. The only type of information they likely can't disclose are personally-identifiable information such as race, gender, names or addresses of graduates, etc. Everything else is fair game.
2) Wait for a response, which will most likely be 'We value your request but no, because X'
3) Depending on what X is, respond with the following:
[if they say that's too much work]: While I understand your office is very busy working with current students and graduates, I am trying to figure out whether or not to make a significant personal investment in your program. It has come to my attention from reading Y or Z that your school regularly collects all of the information I'm requesting, in order to calculate percentages and submit to U.S. News each year. All I'm asking for is that you provide me with the underlying data in your possession, so that I can get a better idea of the jobs obtained by recent graduates. It's really important that I know which employers are hiring so that I can research those firms and decide whether to attend.
[if they say they value your desire to see what the job market looks like right now, but that they don't have data for the Class of 2010]: No worries, I can wait until the end of February and will contact you again at that time.
[if they do not respond at all]: Forward your email to Admissions and explain to them that you have not received a response, but that the information is very important to you in deciding whether to attend the school or go somewhere else.
4) Post responses here, email them to lawschooltransparency@gmail.com, or find another way to share them with other applicants. There are a lot of you out there with a lot riding on making an informed decision. G'luck.

[observationalist]

A few more suggestions about contacting the schools (once you're accepted somewhere, so that adcomms won't attach it to your file):

1) Email the Career Services Dean asking for complete employer lists for the Class of 2009 and Class of 2010. At a minimum, the lists should include the employer name and city for each graduate, not just those in private practice. Also ask for the 25th/median/75th starting salaries for private sector grads, along with how many private sector grads actually provided starting salaries to the school. [This percentage will usually be a fraction of all the graduates who entered private practice, and can be as low as 10-15% at some schools.] Feel free to ask for anything else along these lines that you think you want to know. The only type of information they likely can't disclose are personally-identifiable information such as race, gender, names or addresses of graduates, etc. Everything else is fair game.
2) Wait for a response, which will most likely be 'We value your request but no, because X'
3) Depending on what X is, respond with the following:
[if they say that's too much work]: While I understand your office is very busy working with current students and graduates, I am trying to figure out whether or not to make a significant personal investment in your program. It has come to my attention from reading Y or Z that your school regularly collects all of the information I'm requesting, in order to calculate percentages and submit to U.S. News each year. All I'm asking for is that you provide me with the underlying data in your possession, so that I can get a better idea of the jobs obtained by recent graduates. It's really important that I know which employers are hiring so that I can research those firms and decide whether to attend.
[if they say they value your desire to see what the job market looks like right now, but that they don't have data for the Class of 2010]: No worries, I can wait until the end of February and will contact you again at that time.
[if they do not respond at all]: Forward your email to Admissions and explain to them that you have not received a response, but that the information is very important to you in deciding whether to attend the school or go somewhere else.
4) Post responses here, email them to lawschooltransparency@gmail.com, or find another way to share them with other applicants. There are a lot of you out there with a lot riding on making an informed decision. G'luck.

You can also direct them to the information available on the school's website and explain why you think it's misleading. Oftentimes the career services office is not in charge of putting the statistics online for prospective law students; that job goes to admissions. And as we've come to realize in making our requests to the schools, there can be a disconnect between offices, where admissions will actually take the data from career services and selectively portray it to appeal to prospectives. A lot of schools actually don't tell you that the median salary statistics aren't based on the entire class, even though this is incredibly important for you in figuring out whether the debt is worth it.