Some more on the problem of law students as inexperienced gatekeepers:
How can we judge the quality of student decisions? Well, one way I guess is to ask the experts.
Often when I read an article in a truly elite law journal that's in my area of expertise I'm surprised that it was selected -- this is because I can see and identify problems with it. Of course, I have no idea what the other articles under consideration were, so the fact that I think a particular piece isn't great (and that I'm familiar with better pieces published in "lesser" law reviews around the same time) isn't a great judge.
Another way of looking at this issue is to use citation data. As I wrote about a couple weeks back in the context of Theodore Eisenberg's and Martin Wells' latest on citations, a study of citations also has a lot of problems. But if we would just suspend objections for a moment, I want to talk about a simple study I published a few years back, which looked at citations to articles in a thirteen leading law journals over a 15 year period.
The study found that many articles in our nation's most elite journals did substantially less well than articles published in very good, even if not the most elite journals. There are a lot of things to be said about this -- including that, wow, there's not a lot of space in those journals and some articles do great -- absolutely fantastic in citations -- but a lot don't.
But I think it also suggests that a lot of judgements, even by editors of the best journals, may not be the best decisions they could have made. This is hindsight -- and it poses all kinds of problems related to field bias in citations -- but it also reminds us that some articles in the best journals may not be as good as many other articles published in other journals. And I think that's an important caveat, especially this time of year as hiring and promotion and tenure committees are gearing up.
http://www.thefacultylounge.org/2012/08 ... icles.html