Scaling and the California Bar Exam Forum

[L’Étranger]

Wanted to start a thread to continue a discussion that was going on the Feb CA bar exam thread in case anyone was interested (and obsessing over the Feb exam the way I am).

These are my thoughts on how the bar exam is scaled and what it might mean in terms of our ability to speculate about the results that will come out in a month. This is all speculation based on bits and pieces I've picked up so may be complete bullshit.

Scaling, which is done by the NCBE, is purported to create a kind of absolute scale against which test taking conditions from different exams are standardized. Thus according to the NCBE, a scaled 130 MBE score on the July 2014 exam is supposed to be the same as a scaled 130 MBE on the February 2007 exam. So, different raw scores from different exam dates may scale to the same scaled score based on conditions that include the NCBE's estimation of the difficulty of the exam.

The California Bar cryptically says that they scale their written scores to the same scale as the MBE. When you fail the bar, as I had the misfortune to last July, the CA bar sends you a function in the form of a linear equation into which you are supposed to plug in your raw written score and it spits out a score that is supposedly on the same scale as the MBE. From test to test, the function is always in the form of:

Scaled written score = [Raw written score x (a)] – b

Variables "a" and "b" in the equation above always change, and there are sometimes dramatic differences between what raw score that is required to pass the exam (assuming you also have a solid MBE).

I've tried to think about where this equation comes from and how/if we might be able to use the reported mean scaled MBE (which I believe to be 136) to speculate how much of a boost we can expect on the written portion of the exam. Here are some points that I came up with:

1) All of the actual scaling that occurs, occurs at the MBE. The CA Bar does not actually scale the written exam, but rather piggybacks onto the scaling done by NCBE to the MBE exam. If you are interested, this is a link (sort of) describing how the NCBE scales the MBE. http://www.ncbex.org/assets/media_files ... _Mroch.pdf

2) The linear equation above is generated by CA Bar through linear regression by plotting raw written scores on an exam administration (x-axis) vs. the scaled MBE scores from the same administration (y-axis) for every individual test taker. The line that is generated through linear regression can be expressed as and (sort of) represents what any single raw written score would be if on the same scale as the MBE:

Scaled written score = [Raw written score x (a)] – b

A steeper slope (i.e. a higher value for "a") and a smaller y-intercept (i.e. a smaller value for "b") will mean that a lower raw score will "scale" to a higher score (i.e. the better the curve). A I understand it, the only way to get a steeper slope and a lower y-intercept is if the raw written and scaled MBE scores cluster towards the y-axis. This type of clustering represents a scenario where most people score low on the written score while most people score high on the MBE.

3) This year, the reported mean scaled MBE score of 136 is low for February, which tends to suggest (although not definitively) that we as a group did not do well on the MBE. Thus the "a" variable (i.e. the slope) will likely not be super high. However, February always seems to vary between 136-ish and 138-ish, so the "a" variable will likely still be around 3.xx just on the lower side (i.e. 3.0xx rather than 3.1xx)

The "b" variable is hard to judge and will depend on whether a lot of people did shitty so that the raw scores skew left.

Please feel free to argue, ridicule, or add.

[comet_halley]

Wanted to start a thread to continue a discussion that was going on the Feb CA bar exam thread in case anyone was interested (and obsessing over the Feb exam the way I am).

These are my thoughts on how the bar exam is scaled and what it might mean in terms of our ability to speculate about the results that will come out in a month. This is all speculation based on bits and pieces I've picked up so may be complete bullshit.

Scaling, which is done by the NCBE, is purported to create a kind of absolute scale against which test taking conditions from different exams are standardized. Thus according to the NCBE, a scaled 130 MBE score on the July 2014 exam is supposed to be the same as a scaled 130 MBE on the February 2007 exam. So, different raw scores from different exam dates may scale to the same scaled score based on conditions that include the NCBE's estimation of the difficulty of the exam.

The California Bar cryptically says that they scale their written scores to the same scale as the MBE. When you fail the bar, as I had the misfortune to last July, the CA bar sends you a function in the form of a linear equation into which you are supposed to plug in your raw written score and it spits out a score that is supposedly on the same scale as the MBE. From test to test, the function is always in the form of:

Scaled written score = [Raw written score x (a)] – b

Variables "a" and "b" in the equation above always change, and there are sometimes dramatic differences between what raw score that is required to pass the exam (assuming you also have a solid MBE).

I've tried to think about where this equation comes from and how/if we might be able to use the reported mean scaled MBE (which I believe to be 136) to speculate how much of a boost we can expect on the written portion of the exam. Here are some points that I came up with:

1) All of the actual scaling that occurs, occurs at the MBE. The CA Bar does not actually scale the written exam, but rather piggybacks onto the scaling done by NCBE to the MBE exam. If you are interested, this is a link (sort of) describing how the NCBE scales the MBE. http://www.ncbex.org/assets/media_files ... _Mroch.pdf

2) The linear equation above is generated by CA Bar through linear regression by plotting raw written scores on an exam administration (x-axis) vs. the scaled MBE scores from the same administration (y-axis) for every individual test taker. The line that is generated through linear regression can be expressed as and (sort of) represents what any single raw written score would be if on the same scale as the MBE:

Scaled written score = [Raw written score x (a)] – b

A steeper slope (i.e. a higher value for "a") and a smaller y-intercept (i.e. a smaller value for "b") will mean that a lower raw score will "scale" to a higher score (i.e. the better the curve). A I understand it, the only way to get a steeper slope and a lower y-intercept is if the raw written and scaled MBE scores cluster towards the y-axis. This type of clustering represents a scenario where most people score low on the written score while most people score high on the MBE.

3) This year, the reported mean scaled MBE score of 136 is low for February, which tends to suggest (although not definitively) that we as a group did not do well on the MBE. Thus the "a" variable (i.e. the slope) will likely not be super high. However, February always seems to vary between 136-ish and 138-ish, so the "a" variable will likely still be around 3.xx just on the lower side (i.e. 3.0xx rather than 3.1xx)

The "b" variable is hard to judge and will depend on whether a lot of people did shitty so that the raw scores skew left.

Please feel free to argue, ridicule, or add.

They should just release the results a bit earlier rather than let us do these surmising.

[L’Étranger]

They should just release the results a bit earlier rather than let us do these surmising.

That's a whole other thing. I feel like in general the CA Bar invests disproportionately less into the whole process than the applicants actually have on the line.

[s1m4]

Great thread, hope to contribute later! For now, just wanted to throw these figures in so we have everything relevant under one roof.


(Raw written score x 3.1541 ) – 529.503 - Feb 2012
(Raw written score x 2.8372) – 311.0880 - Jul 2012
(Raw written score x 3.6115) - 684.4297 - Jul 2013
(Raw written score x 3.1098) - 469.4400 July 2014.
(Raw written score x 2.8372) – 311.0880 Feb 2011.
(Raw written score x 3.1584) – 460.6635 Feb 2014.

[L’Étranger]

Reported mean scaled MBE vs. Cal Bar Conversions (sorry that it looks crap).

Year---Feb---Jul-----Conversions
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7
2008 137.7 145.6
2009 135.7 144.5
2010 136.6 143.6
2011 138.6 143.8 (Raw written score x 2.8372) – 311.0880 Feb 2011
2012 137.0 143.4 (Raw written score x 3.1541 ) – 529.503 - Feb 2012 (Raw written score x 2.8372) – 311.0880 - Jul 2012
2013 138.0 144.3 (Raw written score x 3.6115) - 684.4297 - Jul 2013
2014 138.0 141.5 (Raw written score x 3.1098) - 469.4400 July 2014 (Raw written score x 3.1584) – 460.6635 Feb 2014
2015 136.2

[gaagoots]

Reported mean scaled MBE vs. Cal Bar Conversions addendum

Year---Feb---Jul-----Conversions
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7
2008 137.7 145.6
2009 135.7 144.5 (Raw written score x 2.9397) – 403.7226 Feb 2009****
2010 136.6 143.6 (Raw written score x 3.0750) – 490.3599 Feb 2010
2011 138.6 143.8 (Raw written score x 2.8372) – 311.0880 Feb 2011
2012 137.0 143.4 (Raw written score x 3.1541 ) – 529.503 - Feb 2012 (Raw written score x 2.8372) – 311.0880 - Jul 2012
2013 138.0 144.3 (Raw written score x 3.6115) - 684.4297 - Jul 2013
2014 138.0 141.5 (Raw written score x 3.1098) - 469.4400 July 2014 (Raw written score x 3.1584) – 460.6635 Feb 2014
2015 136.2

****Note the 135.7 mean in Feb 09' yielded a 33.5 overall pass (I excluded the attorneys exam as they didn't take the MBE--but note: adding the attorney exam only boosted it to 34.5 overall)

I wish someone has the data from Feb 2013 because the mean MBE was exactly the same as Feb 2014. This makes me then think, was the scaling had something to due with the essay/PT difficulty? The pass rate for Feb 2013 was 41% (excluding attorney's exam) and for Feb 2014 45.3 (excluding attorney's exam).

[comet_halley]

The curve is based on California mean MBE grade, not national mean MBE grade. Although they might have some correlations, they are not the same.

Reported mean scaled MBE vs. Cal Bar Conversions addendum

Year---Feb---Jul-----Conversions
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7
2008 137.7 145.6
2009 135.7 144.5 (Raw written score x 2.9397) – 403.7226 Feb 2009****
2010 136.6 143.6 (Raw written score x 3.750) – 490.3599 Feb 2010
2011 138.6 143.8 (Raw written score x 2.8372) – 311.0880 Feb 2011
2012 137.0 143.4 (Raw written score x 3.1541 ) – 529.503 - Feb 2012 (Raw written score x 2.8372) – 311.0880 - Jul 2012
2013 138.0 144.3 (Raw written score x 3.6115) - 684.4297 - Jul 2013
2014 138.0 141.5 (Raw written score x 3.1098) - 469.4400 July 2014 (Raw written score x 3.1584) – 460.6635 Feb 2014
2015 136.2

****Note the 135.7 mean in Feb 09' yielded a 33.5 overall pass (I excluded the attorneys exam as they didn't take the MBE--but note: adding the attorney exam only boosted it to 34.5 overall)

I wish someone has the data from Feb 2013 because the mean MBE was exactly the same as Feb 2014. This makes me then think, was the scaling had something to due with the essay/PT difficulty? The pass rate for Feb 2013 was 41% (excluding attorney's exam) and for Feb 2014 45.3 (excluding attorney's exam).

[L’Étranger]

The curve is based on California mean MBE grade, not national mean MBE grade. Although they might have some correlations, they are not the same.

I believe that you are correct. California uses the scaled MBE scores of it's own applicants, and the mean scaled MBE is a national marker. However, as a large heterogeneous subpopulation, California's, overall performance on the MBE is likely just about exactly the same as the rest of the country's.

[L’Étranger]

Hey Gaagoots thanks for posting. Can you confirm this -->(Raw written score x 3.750) – 490.3599 ?

This is the best curve that I've seen. A 515 raw score translated to a 1440 scaled written. Yet, the overall pass rate was a 37.1 which is the second lowest pass rate for February in the past 8 years.

If it's right, they must have done horribly on their essays. I'm starting to think that how well the overall group performs on the essays fluctuates a lot more from year to year and has greater effect on the curve than the fluctuations on the MBE.

What you're saying about PTs is interesting. I'd say in general PTs make or break your exam. I could see a really hard PT tanking a lot of people's raw scores and thus making the multiplier higher.

Maybe the question is just how hard was our written exam compared to a year like 2010 (above) where it seems like the curve was really giving?

Maybe the curve on the written exam doesn't matter to us (as anxiously waiting applicants) as much as the overall pass rate which has gone up into 40%s in the last four examinations?

[gaagoots]

Hey Gaagoots thanks for posting. Can you confirm this -->(Raw written score x 3.750) – 490.3599 ?

This is the best curve that I've seen. A 515 raw score translated to a 1440 scaled written. Yet, the overall pass rate was a 37.1 which is the second lowest pass rate for February in the past 8 years.

If it's right, they must have done horribly on their essays. I'm starting to think that how well the overall group performs on the essays fluctuates a lot more from year to year and has greater effect on the curve than the fluctuations on the MBE.

What you're saying about PTs is interesting. I'd say in general PTs make or break your exam. I could see a really hard PT tanking a lot of people's raw scores and thus making the multiplier higher.

Maybe the question is just how hard was our written exam compared to a year like 2010 (above) where it seems like the curve was really giving?

Maybe the curve on the written exam doesn't matter to us (as anxiously waiting applicants) as much as the overall pass rate which has gone up into 40%s in the last four examinations?

It was a typo so I edited it. Its 3.0750. Sorry about that

Yes, CA does not provide the state MBE average, but it does hover slightly above the national mean.

[s1m4]

Off the top of your heads - what year had the hardest scale you guys have seen? I'm at work, don't want to add everything into my excel sheet right now :/

[L’Étranger]

Hey Gaagoots thanks for posting. Can you confirm this -->(Raw written score x 3.750) – 490.3599 ?

This is the best curve that I've seen. A 515 raw score translated to a 1440 scaled written. Yet, the overall pass rate was a 37.1 which is the second lowest pass rate for February in the past 8 years.

If it's right, they must have done horribly on their essays. I'm starting to think that how well the overall group performs on the essays fluctuates a lot more from year to year and has greater effect on the curve than the fluctuations on the MBE.

What you're saying about PTs is interesting. I'd say in general PTs make or break your exam. I could see a really hard PT tanking a lot of people's raw scores and thus making the multiplier higher.

Maybe the question is just how hard was our written exam compared to a year like 2010 (above) where it seems like the curve was really giving?

Maybe the curve on the written exam doesn't matter to us (as anxiously waiting applicants) as much as the overall pass rate which has gone up into 40%s in the last four examinations?

It was a typo so I edited it. Its 3.0750. Sorry about that

Yes, CA does not provide the state MBE average, but it does hover slightly above the national mean.

Yikes. That's actually a pretty harsh curve. Needed a 630 raw to get around 1440 scaled written.

[L’Étranger]

Off the top of your heads - what year had the hardest scale you guys have seen? I'm at work, don't want to add everything into my excel sheet right now :/

I've got to bill now too, but what I'd really like to do if I get a chance is create an excel with columns for:

mean scaled MBE Feb/mean scaled MBE July/formula for written curve Feb/formula for written curve Jul/either use iterative function or just trial and error to compute minimum raw score needed to get 1440 scaled written/pass rates

It would be interesting to play with all those numbers and see if anything interesting jumps out.

[s1m4]

Off the top of your heads - what year had the hardest scale you guys have seen? I'm at work, don't want to add everything into my excel sheet right now :/

I've got to bill now too, but what I'd really like to do if I get a chance is create an excel with columns for:

mean scaled MBE Feb/mean scaled MBE July/formula for written curve Feb/formula for written curve Jul/either use iterative function or just trial and error to compute minimum raw score needed to get 1440 scaled written/pass rates

It would be interesting to play with all those numbers and see if anything interesting jumps out.

Sounds interesting, will read about iterative function a bit later -

Also, where did you guys get all the stats? I looked at the "unsuccessful applicants" pdfs that were on the cali bar's database, but only found the three I posted.

[L’Étranger]

Minimum raw written scores needed to achieve a scaled written compared to mean scaled MBEs. Please provide more written to scaled formulas for missing years if you can find (again, sorry it looks like crap).

Year--Feb---Jul----Feb Min---------Jul Min-----Pass% Feb------Pass% Jul
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7---------------------------------36.8--------------56.1
2008 137.7 145.6---------------------------------39.6--------------61.7
2009 135.7 144.5 627.1805286-----------------33.9---------------56.4
2010 136.6 143.6 627.7593171-----------------37.1---------------54.8
2011 138.6 143.8 617.1887777-----------------42.3---------------54.8
2012 137.0 143.4 624.4263023-----------------42.2---------------55.3
2013 138.0 144.3 ----------------588.2402603--41.0--------------55.8
2014 138.0 141.5 601.7804901 614.0073317--45.3------------- 48.6
2015 136.2

[s1m4]

Minimum raw written scores needed to achieve a scaled written compared to mean scaled MBEs. Please provide more written to scaled formulas for missing years if you can find (again, sorry it looks like crap).

Year--Feb---Jul----Feb Min---------Jul Min-----Pass% Feb------Pass% Jul
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7---------------------------------36.8--------------56.1
2008 137.7 145.6---------------------------------39.6--------------61.7
2009 135.7 144.5 627.1805286-----------------33.9---------------56.4
2010 136.6 143.6 627.7593171-----------------37.1---------------54.8
2011 138.6 143.8 617.1887777-----------------42.3---------------54.8
2012 137.0 143.4 624.4263023-----------------42.2---------------55.3
2013 138.0 144.3 ----------------588.2402603--41.0--------------55.8
2014 138.0 141.5 601.7804901 614.0073317--45.3------------- 48.6
2015 136.2

damn - I just looked at this table in detail and realized that we may have a harsh curve, because the exam is actually way harder this year (extremely low mean mbe.) You would think that a harder exam would yield a higher curve.

You can see the pass rate for Feb all the way back @ http://admissions.calbar.ca.gov/Portals ... 1114_R.pdf

Its very clear that lower mean MBE directly correlates with lower pass rate, and based on the data, harsher curve

However, Feb 2014 curve was significantly better than July 2014 curve. So, what to think?

[L’Étranger]

Minimum raw written scores needed to achieve a scaled written compared to mean scaled MBEs. Please provide more written to scaled formulas for missing years if you can find (again, sorry it looks like crap).

Year--Feb---Jul----Feb Min---------Jul Min-----Pass% Feb------Pass% Jul
2001 136.5 142.8
2002 135.3 141.2
2003 135.7 141.6
2004 135.9 141.2
2005 137.7 141.6
2006 137.5 143.3
2007 136.9 143.7---------------------------------36.8--------------56.1
2008 137.7 145.6---------------------------------39.6--------------61.7
2009 135.7 144.5 627.1805286-----------------33.9---------------56.4
2010 136.6 143.6 627.7593171-----------------37.1---------------54.8
2011 138.6 143.8 617.1887777-----------------42.3---------------54.8
2012 137.0 143.4 624.4263023-----------------42.2---------------55.3
2013 138.0 144.3 ----------------588.2402603--41.0--------------55.8
2014 138.0 141.5 601.7804901 614.0073317--45.3------------- 48.6
2015 136.2

damn - I just looked at this table in detail and realized that we may have a harsh curve, because the exam is actually way harder this year (extremely low mean mbe.) You would think that a harder exam would yield a higher curve.

You can see the pass rate for Feb all the way back @ http://admissions.calbar.ca.gov/Portals ... 1114_R.pdf

Its very clear that lower mean MBE directly correlates with lower pass rate, and based on the data, harsher curve

However, Feb 2014 curve was significantly better than July 2014 curve. So, what to think?

Thanks for noticing this. No one seemed all that into it so I quit once I couldn't figure out how to post a simple excel graph on here.

If you graph mean scaled MBE scores (x-axis) verses pass rates from Feb 2007-2014 (y-axis) you can then plot a best-fit line that may roughly predict the passage rate for our exam. If I'm correct about that, the passage rate will roughly be in the mid 30s.

[L’Étranger]

damn - I just looked at this table in detail and realized that we may have a harsh curve, because the exam is actually way harder this year (extremely low mean mbe.) You would think that a harder exam would yield a higher curve.

You can see the pass rate for Feb all the way back @ http://admissions.calbar.ca.gov/Portals ... 1114_R.pdf

Its very clear that lower mean MBE directly correlates with lower pass rate, and based on the data, harsher curve

However, Feb 2014 curve was significantly better than July 2014 curve. So, what to think?

This is what I think the numbers mean based on some internet research:

1) The mean scaled MBE represents how smart the NCBE believes our exam taking group was relative to exam taking groups in other years.

2) That measure (i.e how smart the NCBE believes we are) at least in part determines the written score curve. Generally speaking, a low mean scaled MBE (i.e. like ours, which was the lowest in 10 years) will mean that a higher raw written score is needed to pass, because we are perceived to be dumber as a test taking group than in other years.

The mean MBE score is not determinative of this by itself, however. I believe the CA bar also looks at the raw written scores and plots them against the mean MBE scores.

The best curve (i.e. lowest raw written score needed to pass) would be in a year with both a high mean scaled MBE and low overall raw written scores.

High overall raw written scores with a low mean scaled MBE would yield a more difficult curve.

To answer your question, Feb 2014 had a relatively high scaled MBE score for Feb. July 2014 had a relatively low scaled MBE score for July.

[s1m4]

I agree with everything you said ^.

Just looking at Feb pass rates in relation to mean MBE and curve with the naked eye, I think its almost certain our pass rate will be low to mid 30s.

But I still don't understand - why do they curve based on the mean mbe based on the belief that we are all dumber, rather than on the belief that the mbe was harder? Why are they punishing test takers with a lower curve in an overall harder exam? How do they justify this at their meetings? Also, did they factor in the difficulty of studying for the new civ pro mbe?

Another point - the current mean MBE does not reflect Cali scores - could Cali scores shift up the mean MBE, since our present figure only reflects all the other released jxs? I mean, Cali has, by far, the most test takers. I read that Cali doesnt release the state mean mbe, but is it factored into the national? Could ours be way higher?

If the pass rate is mid 30s or below and reflects earlier Feb years, its apparent that you need straight 65s on essays and 150 mbes to potentially recover from some bad PT scores.

Ps- typing this on my phone.

[PennJD83]

I wish I understood the math behind your predictions (I barely made it through calc in high school lol) but I just find the whole scaling thing to be unfair. Seriously...a pass rate in the low to mid 30s?! It just seems like the bar examiners have a quota system set in place for every administration which ends up precluding a whole host of qualified applicants from becoming attorneys. Smh...

[s1m4]

I wish I understood the math behind your predictions (I barely made it through calc in high school lol) but I just find the whole scaling thing to be unfair. Seriously...a pass rate in the low to mid 30s?! It just seems like the bar examiners have a quota system set in place for every administration which ends up precluding a whole host of qualified applicants from becoming attorneys. Smh...

A 30% pass rate is asinine - and the barrier of entry to taking this exam is already very high, so you know everyone is more or less prepared - I mean it costs like 1k just to take this exam, more if you get screwed with the hotel pricing. Many qualified people wont pass, unfortunately. Its also asinine that I invested this much money and time into a license I have a 30-40% shot of getting.

[Biotech]

Did the pass rate go up with the better curve in feb 2014?

[s1m4]

Did the pass rate go up with the better curve in feb 2014?

Yes, but ITT we figured out that the curve is correlated with the mean MBE which was high in feb 2014 (thus high curve) and is abysmally low for Feb 2015 - all the years we had similar MBE means, our Feb curve was very tough, resulting in low 30s pass rate.

[s1m4]

So any predictions as to whether Cali will push the mean MBE up to 137? Possible?

[Biotech]

So an easier MBE and more people pass CBX. Why the special brand of torture if it all comes down to that? (I missed Feb 2014 by 3 points. I'm especially bitter about that.) Statistically, it looks like I'm unlikely to have passed this time since my score dropped in July with the curve.