Wednesday, September 15, 2010

Economist's Notebook: Self-Selection and Oregon's SAT Scores

Oregon kids' performance on the SAT exam is better than the national average crows the State Board of Education.

SALEM, Ore. (AP) -- Oregon high school students taking the SAT college entrance exam did better in reading and writing this year and held steady in mathematics.

The state Department of Education said Monday the average scores of 521 in reading and 496 in writing were two points higher than last year. The average in mathematics was 523.

The department said all three scores outpace national averages.

More than 14,000 public school students took the SAT test this year.
The ACT test taken by 11,000 students this year reported scores up slightly from last year.

If the SAT were a standard test administered to all high school students in the US then we could talk about comparing across states.  But the SAT is a voluntary test taken by those kids that would like to continue on to college, so you can't compare across states unless you can control for selection.

Take a simple example - two states with identical distributions of student talent (both inherent and taught).  Suppose however that in one state (for whatever reason) almost all students sit for the exam regardless of preparation or intent to go on to college, while in  the other state only the kids who definitely plan to go on to college sit for the exam.  Now if all students in both states were forced to take the exam we would expect the distribution of scores, including the mean and median, to be the same based on the assumption of equal talent across the two states.  But would the mean and median be the same based on the self-selection story I described above?  No.  In fact we would expect the mean and median to be much higher in the state where only college-bound kids take the test.  And in practice, there is a high correlation between participation rates and average SAT scores across states.

So how does Oregon stack up?  Not bad actually.  In 2008 it was 20th in participation rates, so its better than average performance is not just a self-selection story, but many other factors including demographics are important as well.

This is not to say that Oregon is not doing well, they could be doing fantastically well, it is just to say that you can't tell either way from the data.

By the way, the process of comparing SAT scores across states is actively discouraged by the College board so it is disappointing that the Dept. of Ed. would even mention it. This is the disclaimer that is all over their web site:

A Word About Comparing States and Schools
The SAT is a strong indicator of trends in the college-bound population, but it should never be used alone for such comparisons because demographics and other nonschool factors can have a strong effect on scores.

1 comment:

Jeff Alworth said...

On OPB, they mentioned that Washington state was bragging the highest SAT scores "among states with over 50% participation rates"--a clue that it wants credit for actual educational attainment, not sampling bias.

Oregon and Washington can also take a bow for the work of their schools. In Oregon particularly, slashed budgets might have resulted in performance losses. But apparently, our teachers are doing more with less. Good for them.