Economist's Notebook: Self-Selection and Oregon's SAT Scores

Oregon kids' performance on the SAT exam is better than the national average crows the State Board of Education.

SALEM, Ore. (AP) -- Oregon high school students taking the SAT college entrance exam did better in reading and writing this year and held steady in mathematics.

The state Department of Education said Monday the average scores of 521 in reading and 496 in writing were two points higher than last year. The average in mathematics was 523.

The department said all three scores outpace national averages.

More than 14,000 public school students took the SAT test this year.
The ACT test taken by 11,000 students this year reported scores up slightly from last year.

If the SAT were a standard test administered to all high school students in the US then we could talk about comparing across states.  But the SAT is a voluntary test taken by those kids that would like to continue on to college, so you can't compare across states unless you can control for selection.

Take a simple example - two states with identical distributions of student talent (both inherent and taught).  Suppose however that in one state (for whatever reason) almost all students sit for the exam regardless of preparation or intent to go on to college, while in  the other state only the kids who definitely plan to go on to college sit for the exam.  Now if all students in both states were forced to take the exam we would expect the distribution of scores, including the mean and median, to be the same based on the assumption of equal talent across the two states.  But would the mean and median be the same based on the self-selection story I described above?  No.  In fact we would expect the mean and median to be much higher in the state where only college-bound kids take the test.  And in practice, there is a high correlation between participation rates and average SAT scores across states.

So how does Oregon stack up?  Not bad actually.  In 2008 it was 20th in participation rates, so its better than average performance is not just a self-selection story, but many other factors including demographics are important as well.

This is not to say that Oregon is not doing well, they could be doing fantastically well, it is just to say that you can't tell either way from the data.

By the way, the process of comparing SAT scores across states is actively discouraged by the College board so it is disappointing that the Dept. of Ed. would even mention it. This is the disclaimer that is all over their web site:

A Word About Comparing States and Schools
The SAT is a strong indicator of trends in the college-bound population, but it should never be used alone for such comparisons because demographics and other nonschool factors can have a strong effect on scores.

Economist's Notebook: Selection Claims Another Victim

The normally very reliable James Surowiecki of The New Yorker blunders badly this week in his column about thriving businesses that focus on the tops and bottoms of markets.  Here is his thesis:

Apple’s launch of the iPad next week is a gamble in more ways than one. To start with, it’s obviously a bet that there are millions of people looking for a new way to surf the Web, watch movies, and read magazines. But it’s also a more fundamental gamble; namely, that people will pay for quality. Starting at five hundred dollars, the iPad is significantly more expensive than its competitors. But Apple’s assumption is that, if the iPad is also significantly better, people will happily shell out for it (as they already do for iPods, iPhones, and Macs). That’s why when Steve Jobs first introduced the iPad he said that, if a product wasn’t “far better” than what was already out there, it had “no reason for being.

For Apple, which has enjoyed enormous success in recent years, “build it and they will pay” is business as usual. But it’s not a universal business truth. On the contrary, companies like Ikea, H. & M., and the makers of the Flip video camera are flourishing not by selling products or services that are “far better” than anyone else’s but by selling things that aren’t bad and cost a lot less. These products are much better than the cheap stuff you used to buy at Woolworth, and they tend to be appealingly styled, but, unlike Apple, the companies aren’t trying to build the best mousetrap out there. Instead, they’re engaged in what Wired recently christened the “good-enough revolution.” For them, the key to success isn’t excellence. It’s well-priced adequacy.

These two strategies may look completely different, but they have one crucial thing in common: they don’t target the amorphous blob of consumers who make up the middle of the market. Paradoxically, ignoring these people has turned out to be a great way of getting lots of customers, because, in many businesses, high- and low-end producers are taking more and more of the market. In fashion, both H. & M. and Hermès have prospered during the recession. In the auto industry, luxury-car sales, though initially hurt by the downturn, are reemerging as one of the most profitable segments of the market, even as small cars like the Ford Focus are luring consumers into showrooms. And, in the computer business, the Taiwanese company Acer has become a dominant player by making cheap, reasonably good laptops—the reverse of Apple’s premium-price approach

There is an obvious logical fallacy here: that because Apple and Acer are so successful, this must be a good segment to do business in. The problem with this reasoning is what economists call selection bias: you only see the success of the businesses that are the most successful.  Making inferences about the world based on survivors leads to misleading conclusions.  It is as if you arrive shortly after the great plague and saw only the survivors and thus concluded that the plague was not fatal.

And this is the whole problem. Sure there are outsized reward to being the very best - either the most advanced or the best at cost cutting - but the problem is there is only room for one at the top and bottom. Take Apple for example, it regained its mojo only when it started producing 'insanely great' stuff again. You think it is obvious to try and be the producer of the best smartphone? Just ask Palm how well that strategy is working.

The truth is that most firms are in the middle of the market and so competition there is fierce. Thus these firms do not have the outsized success of the outliers. But the winner-take-all nature of trying to capture either end of the price/quality spectrum suggests that the expected returns for that strategy are probably just about equal to the expected returns from competing in the middle. To understand if this strategy is so good you have to look at both the winners and the losers in the battle to be the best and cheapest.  Only then can you make reliable inference about the world.

In making this claim, Surowiecki falls into the common business literature pitfall: studying the winners and making general statements about their success. It is no surprise that he quotes consultants and not economists in this piece, economists obsess about selection issues.  Perhaps if he had asked one or two they could have prevented this embarrassing blunder...  ;-)