Economist's Notebook: Randomized Experiments

Economics is a social science, meaning most of what we study cannot be replicated in a laboratory. We have to find causal links in data that are contaminated by lots of real-world complication and interconnectedness. We have become exceedingly good at doing so, but our conclusions will never rise to the level of "truth." We can propose hypotheses and fail to falsify them, but generally no more.

Often we have to deal with the fact that we don't have the counter-factual. Case in point: how will we know how effective federal fiscal stimulus was? Answer: we won't. We will never know what would have happened in the absence of federal fiscal stimulus.

Randomized experiments therefore are the holy grail of many economists. They are not laboratory experiments (meaning you still can't control all environmental variables), but at least you can create a counter-factual. Often we try to use real world data to estimate treatment and control groups but they are not random. For example, we know that college graduates make a lot more money that those with only a high school diploma on average but does this tell us the extra income the education itself commands? No, because the population of college graduate is a non-random sample of people in general: they are probably richer, better students, more highly motivated, etc. So using average differences will bias the inference of the value of the college education because part of this is due to self-selection into college.

Randomization then, allows the researcher to assume that the people who are effected by a variable (a college education say) are no different than those that don't. The problem in economics (like medicine) is that we are often talking about policies that affect peoples lives and welfare and randomizing treatment for the purposes of study is of questionable ethics, and more importantly for economics, is politically untenable.

It is happening more and more, however, as economic researchers get more involved in policy design and roll-out. It is often possible to roll-out a new policy in phases and in so doing randomize treatment and control groups, but it is still hard and as the control groups usually end up getting treated eventually, it gives you only a short-term view.

I say all this because yesterday The Oregonian published a fascinating article on a unique randomization event that happened in Oregon by Don Colburn. The essence is that Oregon ran short of money and randomly picked from a population a sub-group who got Medicaid health insurance. This allows for a careful study of how insurance affects behavior. Turns out in predictable ways, but the real value is in estimating the marginal behavior: e.g., by how much do doctor and hospital visits decline when the patient has to pay, how much is spent on medical care and how do health outcomes differ in relation to the first two? The results will have huge utility to those contemplating wholesale changes to our nation's health system.