Economist's Notebook: Esther Duflo and Randomized Trials

Here is a video of this year's winner of the John Bates Clark medal in economics (given to the most influential economist under the age of 40), Esther Duflo, promoting the core of her work: using randomized trials to understand the effectiveness of policy interventions in developing countries.

I have talked about this in the past - while I think that randomized trials are great, they have many limitations.  One criticism I have of Duflo is that she is very dismissive of other methods of empirical analyses and much too quick to compare her trials to drug trials.   The biggest problem with randomized trials as practiced by Duflo and others is their lack of generalizability and the fact that they are not true laboratory experiments.  To use an example from the video, just because a few villages in India respond to a particular incentive (some extra lentils) doesn't mean that the same will be true all over the world, or to the same degree.  Also, in drugs trials they have a placebo to account for the fact that people understand that they are being tested and might behave differently because if it, not so in these economics experiments.  Finally, the biological mechanisms within the human body are pretty similar all over the world, but the market, social and cultural structures humans inhabit are different everywhere, so while drugs trials are pretty general, these experiments are not.  I am not sure why Duflo feels the need to be so dismissive of other approaches, but I think it is not very helpful - there is still a lot to be learned from natural experiments and ex-post data analyses when done carefully.  In all of these things, the social scientist must, above all else, remain humble and understand the limitations of all approaches.

Despite all of my complaints, her work is very important and very interesting.  The video is only about 16 minutes long and is pretty engaging.  Enjoy.

Economist's Notebook: Randomized Experiments

Economics is a social science, meaning most of what we study cannot be replicated in a laboratory. We have to find causal links in data that are contaminated by lots of real-world complication and interconnectedness. We have become exceedingly good at doing so, but our conclusions will never rise to the level of "truth." We can propose hypotheses and fail to falsify them, but generally no more.

Often we have to deal with the fact that we don't have the counter-factual. Case in point: how will we know how effective federal fiscal stimulus was? Answer: we won't. We will never know what would have happened in the absence of federal fiscal stimulus.

Randomized experiments therefore are the holy grail of many economists. They are not laboratory experiments (meaning you still can't control all environmental variables), but at least you can create a counter-factual. Often we try to use real world data to estimate treatment and control groups but they are not random. For example, we know that college graduates make a lot more money that those with only a high school diploma on average but does this tell us the extra income the education itself commands? No, because the population of college graduate is a non-random sample of people in general: they are probably richer, better students, more highly motivated, etc. So using average differences will bias the inference of the value of the college education because part of this is due to self-selection into college.

Randomization then, allows the researcher to assume that the people who are effected by a variable (a college education say) are no different than those that don't. The problem in economics (like medicine) is that we are often talking about policies that affect peoples lives and welfare and randomizing treatment for the purposes of study is of questionable ethics, and more importantly for economics, is politically untenable.

It is happening more and more, however, as economic researchers get more involved in policy design and roll-out. It is often possible to roll-out a new policy in phases and in so doing randomize treatment and control groups, but it is still hard and as the control groups usually end up getting treated eventually, it gives you only a short-term view.

I say all this because yesterday The Oregonian published a fascinating article on a unique randomization event that happened in Oregon by Don Colburn. The essence is that Oregon ran short of money and randomly picked from a population a sub-group who got Medicaid health insurance. This allows for a careful study of how insurance affects behavior. Turns out in predictable ways, but the real value is in estimating the marginal behavior: e.g., by how much do doctor and hospital visits decline when the patient has to pay, how much is spent on medical care and how do health outcomes differ in relation to the first two? The results will have huge utility to those contemplating wholesale changes to our nation's health system.