Bike-o-nomics: Bike Research (Correlation and Causation Redux)

I have held my tongue for a while about some of the bike research that is being reported on by the Oregonian, The Atlantic Cities and others because I did not want to seem to be bashing the research.  Nor do I want to really bash the fact that it is being reported on.  Mostly though I do not want to be branded an enemy of the state by bike fanatics whom I have found at times to be a little less than circumspect about their own cause.  But I do want to point out some serious problems with the way the findings are being presented and discussed.

So, what studies am I talking about? First there was a lot of coverage of the PSU study that found that bikers spent more money in local stores.  Then there was a study from PSU that found that bike commuters were happier, and today I learn from Joseph Rose that the Danish have produced a study that said that bike riding children concentrate better in school.

Now, a few things first: I am an avid biker, I chose to live in a place where I can walk and bike to most things, my kids bike to school almost every day of the school year, rain or shine, my wife, when she worked downtown bike commuted everyday and has led the Walk + Bike to school activities for my children's school, so my family has plenty of bike cred.  Additionally, I want to state again that I have nothing against these studies nor their being reported in local and national media.

What I have a problem with is the way the media (and sometimes the authors themselves) fudge the distinction between what the studies find - some very interesting correlations - and the inappropriate causal interpretation of the results.

Here is an example:

Kelly Clifton has heard this stereotype a number of times: "Cyclists are just a bunch of kids who don’t have any money," says the professor of civil and environmental engineering at Portland State University. "They ride their bikes to a coffee shop, they sit there for four hours with their Macintoshes, they’re not really spending any money."

...Clifton says, "we start to see businesses say, ‘Hey, wait a minute. You’re taking away on-street parking to put in bike lanes, you’re taking away the one parking spot in front of my store to put in a bike corral. I don’t see many bikers around here. So what does this mean for me?" 

...Until now there hasn’t been much empirical evidence to allay such concerns. Clifton and several colleagues have attempted to fill that research gap in a project for the Oregon Transportation Research and Education Consortium.

But the study, though interesting and of plenty of merit in its own right doesn't really address the business owners concerns at all.  The study asked folks exiting a set of selected stores how they got there and how much they spent.  There is a problem of selection here in the way stores were chosen but mostly there is a problem with establishing any sort of counterfactual.  The business owner's concern is about the relative revenue impact of lost car parking and additional bike parking.  I'd like to think it is positive, and I think there is a good chance it is, but the study does not address that at all.

The study asks about the money spent at the time of the interview, in some cases the average was higher for car drivers than bikers (grocery stores) and some times it was lower (smaller stores).  They do a good job controlling for some household characteristics (especially income) but they still end up with some controlled correlations of one variable: spending for that trip.  This also does not address overall spending habits hoe biking correlates with spending or how biking affects consumer behavior.

Now both the author and the reporter are careful not to say that you get any firm answers from this study but they sure do fudge the point and make it appear that you would do fine drawing your own conclusions.  What would I like to see?  A better factual description of the actual study, a frank discussion about what it found and some of the potential conclusions an equally frank discussion about the limits of the study and what you cannot say from the findings.

Here is another example:

A new study by Portland State University urban studies doctoral candidate Oliver Smith found that getting to work via “active transportation” – e.g., under your own power – “increases commute well-being, even when controlling for distance, income and other factors.”

Um, no it doesn't. It finds a correlation between commuting and self-reported happiness. It doesn't say that if you take car commuters and force them to ride bikes to work they would be happier, it says that those that self-selected biking to work instead of driving are also more likely to describe themselves as happy. Full stop.

The fact that you control for income, distance and other factors does not change the fact that this is a correlation from which you simply cannot make a causal statement.  Again, this interpretation is, I hope, true, and certainly excercise and happiness are very, very closely related for myself, but this study does not allow that type of causal interpretation.

Finally, we have this:

Every day outside my son’s Brooklyn school, no matter what the weather, you will see a distinctive pale blue bicycle locked to the rack. It belongs to a 7th-grade girl from a Dutch family whose members have stuck with their traditional practice of riding to school each day, despite finding themselves in the not-so-bike-friendly United States for a few years. This lovely blue city bike was a gift from the parents to their eldest child... 

According to the results of a Danish study released late last year, my Dutch friends are giving their daughter a less tangible but more lasting gift along with that bicycle: the ability to concentrate better. (emphasis mine)

No, no a thousand times no!  According to your interpretation of the results of the study which found that bike riders could concentrate better (a correlation), bike riding improves concentration.  Again, this is likely true but the study was of self-selected bikers (as far as I can tell from second-hand descriptions, the study itself in in Danish) not of a group of students some of whom were randomly chosen to bikes and other who were driven - so no such causal interpretation is appropriate.

So as not to be a scold, here are some helpful hints:

-Report that studies find correlations or say "those who bike also X..."

-Be careful not to say or imply that "biking causes, or leads to, X..."

-If you want to make the link, use language like "it is possible that biking causes X..." but if you do make sure you qualify by saying that the results do not support such a conclusion.

-Yes I know reporters rely on academics to interpret their work, but we are not copyeditors, and since you don't have them anymore anyway, you have to be careful in your use of language not to imply causality when there is none.

-Academics, make sure you tell reporters the limitations of your study so it does not look like you are selling conclusions that your research cannot support.