And how long is your… survey?

When waiting for my plane at JFK last Monday, I was asked to fill out a survey. The US Department of Transport (DOT) wanted to know about the behavior of travelers in order to improve JFK airport. Good initiative, but how useful is a survey like this? It's not random at all! Most people said 'no' when the surveyor asked people to fill out the questionnaire. I said 'yes', but only because I'm doing survey work myself and felt for the surveyor. Let's say that the population consists out of 1% friendly people, and that only friendly people (and the rare confused grad-student) say 'yes' to a survey. Let's say that in this world only friendly people like candy-machines, and that that is what they suggest in the survey as an improvement to the airport. The DOT will thus conclude that "100% of people surveyed want candy-machines" and will spend a lot of money to build candy-machines at JFK. But they’ll build 90% too many of them! It’s a bit stupid example, but you see the point. One needs a proper random sample, and just asking people on the fly is really not doing that.

 

But there is another problem. When I was about 2 minutes into the survey I was no longer interested and, to be very honest, didn't really read more than half of the question before answering. My answers were probably off several times. Now I'm probably particularly quick to lose interest in things, but what does this say about the quality of answers to, for example, lengthy surveys we conduct in Africa for research?

It's our enumerator sitting comfy in that chair. :)

The DOT survey (at least that’s what was written on the front page) takes around 15 minutes. The survey we conducted in the Congo between 2010-2011 with around 10,000 households, took on average two hours! The design can be found here. Now, of course, I'm a particularly impatient person (much more than any African villager I have met). Moreover, our enumerators sit down with the respondents and ask the questions to them instead of the respondent filling out the survey alone. But still, two hours on average per survey cannot do the quality of replies much good.

Of course this was not the first time we thought about this. Two years ago, when creating the design documents, we were very well aware of this potential problem. As a result, we implemented the survey with two variations to learn about this:

1. Random ordering of questions: The first variation was the random ordering of a set of questions. That is, half of the surveys (randomly selected) had sections ordered as X Y Z, and another half of the surveys had the order: X Z Y. 
2. Mandatory break: Second, in half of the surveys (randomly selected) we instituted a mandatory break. We were able to do this because we did not make use of hardcopy surveys, but PDAs. We had programmed the PDA in such a way that at a certain point it would say “Now take a 30 minutes break”, and the PDA would be blocked for 30 minutes. The idea here is that after taking a break the person being interviewed (and the surveyor) would start fresh again improving the quality of the responses. 

Needless to say we added several trick questions to the surveys in order to measure whether people answered carefully. By doing so we can now learn whether the length of the survey is important.