Causality, RCTs and Impact Evaluations

Early August 2012 I was briefly in Congo (slightly more here). One reason for this was to give a presentation to the development community in Bukavu about conducting evaluations. The presentation deals with, among others:

• Why RCT's are so important to get at causality;
• The limitations of RCTs;
• Alternatives to RCTs;
• Techniques to get at sensitive questions.

The slides are not technical. They are also not very clear on their own because I prefer to write little on slides when giving presentations. The plan was to add more info before posting them, but I haven't had the time yet. So with the idea that something is better then nothing, here we go (you have to scroll down a bit):

Causality, RCTs and Impact Evaluations

 

Also, if you have any questions, feel free to send me an email. 

 

Random

I'm about to take a plane to Berlin, and then tomorrow back to NYC! Hereby a quick post with some links:

• Notes from NASA's Curiosity on Mars. "I've got a nuclear-powered laser and control of an entire planet… so I'm essentially a Bond villain. #suckit007. Smart little fellow. More here (h/t Ali);
• "DRC: Bushmeat blamed for Ebola outbreak". Mmm. Maybe I should cut down on eating monkey. Old blogpost here, and an old picture of my bike while doing fieldwork in Maniema, Congo (FYI: I only ate it once):

• Yesterday I watched "Inside Job" by Charles Ferguson. A 2011 documentary about the current financial crisis. It's a must-watch. Trailer:


• Last week, OCHA released 500 free humanitarian symbols on both ReliefWeb and the Noun Project websites. More here. This is "carjacking":

Wonderful Days (also in IKEA) by Charly Lownoise and Mental Theo

That little bit of credibility that I have build with the last 300+ posts will be gone after this one -- but I just have to. The song of my youth ("Wonderful Days" by Charly Lownoise and Mental Theo) on xylofoon (h/t Lucas):

 

For those that did not had such a sophisticated youth as I had, hereby the 'real' version.

And why does the following never happen when I'm at the IKEA?

Migration (and cellphones).

Partly for work but mostly as a treat I often read articles in Science, Nature, PNAS, etc. As you know I'm interested in "migration", and also have been working on a phone-based project in Congo (more here). How cool is it to combine both? Well, this cool:

Most severe disasters cause large population movements. These movements make it difficult for relief organizations to efficiently reach people in need. Understanding and predicting the locations of affected people during disasters is key to effective humanitarian relief operations and to long-term societal reconstruction. We col- laborated with the largest mobile phone operator in Haiti (Digicel) and analyzed the movements of 1.9 million mobile phone users during the period from 42 d before, to 341 d after the devastating Haiti earthquake of January 12, 2010. Nineteen days after the earthquake, population movements had caused the population of the capital Port-au-Prince to decrease by an estimated 23%. Both the travel distances and size of people’s movement trajectories grew after the earthquake. These findings, in combination with the disorder that was present after the disaster, suggest that people’s movements would have become less predictable. Instead, the predictability of people’s trajectories remained high and even in- creased slightly during the three-month period after the earth- quake. Moreover, the destinations of people who left the capital during the first three weeks after the earthquake was highly cor- related with their mobility patterns during normal times, and spe- cifically with the locations in which people had significant social bonds. For the people who left Port-au-Prince, the duration of their stay outside the city, as well as the time for their return, all followed a skewed, fat-tailed distribution. The findings suggest that population movements during disasters may be significantly more predictable than previously thought.

 

From Lu, X., Bengtsson, L., & Holme, P. (2012). Predictability of Population Displacement After the 2010 Haiti Earthquake. In Proceedings of the National Academy of Sciences.

Reading on Migration:

Btw, there is quite a bit of reading out there on migration (and with a wide variance in quality). Some journals below (with their impact factor if I could find it):

• International Migration Review (1.149). By the Center for Migration Studies
• Journal of Refugee Studies (1.191). By the Refugee Studies Centre, University of Oxford
• International Migration (0.773) By the IOM (International Organization for Migration)
• Journal of International Migration and Integration
• Migration Letters
• Journal of Immigrant & Refugee Studies

I also regularly check the website of the Internal Displacement Monitoring Centre, and Forced Migration Online should be in this list as well.

A Few Hours of Solid Entertainment.

While entering data from the Congo I like to have a tv-series on in the background. I've already watched "Peepshow", "The Young Ones", and "Bottom" too many times, and "Mr. Bean" doesn't say very much, so the last few weeks it has been "I'm Alain Partridge", "Sax" or " "Men Behaving Badly". It seems I have a preference for type of humor on the other side of the North Sea:

I'm Alain Partridge:

Sax:

Men Behaving Badly:

Random Things.

Four very random things:

1. Some solid Irish commentary at the London Olympics. Despite the bad video quality, it's definitely worth watching. (h/t Pierce)

2. This is lovely. CNN's Soledad O'Brien taking on somebody from the Romney campaign.

3. Beautiful photos at WIRED: Mars-Inspired Art as commissioned by NASA:

4. We've got -- yet again -- elections coming up in the Netherlands: September 12. There is a website that helps Dutch people: Kieswijzer. Based on your reply to 30 statements it will tell you what party to vote for. Last elections over 4 million people made use of this website -- almost 1/3rd of the Dutch electorate! Instead of this being great, isn't it really very scary. Our decisions depending on a reply to only 30 statements and an algorithm that we don't know. It reminded me of a thought-provocing article in NewScientist how algorithms are -- without us noticing it -- taking over more and more of our lives.

The Power of Introverts.

Whenever I'm alone reading a book or staring in front of me... Please do not disturb! It better for all of us. :) An interesting presentation by Susan Cain about the power introverts:

ThreeHundredFifty Faces From Congo.

How to thank the many Congolese for the information they have given me over the last four years so I could learn about their lives, and write a dissertation? How to thank these people with whom I have lived with — sometimes for months — while conducting fieldwork in Eastern Congo? I hereby present 350 out of 420 pictures we recently took for a lab-in-the-field experiment (more here).

Via these pictures I hope to say thanks. When going through the slideshow please do two things. First, look at the backgrounds to get a glimpse into their extraordinary lives. Texts that are written on the wall and doors, the public goods in front of which the pictures are made, the election posters to decorate the livingrooms, the children that peer in via the window while the picture is taken, etc. Secondly, look into their eyes. Some show the worry and hardship the Congolese face in their daily lives. But for others the eyes also speak of the extraordinary warmth, the strength among the women and the joy I found among many of them.

I had the honor and pleasure to live closely with these people for an extended period of time, and I hope this is a good way to share these experiences. Click the following link to start the slideshow:

ThreeHundredFifty Faces From Congo

If you want an image in full-size, feel free to send me an email.

An Idiot Abroad.

No, not a post about myself. A series by Ricky Gervais. Karl is visiting the seven world wonders. Good fun! (h/t Roy).

A HUNDRED FACES FROM CONGO - Preview

How to thank the many Congolese for the information they have given me over the last four years so I could learn about their lives, and write a dissertation? How to thank these people with whom I have lived with -- sometimes for months -- while conducting fieldwork in Eastern Congo?

For a lab-in-the-field experiment in which we hope to learn about migrant-native networks (more here) pictures were taken of 420 villagers -- sixteen in six villages and eighteen in eighteen other villagers. These people were randomly selected in (on their turn) randomly selected villages, and are thus the literally representative of the area we work in Eastern Congo. In upcoming days I plan to upload a selection of one hundred of these pictures on my website.

I was inspired to do this by an exhibition Raul -- close friend and way too smart co-author and colleague -- has at the M55 Art Gallery in New York from July 25th to August 11th 2012. During his trips to Congo he makes beautiful pictures, which include the ones he will show at the exhibition (see here). For some other beautiful pictures please have a look here. I am unfortunately in the Netherlands and can't make it to the exhibition, but I strongly recommend going -- tomorrow is the last day.

Before posting the four pictures below (and the other 96 still to come) I thought carefully whether this could be done at all -- the villagers did not give me permission to present them publicly. But... First, I do not connect (sensative) information to these pictures. There is also no way somebody can find these people back in Congo -- I will not tell you their names, or in which villages I collected my data. Secondly, the reason to do this is to say thanks to these villagers, and by doing so provide a glance into their extraordinary lives; something I think is very important people should learn about. I had the honor and pleasure to live closely with these people for an extended period of time, and I hope this is a good way to share these experiences. Finally, although I didn't ask them for permission I have spent much time with many of them and know they would welcome this. If you disagree though and have good reasons, let me know.

Thus, in upcoming days I plan to show a selection of one hundred of the 420 pictures on my website under the title:

"A HUNDRED FACES FROM CONGO"

Not only are the people in these pictures beautiful, their eyes speak. Some show the worry and hardship the Congolese go through in their daily lives. But for others the eyes also speak of the extraordinary warmth and joy I found among many of them. Hereby a small preview:

Congo Trip #7: The Voyage.

So I just came back from a 5.5 day trip to Congo -- of which I travelled more than halve. Rather nuts, but the trip was good fun though.
Getting to Bukavu
I had already visited Brussels to get a Congo-visa. And via internet I obtained a Rwandan entry-visa. Monday afternoon around 4pm my little brother dropped me off at Woerden trainstation from where I took a train to Schiphol. And from there by plane: Amsterdam to Frankfurt, to Addis Ababa, to Kigali. The plan was to prepare for my time in the Congo (so much to do), and to sleep as much as possible because I knew the following few days would be tough. I failed miserably. In the plane to Addis I sat next to a very nice girl from Vienna. We talked for the whole flight (more than 6 hours), and had a coffee in Addis while we waited for our next flights (me to Kigali, she to Lusaka). Once in Kigali I got myself a taxi. Normally I take a bus (5,200 Rwandan francs, around $10), but given I had so little time in Bukavu the risk of arriving late at the border could not be taken so the mode of transport was taxi ($180!). Once across the border, I got myself a motorbike and arrived around 8pm at the hotel. 28 hours in total: I think a record.

And back to the Netherlands
The trip back went smoothly as well. I left Bukavu around noon. Given that my plane would leave from Kigali only at 3am I had enough time to buss myself to Kigali. Unfortunately, upon arrival at the Congo-DRC border all busses were full. Shit! Luckily I had met a friendly aid-worker a few minutes before who gave me a lift to the main bus-station of Kamembe, Rwanda, from where I was able to get a bus to Kigali. It was a lovely ride and, because this trip was booked rather abruptly, instead of my field phone I had my smart-phone with me: full with music. Then after waiting for a few hours at Kigali Airport (and working with Macartan and Raul during these hours), it was: Kigali to Addis, to Frankfurt, to Amsterdam. After another train and car-ride, I arrived home around 8pm: 32 hours in total. Again not that bad.



When travelling by road between Kigali and Bukavu you
go through Nyungwe Forest National Park -- East
Africa's largest protected high-altitude rainforest, and
at times breath-taking.

Haji Nyanza: the ultimate place to eat goat brochette!
I'm not joking. When in the plane I already
look forward stopping here (and all busses always do).

Not only the park, but also the seas upon seas of tea before

and after the park are gorgeous.

And don't to forget the African sunsets. How I love this continent!

I've been using Rwanda as transit country for more than three years
now. Say what you want about Rwanda: the lack of freedom of
(political) expression, their involvement in Eastern Congo, etc. I agree. 
BUT each time I'm there the country developed yet another
bit further. This is visible even at that short track
between Kigali and Bukavu. The busses are getting better and better
each year. Work is done on the road throughout, km-signs are
constructed, etc. A new bus-station is build at Kamembe. Haji Nyanza
(that place with great goat brochettes) created proper toilets. And... 
in Kigali there are now small lights build into the roads. Each
time I'm there the country has taken a step forward. How in contrast
to Congo. 

Once in Kigali I had to get to the Airport. Best mode of
transport: Motorbike!

Congo Trip #7: Search for Common Ground

No posts for more than a week. Did nothing interesting happen? On the contrary! Last Saturday evening I came back from a whole 2.5 days in Congo: 1.5 days travel to get there, 2.5 days in-country, and 1.5 days again to get back home. Why?

The reason to be in Bukavu was to work with Search For Common Ground (SFCG). As the readers of this blog know the Congo suffers from high levels of sexual violence and other abuses against civilian populations. The instigators of this violence are rebel armed groups, like the FDLR or local militia, but a major perpetrator is also the national army, the Forces Armées de la République Démocratique du Congo (FARDC). Search is the organization in Congo that works directly with the Congolese military to decrease these abuses. They do this by a set of social norms-changing interventions: cinema, distribution of comic books, sensabilisation sessions that include role-playing games, participatory theater, joint military-civilian activities, etc.

To date, none of the interventions that are intended to directly address abuses against civilian populations has been seriously evaluated (read: RCT). Important questions remain: Is this violence due to lack of community ties? Lack of monitoring? Because soldiers are underpaid? Because of social norms explanations? Lack of access to justice? And the body of evidence accumulated about what works is very weak. However, there exists evidence that social norms-changing interventions can be effective in other contexts, addressing related problems of prejudice (see for example, work by Betsy Paluck). Combined with the importance of the topic SFCG and us (Macartan, Raul and I) thought this to be a potential program for an solid evaluation.

Last Monday (yesterday) there was a deadline for a 3ie grant application -- 3ie is an organization that funds impact evaluations (they for example funded our TUUNGANE study with the International Rescue Committee). To learn about the causal impact of the SFCG program we want to conduct an RCT evaluation (randomly assigning treatment to units) and thus a lot had to be learned on both sides before we could send a successful application to 3ie:

• What is the treatment? What is the theory of change? Do we and the organization actually belief the program makes a measurable impact, and how is this impact achieved? What are the mechanisms? And can we measure these mechanisms? Do we think some parts of the program to be particularly strong? Does a variation in treatment make sense? 
• What is the unit of analysis? Search works with the military units, but the outcome is at the population level. Is there geographical overlap between the two?
• Is there enough statistical power? And at what military unit level should Search work? To have the largest impact possible, but also keeping into account that more treatment units increases statistical power. The more power the more likely false negatives (there is actually an effect but we didn't find it) are avoided.
• What about spillovers? Many parts of Congo are "military active", and thus military units are moved around a lot. Does this lead to spill-overs? The treatment is information, and that might spread quickly between treatment and control units.
• How to measure abuses by the military? Asking a soldier "Did you rape last week?" and then comparing treatment with control communities, is not very likely to work. So how to measure civilian abuses. Might we implement an SMS-based monitoring system (a la Voix des Kivus) in hundreds of treatment and control villages to learn about levels of abuse?
• Are we ethically ok to conduct an RCT? We should not keep control units away from treatment if over the course of the program we find out that some control military units are abusing to an extend larger than some treatment units. Maybe methods different than RCT are better suited (regression discontinuity, process tracing, etc.)?

It were 2.5 very busy days. First for us to learn about the program. Then for Search to learn about evaluations. And then working together (and also with the Congolese army) to answer the above questions. In the end we decided not to go for the 3ie application yet, but probably we will in the future.

Me learning from an army major about the FARDC's hierarchical structure.

Lots to be learned.

Regions, regiments, battalions, companies, pelotons.
And then differences across provinces.

Movements of companies within  regiments: Does

the FARDC keep track of these? How far are

peletons geo-graphically separated from each

other? Etc.