Causality, RCTs and Impact Evaluations

Early August 2012 I was briefly in Congo (slightly more here). One reason for this was to give a presentation to the development community in Bukavu about conducting evaluations. The presentation deals with, among others:

• Why RCT's are so important to get at causality;
• The limitations of RCTs;
• Alternatives to RCTs;
• Techniques to get at sensitive questions.

The slides are not technical. They are also not very clear on their own because I prefer to write little on slides when giving presentations. The plan was to add more info before posting them, but I haven't had the time yet. So with the idea that something is better then nothing, here we go (you have to scroll down a bit):

Causality, RCTs and Impact Evaluations

 

Also, if you have any questions, feel free to send me an email. 

 

Random

I'm about to take a plane to Berlin, and then tomorrow back to NYC! Hereby a quick post with some links:

• Notes from NASA's Curiosity on Mars. "I've got a nuclear-powered laser and control of an entire planet… so I'm essentially a Bond villain. #suckit007. Smart little fellow. More here (h/t Ali);
• "DRC: Bushmeat blamed for Ebola outbreak". Mmm. Maybe I should cut down on eating monkey. Old blogpost here, and an old picture of my bike while doing fieldwork in Maniema, Congo (FYI: I only ate it once):

• Yesterday I watched "Inside Job" by Charles Ferguson. A 2011 documentary about the current financial crisis. It's a must-watch. Trailer:


• Last week, OCHA released 500 free humanitarian symbols on both ReliefWeb and the Noun Project websites. More here. This is "carjacking":

Wonderful Days (also in IKEA) by Charly Lownoise and Mental Theo

That little bit of credibility that I have build with the last 300+ posts will be gone after this one -- but I just have to. The song of my youth ("Wonderful Days" by Charly Lownoise and Mental Theo) on xylofoon (h/t Lucas):

 

For those that did not had such a sophisticated youth as I had, hereby the 'real' version.

And why does the following never happen when I'm at the IKEA?

Migration (and cellphones).

Partly for work but mostly as a treat I often read articles in Science, Nature, PNAS, etc. As you know I'm interested in "migration", and also have been working on a phone-based project in Congo (more here). How cool is it to combine both? Well, this cool:

Most severe disasters cause large population movements. These movements make it difficult for relief organizations to efficiently reach people in need. Understanding and predicting the locations of affected people during disasters is key to effective humanitarian relief operations and to long-term societal reconstruction. We col- laborated with the largest mobile phone operator in Haiti (Digicel) and analyzed the movements of 1.9 million mobile phone users during the period from 42 d before, to 341 d after the devastating Haiti earthquake of January 12, 2010. Nineteen days after the earthquake, population movements had caused the population of the capital Port-au-Prince to decrease by an estimated 23%. Both the travel distances and size of people’s movement trajectories grew after the earthquake. These findings, in combination with the disorder that was present after the disaster, suggest that people’s movements would have become less predictable. Instead, the predictability of people’s trajectories remained high and even in- creased slightly during the three-month period after the earth- quake. Moreover, the destinations of people who left the capital during the first three weeks after the earthquake was highly cor- related with their mobility patterns during normal times, and spe- cifically with the locations in which people had significant social bonds. For the people who left Port-au-Prince, the duration of their stay outside the city, as well as the time for their return, all followed a skewed, fat-tailed distribution. The findings suggest that population movements during disasters may be significantly more predictable than previously thought.

 

From Lu, X., Bengtsson, L., & Holme, P. (2012). Predictability of Population Displacement After the 2010 Haiti Earthquake. In Proceedings of the National Academy of Sciences.

Reading on Migration:

Btw, there is quite a bit of reading out there on migration (and with a wide variance in quality). Some journals below (with their impact factor if I could find it):

• International Migration Review (1.149). By the Center for Migration Studies
• Journal of Refugee Studies (1.191). By the Refugee Studies Centre, University of Oxford
• International Migration (0.773) By the IOM (International Organization for Migration)
• Journal of International Migration and Integration
• Migration Letters
• Journal of Immigrant & Refugee Studies

I also regularly check the website of the Internal Displacement Monitoring Centre, and Forced Migration Online should be in this list as well.

A Few Hours of Solid Entertainment.

While entering data from the Congo I like to have a tv-series on in the background. I've already watched "Peepshow", "The Young Ones", and "Bottom" too many times, and "Mr. Bean" doesn't say very much, so the last few weeks it has been "I'm Alain Partridge", "Sax" or " "Men Behaving Badly". It seems I have a preference for type of humor on the other side of the North Sea:

I'm Alain Partridge:

Sax:

Men Behaving Badly:

Random Things.

Four very random things:

1. Some solid Irish commentary at the London Olympics. Despite the bad video quality, it's definitely worth watching. (h/t Pierce)

2. This is lovely. CNN's Soledad O'Brien taking on somebody from the Romney campaign.

3. Beautiful photos at WIRED: Mars-Inspired Art as commissioned by NASA:

4. We've got -- yet again -- elections coming up in the Netherlands: September 12. There is a website that helps Dutch people: Kieswijzer. Based on your reply to 30 statements it will tell you what party to vote for. Last elections over 4 million people made use of this website -- almost 1/3rd of the Dutch electorate! Instead of this being great, isn't it really very scary. Our decisions depending on a reply to only 30 statements and an algorithm that we don't know. It reminded me of a thought-provocing article in NewScientist how algorithms are -- without us noticing it -- taking over more and more of our lives.

The Power of Introverts.

Whenever I'm alone reading a book or staring in front of me... Please do not disturb! It better for all of us. :) An interesting presentation by Susan Cain about the power introverts:

ThreeHundredFifty Faces From Congo.

How to thank the many Congolese for the information they have given me over the last four years so I could learn about their lives, and write a dissertation? How to thank these people with whom I have lived with — sometimes for months — while conducting fieldwork in Eastern Congo? I hereby present 350 out of 420 pictures we recently took for a lab-in-the-field experiment (more here).

Via these pictures I hope to say thanks. When going through the slideshow please do two things. First, look at the backgrounds to get a glimpse into their extraordinary lives. Texts that are written on the wall and doors, the public goods in front of which the pictures are made, the election posters to decorate the livingrooms, the children that peer in via the window while the picture is taken, etc. Secondly, look into their eyes. Some show the worry and hardship the Congolese face in their daily lives. But for others the eyes also speak of the extraordinary warmth, the strength among the women and the joy I found among many of them.

I had the honor and pleasure to live closely with these people for an extended period of time, and I hope this is a good way to share these experiences. Click the following link to start the slideshow:

ThreeHundredFifty Faces From Congo

If you want an image in full-size, feel free to send me an email.

An Idiot Abroad.

No, not a post about myself. A series by Ricky Gervais. Karl is visiting the seven world wonders. Good fun! (h/t Roy).

A HUNDRED FACES FROM CONGO - Preview

How to thank the many Congolese for the information they have given me over the last four years so I could learn about their lives, and write a dissertation? How to thank these people with whom I have lived with -- sometimes for months -- while conducting fieldwork in Eastern Congo?

For a lab-in-the-field experiment in which we hope to learn about migrant-native networks (more here) pictures were taken of 420 villagers -- sixteen in six villages and eighteen in eighteen other villagers. These people were randomly selected in (on their turn) randomly selected villages, and are thus the literally representative of the area we work in Eastern Congo. In upcoming days I plan to upload a selection of one hundred of these pictures on my website.

I was inspired to do this by an exhibition Raul -- close friend and way too smart co-author and colleague -- has at the M55 Art Gallery in New York from July 25th to August 11th 2012. During his trips to Congo he makes beautiful pictures, which include the ones he will show at the exhibition (see here). For some other beautiful pictures please have a look here. I am unfortunately in the Netherlands and can't make it to the exhibition, but I strongly recommend going -- tomorrow is the last day.

Before posting the four pictures below (and the other 96 still to come) I thought carefully whether this could be done at all -- the villagers did not give me permission to present them publicly. But... First, I do not connect (sensative) information to these pictures. There is also no way somebody can find these people back in Congo -- I will not tell you their names, or in which villages I collected my data. Secondly, the reason to do this is to say thanks to these villagers, and by doing so provide a glance into their extraordinary lives; something I think is very important people should learn about. I had the honor and pleasure to live closely with these people for an extended period of time, and I hope this is a good way to share these experiences. Finally, although I didn't ask them for permission I have spent much time with many of them and know they would welcome this. If you disagree though and have good reasons, let me know.

Thus, in upcoming days I plan to show a selection of one hundred of the 420 pictures on my website under the title:

"A HUNDRED FACES FROM CONGO"

Not only are the people in these pictures beautiful, their eyes speak. Some show the worry and hardship the Congolese go through in their daily lives. But for others the eyes also speak of the extraordinary warmth and joy I found among many of them. Hereby a small preview:

Congo Trip #7: The Voyage.

So I just came back from a 5.5 day trip to Congo -- of which I travelled more than halve. Rather nuts, but the trip was good fun though.
Getting to Bukavu
I had already visited Brussels to get a Congo-visa. And via internet I obtained a Rwandan entry-visa. Monday afternoon around 4pm my little brother dropped me off at Woerden trainstation from where I took a train to Schiphol. And from there by plane: Amsterdam to Frankfurt, to Addis Ababa, to Kigali. The plan was to prepare for my time in the Congo (so much to do), and to sleep as much as possible because I knew the following few days would be tough. I failed miserably. In the plane to Addis I sat next to a very nice girl from Vienna. We talked for the whole flight (more than 6 hours), and had a coffee in Addis while we waited for our next flights (me to Kigali, she to Lusaka). Once in Kigali I got myself a taxi. Normally I take a bus (5,200 Rwandan francs, around $10), but given I had so little time in Bukavu the risk of arriving late at the border could not be taken so the mode of transport was taxi ($180!). Once across the border, I got myself a motorbike and arrived around 8pm at the hotel. 28 hours in total: I think a record.

And back to the Netherlands
The trip back went smoothly as well. I left Bukavu around noon. Given that my plane would leave from Kigali only at 3am I had enough time to buss myself to Kigali. Unfortunately, upon arrival at the Congo-DRC border all busses were full. Shit! Luckily I had met a friendly aid-worker a few minutes before who gave me a lift to the main bus-station of Kamembe, Rwanda, from where I was able to get a bus to Kigali. It was a lovely ride and, because this trip was booked rather abruptly, instead of my field phone I had my smart-phone with me: full with music. Then after waiting for a few hours at Kigali Airport (and working with Macartan and Raul during these hours), it was: Kigali to Addis, to Frankfurt, to Amsterdam. After another train and car-ride, I arrived home around 8pm: 32 hours in total. Again not that bad.



When travelling by road between Kigali and Bukavu you
go through Nyungwe Forest National Park -- East
Africa's largest protected high-altitude rainforest, and
at times breath-taking.

Haji Nyanza: the ultimate place to eat goat brochette!
I'm not joking. When in the plane I already
look forward stopping here (and all busses always do).

Not only the park, but also the seas upon seas of tea before

and after the park are gorgeous.

And don't to forget the African sunsets. How I love this continent!

I've been using Rwanda as transit country for more than three years
now. Say what you want about Rwanda: the lack of freedom of
(political) expression, their involvement in Eastern Congo, etc. I agree. 
BUT each time I'm there the country developed yet another
bit further. This is visible even at that short track
between Kigali and Bukavu. The busses are getting better and better
each year. Work is done on the road throughout, km-signs are
constructed, etc. A new bus-station is build at Kamembe. Haji Nyanza
(that place with great goat brochettes) created proper toilets. And... 
in Kigali there are now small lights build into the roads. Each
time I'm there the country has taken a step forward. How in contrast
to Congo. 

Once in Kigali I had to get to the Airport. Best mode of
transport: Motorbike!

Congo Trip #7: Search for Common Ground

No posts for more than a week. Did nothing interesting happen? On the contrary! Last Saturday evening I came back from a whole 2.5 days in Congo: 1.5 days travel to get there, 2.5 days in-country, and 1.5 days again to get back home. Why?

The reason to be in Bukavu was to work with Search For Common Ground (SFCG). As the readers of this blog know the Congo suffers from high levels of sexual violence and other abuses against civilian populations. The instigators of this violence are rebel armed groups, like the FDLR or local militia, but a major perpetrator is also the national army, the Forces Armées de la République Démocratique du Congo (FARDC). Search is the organization in Congo that works directly with the Congolese military to decrease these abuses. They do this by a set of social norms-changing interventions: cinema, distribution of comic books, sensabilisation sessions that include role-playing games, participatory theater, joint military-civilian activities, etc.

To date, none of the interventions that are intended to directly address abuses against civilian populations has been seriously evaluated (read: RCT). Important questions remain: Is this violence due to lack of community ties? Lack of monitoring? Because soldiers are underpaid? Because of social norms explanations? Lack of access to justice? And the body of evidence accumulated about what works is very weak. However, there exists evidence that social norms-changing interventions can be effective in other contexts, addressing related problems of prejudice (see for example, work by Betsy Paluck). Combined with the importance of the topic SFCG and us (Macartan, Raul and I) thought this to be a potential program for an solid evaluation.

Last Monday (yesterday) there was a deadline for a 3ie grant application -- 3ie is an organization that funds impact evaluations (they for example funded our TUUNGANE study with the International Rescue Committee). To learn about the causal impact of the SFCG program we want to conduct an RCT evaluation (randomly assigning treatment to units) and thus a lot had to be learned on both sides before we could send a successful application to 3ie:

• What is the treatment? What is the theory of change? Do we and the organization actually belief the program makes a measurable impact, and how is this impact achieved? What are the mechanisms? And can we measure these mechanisms? Do we think some parts of the program to be particularly strong? Does a variation in treatment make sense? 
• What is the unit of analysis? Search works with the military units, but the outcome is at the population level. Is there geographical overlap between the two?
• Is there enough statistical power? And at what military unit level should Search work? To have the largest impact possible, but also keeping into account that more treatment units increases statistical power. The more power the more likely false negatives (there is actually an effect but we didn't find it) are avoided.
• What about spillovers? Many parts of Congo are "military active", and thus military units are moved around a lot. Does this lead to spill-overs? The treatment is information, and that might spread quickly between treatment and control units.
• How to measure abuses by the military? Asking a soldier "Did you rape last week?" and then comparing treatment with control communities, is not very likely to work. So how to measure civilian abuses. Might we implement an SMS-based monitoring system (a la Voix des Kivus) in hundreds of treatment and control villages to learn about levels of abuse?
• Are we ethically ok to conduct an RCT? We should not keep control units away from treatment if over the course of the program we find out that some control military units are abusing to an extend larger than some treatment units. Maybe methods different than RCT are better suited (regression discontinuity, process tracing, etc.)?

It were 2.5 very busy days. First for us to learn about the program. Then for Search to learn about evaluations. And then working together (and also with the Congolese army) to answer the above questions. In the end we decided not to go for the 3ie application yet, but probably we will in the future.

Me learning from an army major about the FARDC's hierarchical structure.

Lots to be learned.

Regions, regiments, battalions, companies, pelotons.
And then differences across provinces.

Movements of companies within  regiments: Does

the FARDC keep track of these? How far are

peletons geo-graphically separated from each

other? Etc.

Heading to DRC this Monday and US F1-VISA.

A few hours ago I got myself a plane to Kigali: flying out from Amsterdam this Monday (yes, the day after tomorrow), and arriving Tuesday afternoon. Upon arrival I hope to get myself a bus and head to the Rwanda-Congo border that same day. I'll only be in the Congo for 3 days, I leave again on Friday afternoon (plane leaves early Saturday morning from Kigali). It's a short trip to work together with Search for Common Ground (95% of the time), to give a presentation on how to conduct evaluations, and to meet my team that has been in the field over the last months. Because it's so short I will only stay in Bukavu. No field for me this time. It's a pity, otherwise I could have used the following sleeping back (h/t Simon):

 

One reason to leave quickly again is because on August 6 I have an appointment at the US Embassy in Amsterdam for my F1-visa -- after 5 years my current visa expired. I already filled out the online application, which included the following questions:

1. Are you coming to the United States to engage in prostitution or unlawful commercialized vice or have you been engaged in prostitution or procuring prostitutes within the past 10 years?
2. Have you ever been involved in, or do you seek to engage in, money laundering?
3. Have you ever committed or conspired to commit a human trafficking offense in the United States or outside the United States?
4. Are you the spouse, son, or daughter of an individual who has committed or conspired to commit a human trafficking offense in the United States or outside the United States and have you within the last five years, knowingly benefited from the trafficking activities? 
5. Do you seek to engage in terrorist activities while in the United States or have you ever engaged in terrorist activities?
6. Have you ever been directly involved in the establishment or enforcement of population controls forcing a woman to undergo an abortion against her free choice or a man or a woman to undergo sterilization against his or her free will?

:)

From Brussels to... Congo?

Yesterday morning 5am. Alarm went off. 530am: on my way to Rotterdam Centraal, and from there to Brussels to get a single-entry, one-month visa for the Congo (I know. Again!? More on this later). Depositing the necessary material is between 9-12pm, and the pick-up is at 430pm. Thus a lot of time in Brussels. Of course I had my laptop with me and the grandiose plan to work. Well that didn't work out at all with me being in a city that I haven't explored a lot yet and the sun shining. So I hired a bike for a day [Genious system btw. They have bikes positioned throughout the city. It costed me $1.60 for a day], and spent a day playing the tourist.

Brussels is a great city.

Brussels' Grote Markt

The Worst Mistake in the History of the Human Race.

Agriculture.

...

To science we owe dramatic changes in our smug self-image. Astronomy taught us that our earth isn't the center of the universe but merely one of billions of heavenly bodies. From biology we learned that we weren't specially created by God but evolved along with millions of other species. Now archaeology is demolishing another sacred belief: that human history over the past million years has been a long tale of progress. In particular, recent discoveries suggest that the adoption of agriculture, supposedly our most decisive step toward a better life, was in many ways a catastrophe from which we have never recovered. With agriculture came the gross social and sexual inequality, the disease and despotism, that curse our existence. 

...

One straight forward example of what paleopathologists have learned from skeletons concerns historical changes in height. Skeletons from Greece and Turkey show that the average height of hunger-gatherers toward the end of the ice ages was a generous 5' 9'' for men, 5' 5'' for women. With the adoption of agriculture, height crashed, and by 3000 B. C. had reached a low of only 5' 3'' for men, 5' for women. By classical times heights were very slowly on the rise again, but modern Greeks and Turks have still not regained the average height of their distant ancestors. 

....

There are at least three sets of reasons to explain the findings that agriculture was bad for health. First, hunter-gatherers enjoyed a varied diet, while early fanners obtained most of their food from one or a few starchy crops. The farmers gained cheap calories at the cost of poor nutrition, (today just three high-carbohydrate plants -- wheat, rice, and corn -- provide the bulk of the calories consumed by the human species, yet each one is deficient in certain vitamins or amino acids essential to life.) Second, because of dependence on a limited number of crops, farmers ran the risk of starvation if one crop failed. Finally, the mere fact that agriculture encouraged people to clump together in crowded societies, many of which then carried on trade with other crowded societies, led to the spread of parasites and infectious disease. (Some archaeologists think it was the crowding, rather than agriculture, that promoted disease, but this is a chicken-and-egg argument, because crowding encourages agriculture and vice versa.) Epidemics couldn't take hold when populations were scattered in small bands that constantly shifted camp. Tuberculosis and diarrheal disease had to await the rise of farming, measles and bubonic plague the appearnce of large cities. 

....

Besides malnutrition, starvation, and epidemic diseases, farming helped bring another curse upon humanity: deep class divisions. Hunter-gatherers have little or no stored food, and no concentrated food sources, like an orchard or a herd of cows: they live off the wild plants and animals they obtain each day. Therefore, there can be no kings, no class of social parasites who grow fat on food seized from others. Only in a farming population could a healthy, non-producing elite set itself above the disease-ridden masses. Skeletons from Greek tombs at Mycenae c. 1500 B. C. suggest that royals enjoyed a better diet than commoners, since the royal skeletons were two or three inches taller and had better teeth (on the average, one instead of six cavities or missing teeth). Among Chilean mummies from c. A. D. 1000, the elite were distinguished not only by ornaments and gold hair clips but also by a fourfold lower rate of bone lesions caused by disease.

...

From an 1987 article by Jarod Diamond. Not sure how it fits in with his "Guns, Germs and Steel".

Their Brussels, Our Netherlands. Wrong!

On 12 September 2012 we'll have, yet again, elections in the Netherlands. Today I read the program of the Partij Voor de Vrijheid (PVV) - indeed, the infamous Dutch right-wing party of Geert Wilders. The people that read this blog know that I'm against almost all that this party stands for: anti-Islam, anti-immigration, anti-development aid, anti-Europe, etc.



"Party for Freedom. Their Brussels, our Netherlands"

Their program can be found here (only in Dutch). It's quite painful to read. It's very populistic. An important topic again is "Islam = Evil". But this time the over-arching theme is "Europe = Evil". In brief, the party program read like: Let's-all-put-our-heads-in-the-sand-build-a-big-fence-around-the-Netherlands-and-not-acknowledge-what-has-happened-in-the-world-over-the-last-decades.

This is not a party for the future.

How to Stop Atrocities?

Easy. Just present these slides.

Verandering van Spijs Doet Eten.

In line with the Dutch saying (literally translated): "Change of food makes one eat"[1], I just changed the layout of my blog. Also, it was about time after 3 years. It has a bit more serious appearance. I do not think this will lead to more serious posts though. :)

[1] Originally from "Varietas delectat" (variety delights). 

Just Post It.

I argue that journals should require authors to post the raw data supporting their published results. I illustrate some of the benefits of doing so by describing two cases of fraud I identified exclusively through statistical analysis of reported means and standard deviations. Analyses of the raw data provided important confirmation of the initial suspicions, ruling out benign explanations (e.g., reporting errors; unusual distributions), identifying additional signs of fabrication, and also ruling out one of the suspected fraudster’s explanations for his anomalous results. If we want to reduce fraud, we need to require authors to post their raw data.

 

A new working paper by Uri Simonsohn: "Just Post it: The Lesson from Two Cases of Fabricated Data Detected by Statistics Alone". Please find a link to the paper here. 

Bla bla bla.

The previous post reminded me of the training Raul and I did in the Congo in the summer of 2010 (more here). During the training the students used a wide variety of ways to start a sentence. Anyone who has worked in the DRC knows about this: there are often a lot of words necessary before getting to the point. You're ready? Here we go: 

• "Je  voudrais donner un petit clarification… XXXX"
•  "Merci pour me donner le parole… XXXX"
•  "J'ai un petit preoccupation… XXXX"
•  "Je voudrais savoir dans le cadre de … XXXX"
•  "Je suis dans un confusion total… XXXX"
•  "Ca me complique un peu... XXXX"
•  "Je voudrais me assure un chose… XXXX"
•  "Je me nager dans l'eau"
•  "C'est un engagement pikante"
•  "C'est un question que a demande beaucoup de tactique… XXXX"
•  "C'est un question que necessite des problemes… XXXX"
•  "Je veux simplement eclaircir la nuit"
•  "Cette question n'a pas de raison d'etre"'
•  "Le question risque de avoir un petit probleme au  niveau… XXXX"

Genious!