Under the unassuming title of “Behavioural Responses and the Impact of New Agricultural Technologies: Evidence from a Double-Blind Field Experiment in Tanzania”, Erwin Bulte, , , , and have delivered probably the most serious challenge yet on the theology of the Church of the Randomized Controlled Trial.
For years, people have known that RCTs in Development Economics have a serious external validity problem: just because poor farmers in Kenya react in a certain way to, say, a given incentive to use fertilizer, that’s no reason to suppose that farmers in Guatemala, or Nepal or anywhere else will. As a result rigorous evidence isn’t, to quote Pritchett’s enviable knack for a snappy headline.
So long as that was the standard rap against RCTs, the randomistas just about got away with it via their claim to superior internal validity: they can say things about cause and effect that are true in a way other economists could only dream of.
And yet, there was always a discordant note in the RCT literature. The term itself, as well as its epistemic claims, are straightforwardly borrowed from medical research. But in medical research it’s not enough for a trial to be randomized and controlled. It generally has to be double-blind, as well, to aspire to “gold standard” status.
Of course, in most development settings, there’s no viable way to make trials double-blind: presumably, if you only pretend to give that Kenyan farmer a way to save for fertilizer, but you don’t really, the guy’s going to notice.
So the “double-blind” bit of the medical analogy is quietly dropped and swept under the epistemic rug, in hopes nobody will notice…or at least it was, until Bulte and his collaborators realized seeds have some interesting things in common with pills, and so if you can double-blind a medicine trial, you can probably double-blind a seed trial, too.
So they deviously ran an open RCT comparing traditional and improved cowpea seeds alongside a double-blind RCT testing the same thing. Their results are deeply troubling for Banerjee-and-Dufloites.
In the open RCT, Tanzanian cowpea farmers who knew they getting improved seed easily outperformed farmers who knew they were getting traditional seed. But in the double-blind study, farmers who weren’t told whether the seed they got was improved or not performed just as well whether that seed they got was improved or traditional.
In fact, farmers who used traditional seed without knowing it did just as well as farmers who used improved seed, whether they knew it or not. Only farmers who knew the seed they were given wasn’t improved lagged behind in productivity.
This gap between the results of the open and the double-blind RCTs raises deeply troubling questions for the whole field. If, as Bulte et al. surmise, virtually the entire performance boost arises from knowing you’re participating in a trial, believing you may be using a better input, and working harder as a result, then all kinds of RCT results we’ve taken as valid come to look very shaky indeed.
Of course, this is just one study, and so until it’s replicated we’d probably do best not to get too too excited. The pre-publication version of the paper came under some heavy fire, which was perhaps to be expected.
It does strike me as problematic that, while the study was done on cowpea seeds, cowpeas were a secondary crop for all farmers involved, and the productivity gap between improved and traditional cowpea seeds isn’t comparable to the gap in maize or wheat. Replication is badly needed here, and preferably in a study dealing with a crop more central to farmers’ livelihood strategies.
Still, the study is an instant landmark: a gauntlet thrown down in front of the large and growing RCT-Industrial Complex. At the very least, it casts serious doubt on the automatic presumption of internal validity that has long attached to open RCTs. And without that presumption, what’s left, really?