A Torpedo Aimed Straight at H.M.S. Randomista

It can’t be that often that the American Journal of Agricultural Economics publishes a genuinely spectacular result, but their most recent issue is the exception that proves the rule.

Under the unassuming title of “Behavioural Responses and the Impact of New Agricultural Technologies: Evidence from a Double-Blind Field Experiment in Tanzania”, Erwin Bulte, Gonne BeekmanSalvatore Di FalcoJoseph Hellaand Pan Lei have delivered probably the most serious challenge yet on the theology of the Church of the Randomized Controlled Trial.

For years, people have known that RCTs in Development Economics have a serious external validity problem: just because poor farmers in Kenya react in a certain way to, say, a given incentive to use fertilizer, that’s no reason to suppose that farmers in Guatemala, or Nepal or anywhere else will.  As a result rigorous evidence isn’t, to quote Pritchett’s enviable knack for a snappy headline.

So long as that was the standard rap against RCTs, the randomistas just about got away with it via their claim to superior internal validity: they can say things about cause and effect that are true in a way other economists could only dream of.

And yet, there was always a discordant note in the RCT literature. The term itself, as well as its epistemic claims, are straightforwardly borrowed from medical research. But in medical research it’s not enough for a trial to be randomized and controlled. It generally has to be double-blind, as well, to aspire to “gold standard” status.

Of course, in most development settings, there’s no viable way to make trials double-blind: presumably, if you only pretend to give that Kenyan farmer a way to save for fertilizer, but you don’t really, the guy’s going to notice.

So the “double-blind” bit of the medical analogy is quietly dropped and swept under the epistemic rug, in hopes nobody will notice…or at least it was, until Bulte and his collaborators realized seeds have some interesting things in common with pills, and so if you can double-blind a medicine trial, you can probably double-blind a seed trial, too.

So they deviously ran an open RCT comparing traditional and improved cowpea seeds alongside a double-blind RCT testing the same thing. Their results are deeply troubling for Banerjee-and-Dufloites.

In the open RCT, Tanzanian cowpea farmers who knew they getting improved seed easily outperformed farmers who knew they were getting traditional seed. But in the double-blind study, farmers who weren’t told whether the seed they got was improved or not performed just as well whether that seed they got was improved or traditional.

In fact, farmers who used traditional seed without knowing it did just as well as farmers who used improved seed, whether they knew it or not. Only farmers who knew the seed they were given wasn’t improved lagged behind in productivity.

This gap between the results of the open and the double-blind RCTs raises deeply troubling questions for the whole field. If, as Bulte et al. surmise, virtually the entire performance boost arises from knowing you’re participating in a trial, believing you may be using a better input, and working harder as a result, then all kinds of RCT results we’ve taken as valid come to look very shaky indeed.

Of course, this is just one study, and so until it’s replicated we’d probably do best not to get too too excited. The pre-publication version of the paper came under some heavy fire, which was perhaps to be expected.

It does strike me as problematic that, while the study was done on cowpea seeds, cowpeas were a secondary crop for all farmers involved, and the productivity gap between improved and traditional cowpea seeds isn’t comparable to the gap in maize or wheat. Replication is badly needed here, and preferably in a study dealing with a crop more central to farmers’ livelihood strategies.

Still, the study is an instant landmark: a gauntlet thrown down in front of the large and growing RCT-Industrial Complex. At the very least, it casts serious doubt on the automatic presumption of internal validity that has long attached to open RCTs. And without that presumption, what’s left, really?

It’s one of the biggest things keeping Ugandan farmers poor. And nobody talks about it.

My piece on seed counterfeiting is out in The Guardian today. Have a look,

Of the many factors that keep small-scale Ugandan farmers poor, seed counterfeiting may be the least understood. Passing under the radar of the international development sector, a whole illegal industry has developed in Uganda, cheating farmers by selling them seeds that promise high yields but fail to germinate at all – with results that can be disastrous.

Counterfeiting gangs have learned to dye regular maize with the characteristic pinkish orange colour of industrially processed maize seed, duping farmers into paying good money for seed that just won’t grow. The result is a crisis of confidence in commercially available high-yield seed.

59 Reforms Later

Some books just fall into your hands at just the right time, and for me Matt Andrews’s The Limits of Institutional Reform in Development was just such a book. If you’re wondering why international institutions are so damn bad at helping developing countries govern themselves better, it really belongs on your reading list.

But actually, the facts reported are as much of an eye-opener as the argument. The sheer number of internationally-backed institutional reforms your average developing country has gone through startled me. Andrews cites the case of Honduras, which has undertaken fifty-nine separate World Bank backed reforms since 1988. Fifty-nine!

They cover just about everything. Banking, trade, privatization, civil service reform, public financial management, civil service payments, procurement, competition, external auditing, procurement monitoring, results-based management, anti-corruption, participatory budgeting, merit-based hiring and performance based compensation. Several of these, like civil service reforms, are “hardy perennials” that seem to crop up again decade after decade.

The bloat is multidimensional. And notice: that’s just the reforms backed by the World Bank, before we even start talking about the agendas pushed by IMF, IADB, WTO, USAID,  DFID, etc. etc.

It goes without saying that, 59 reforms later, Honduras is just as badly governed as before.

In Andrews’s telling, this volume of partner-driven reform activity is by no means unique to Honduras. The Bank requires all sorts of developing countries to jump through various reform hoops before it’ll approve a loan, so all kinds of countries makes a show of approving the desk-based portion of reforms as a signalling mechanism, with no real commitment or intention to really follow through.

That part sounds about right, though it does rather beg the question of why the Bank keeps falling for it again and again. Then again, bank staffers are compensated for the number of reforms “satisfactorily adopted”, not by the number that actually work, so maybe it’s not that big a mystery.

But to me the bigger question is normative. Readers know I don’t usually have much truck with handwringing about neoliberal imposition. But that probably betrays my biases as a guy from an upper-middle-income country with lots of oil that’s seldom had to jump through hoops to qualify for a desperately needed loan. But c’mon, Honduras is averaging well over two semi-imposed, voluntary-in-name-only reforms per year!

Can this really be right?

At the very least it looks like a colossal waste of resources. Who can put a cost on the wasted bureaucratic energy spent on approving reforms nobody ever seriously intended to implement?

But at worse, it looks like a hijacking of basic governance processes. At that pace of imposition, what can “democracy” really mean for a country like Honduras? And what chance for Honduras to strike out creatively and find its own ad hoc solutions for its own unique problems when all its institutional energy has to be devoted to dreaming up fake paper reforms designed to do nothing beyond get that next tranche of loan released?

The opportunity cost to outsider driven reform is enormous. Because whatever it is you’re doing when you’re chasing that next loan tranche, what’s clear what you’re not doing: Problem-Driven Iterative Adaptation.