It has been a bit of a rough week for people who prefer their grand policy pronouncements backed with a teeny bit of evidence. Like us. Two big papers, in important journals, have concluded that there is very little evidence that agriculturally improving dietary diversity feeds into better nutrition and health. In the British Medical Journal, ((Masset, E., Haddad, L., Cornelius, A., & Isaza-Castro, J. (2012). Effectiveness of agricultural interventions that aim to improve nutritional status of children: systematic review BMJ, 344 (jan17 1) DOI: 10.1136/bmj.d8222)) a systematic review of agricultural interventions that aim to improve nutritional status of children concluded:
The data available show a poor effect of these interventions on nutritional status, but methodological weaknesses of the studies cast serious doubts on the validity of these results.
That’s based on a review of 23 published studies from a range of countries. In PLoS ONE, a study of the contribution of wild edible plants (WEP) to diets in DR Congo ((Termote, C., Bwama Meyi, M., Dhed’a Djailo, B., Huybregts, L., Lachat, C., Kolsteren, P., & Van Damme, P. (2012). A Biodiverse Rich Environment Does Not Contribute to a Better Diet: A Case Study from DR Congo PLoS ONE, 7 (1) DOI: 10.1371/journal.pone.0030533)) similarly concluded that:
[I]n a high biodiverse region with precarious food security, WEP are insufficiently consumed to increase nutrition security or dietary adequacy.
Both papers are making an important point. The second is perhaps less generally applicable. The authors were looking only at wild edible plants, not agricultural biodiversity as a whole, and found that they were “rarely consumed and do not contribute substantially to diets” in either urban or rural areas of DR Congo. It isn’t as if the people of DRC can afford to ignore WEP; their diets lack macro- and micronutrients and are not well balanced. And it isn’t as if there are no WEP for them to eat. 135 wild foods besides condiments, tubers, tea substitutes, etc. have been inventoried in the study area, but while the inhabitants described 77 WEP, only 11 species figured in their diets, and that during in the period of highest availability. More, clearly, could be done, both to capture the knowledge people have about WEP, and maybe to start attempting to domesticate some of the more important species.
The BMJ review paper looked agricultural interventions that explicitly sought to improve the nutritional status of children. These included bio-fortification, home gardens, aquaculture and small fisheries, poultry, dairy, and other livestock. These represent “own production” pathways, as opposed to the “market” pathways that seek both to improve incomes for farmers and make more nutritious foods more available. Criteria for inclusion were very strict. Studies had to include a control group, no before-and-after designs. Cross-sectional and longitudinal studies were included as long as the possibility of selection bias was acknowledged and at least partly controlled for. Twenty-three studies made the final cut, 15 of them on home gardens. They’re not an impressive bunch.
Overall, the methodological quality of these studies was not high by the standards set by the review. … None of the primary studies reviewed was based on a randomised design. Studies were cross sectional or longitudinal project-control comparisons in which the controls, either households or villages, were selected on the basis of similar characteristics that were either very few or not made explicit. Power calculations were rarely done or presented, and samples were often small in terms of both individual participants and clusters. No study did a rigorous subgroup analysis of effect differentiating, for example, households of different wealth, sex of head of household, or location of residence.
There were, of course, effects associated with the interventions. Incomes, for example.
Five studies reported a large positive effect of the interventions on total household income. However, only in one case was the difference between project and control groups statistically tested.
Diets changed too, but sometimes a focus on one component, for example fish, resulted in a decrease in another, in this case pulses. Changes in micronutrient intake were similarly inconclusive, with the possible exception of vitamin A intake, where the review authors’ meta-analysis
“[P]rovides some support to the hypothesis that agricultural home gardens interventions improve vitamin A intake among children under the age of 5”.
Thirteen of the 23 studies measured the children, but only 8 used these data to calculate the prevalence of under-nutrition. Three studies found a positive effect on underweight, two on wasting, and only one on stunting. This could be because stunting reflects long-term under-nutrition. The interventions might not have continued long enough to see effects on stunting.
Overall, these results provide little support for the hypothesis that agricultural interventions help to reduce under-nutrition.
That does indeed sound bad.
However, they should not be interpreted as evidence of the absence of an effect. Lack of significance can be the result of absence of effect or of absence of statistical power, and many of the studies reviewed included small samples of children.
To see if bad experimental design, particularly small sample sizes, might be to blame the review authors calculated the statistical power of the eight studies that measured children’s nutritional status. They conclude that they just weren’t that powerful, with only a 50% chance of detecting even a large (30% or more) reduction in malnutrition.
And that’s really the nub of the matter. One is tempted to bleat fatuously that “absence of evidence is not evidence of absence”. But that is exactly what it is. So why am I upset by this absence of evidence? Because there is actually quite good evidence that dietary diversity is indeed linked to better health, although most of it is from rich countries and is concerned with diet, not agriculture. Heck, I’ve cited many of them myself ((Frison EA, Smith IF, Johns T, Cherfas J, & Eyzaguirre PB (2006). Agricultural biodiversity, nutrition, and health: making a difference to hunger and nutrition in the developing world. Food and nutrition bulletin, 27 (2), 167-79 PMID: 16786983)), ((Frison, E., Cherfas, J., & Hodgkin, T. (2011). Agricultural Biodiversity Is Essential for a Sustainable Improvement in Food and Nutrition Security Sustainability, 3 (1), 238-253 DOI: 10.3390/su3010238)) and blithely extrapolated from those rich-world studies to argue for agricultural interventions, and specifically dietary diversity of the type found in home gardens, as a way of improving nutrition in poor countries.
Tackling malnutrition, the Copenhagen Consensus agreed, is one of the most cost effective investments in development it is possible to make. The report on malnutrition, however, considers only supplements (direct and in fortified foods) and biofortification. Dietary diversity simply does not figure. I asked why. The lead author replied:
Clearly diversifying diets is an important part of the long run solution to improving nutritional status. However in the paper I was asked to focus on benefit-cost ratios which could be calculated, and it is typically harder to do this for such long run solutions.
Not so much absence of evidence, then, as no effort to find anyway.
What then does all this mean. To me, the answer is very clear. The people making grand policy pronouncements about agriculture and nutrition need to seriously up their game. Both of these papers, and especially the big BMJ review, offer an opportunity to get to grips with what it takes to produce the evidence that is currently lacking. And decent evidence will make it far more likely that agrobiodiversity is allowed to play a much fuller part in sustainable improvements in nutrition and health.