‘Technical rigor must
not take precedence over other knds of valuable lessons’
Howard White
8
Abhijit Banerjee calls for an end to lazy thinking in the design
of aid programs. What we need instead, he says, are randomized
impact evaluationsof the sort promoted by his organization, MIT’s
Abdul Latif Jameel Poverty Action Lab.
I agree that aid
agencies should do more randomized impact
evaluations. In fact, they should be implemented
whenever possible. But this statement needs to be
put into perspective, as the portion of
development aid that can be subject to randomized
impact evaluation is severely limited. Testing
must not be promoted exclusively and at the
expense of other valuable approaches. And while
randomized impact evaluations can yield useful
information, the search for technical rigor must
not take precedence over practical
lesson-learning.
When, then, can
randomized impact evaluations be used? Banerjee
compares aid programs to drugs; the analogy is a
good one. Randomized approaches can be used to
evaluate discrete, homogenous interventions, much
like a pill in a drug trial. But most of the
projects of large official agencies—which
constitute the bulk of aid—do not resemble the
conditions of medical testing.
Over the last 20
years a large share of aid has been designated
for broad reforms: labor market reform,
reductions of producer and consumer subsidies,
privatization, interest rate increases, and the
exchange rate reform noted by Banerjee. In the
last 15 years increasing amounts of aid have
funded the development of democratic
institutions, anti-corruption bodies, policy
think tanks, and the collation of voter roles.
Donor agencies also supply experts to support
these kinds of reforms and improve the quality of
developing-country institutions. For example,
activities to improve learning outcomes—such as
the use of flip charts or textbooks—may be part
of a larger project that includes computerizing
the country’s education database, providing
training to central ministry staff, and
constructing a new teacher training college. Such
projects cannot be subject to randomized
evaluation.
Unlike the smaller,
NGO-supported programs cited by Banerjee,
large-scale infrastructure projects supported by
official agencies—such as rehabilitating a port
or building a major bridge—are not amenable to
randomization. Donors have supported systematic
social-service provision across countries through
the construction of health and education
facilities and training for their staff based on
mapping exercises and manpower planning: children
need schools and teachers before they can go to
school and learn. These are not sufficient
conditions, but they are necessary. Like broad
policy reforms and large-scale infrastructure
projects, they cannot be subject to
randomization.
And finally, donor
agencies increasingly provide budget
support—money the government can use as it
wishes. In this case it is the government’s
programs, not the donors', which need to be
evaluated. Donors must ask how their budget
support affects the level and composition of
expenditure. Again, randomized approaches are of
no help.
But other kinds of evaluations
are both possible and useful, and they are being
put into practice. The World Bank’s Independent
Evaluation Group carries out project assessments
for one quarter of all its completed projects. In
recent years, the group has reported on the
issues of impact and effectiveness of trade and
financial sector reform, debt relief, and support
to poverty reduction strategies. It has also
demonstrated the importance of investments in
social infrastructure for improving social
outcomes—for example, in reports on basic
education in Ghana and health services in
Bangladesh.
Are there other ways to evaluate
projects? There is a well-established method of
identifying which projects should get funded
called cost-benefit analysis. This analysis lists
all the costs and benefits from a project and
assigns a value to them. This includes putting a
price on things that the market may not value, or
may not value appropriately, such as adverse
environmental impacts. The method was developed
over 30 years ago and was widely adopted by the
World Bank and other agencies. It subsequently
fell out of favor for the poor reason that it was
assumed not applicable to increasingly important
social investments (it is), and for the better
reason that it couldn’t asses whether the
institutions responsible for service delivery
actually worked. When they didn’t, few if any
benefits were realized.
Ex post evaluations, or
process evaluations, therefore started to focus much more on process
issues. Process evaluations look at how well project management
is working and if implementation has been satisfactory. They can
draw conclusions about sustainability based on the strength of
the institution delivering the services and the adequacy of financing
once aid to the project stops.
Donor agencies
conduct national evaluations which assess aid
partnership over a ten-year period. Such studies
are more than the sum of their parts, since
donors seek influence through demonstration
effects, informal interactions with government,
and more formal discussions about country and aid
strategies. Stakeholder analysis is needed to
conduct such studies, not randomization. Policy
reforms may be more formally evaluated using
modelling approaches. Process-oriented approaches
have general applicability to programs, policies,
and projects, though they are not designed to
evaluate impact.
Cost-benefit analysis was
introduced to do precisely what Banerjee calls
for—deciding what should be financed—but not
by examining all the possible uses of funds, as
he suggests. Calculating the benefits requires an
assumption about impact, and it is obviously best
if these assumptions are based on evidence from
impact evaluations. But such impact evaluations
can only adopt a randomized approach in a small
minority of cases.
Cost-benefit analysis
may be applied more widely, though it must
incorporate a more comprehensive view of
institutional conditions. For instance, if we
build a road, we need to know who will maintain
it, whether they have the necessary skills and
equipment, and where the money for maintenance
will come from. In the case of education, we need
to know what incentives teachers have to put
improved methods into practice.
At the
very least impact evaluations need to report
cost-effectiveness, but better still they should
offer a proper cost-benefit analysis. They must
attach an economic value to things not priced by
the market. To understand why aid has worked or
not, the analysis should examine not only a
specific impact but all levels of project
implementation. An Independent Evaluation Group
analysis of support to extension services in
Kenya, for example, found little impact. The
failure was readily explained by data that showed
both a weak link between new research and
extension advice and little change in the
practice of extension workers. Similarly, a study
of a nutrition program in Bangladesh found little
impact. In this case the study identified the
mis-targeting of potential beneficiaries and the
failure of mothers to put into practice the
nutritional knowledge they acquired through
counselling. A randomized approach might have
only shown that the project didn’t work, and
would not have helped explain the causal
factors.
In the end Banerjee gives the impression
that aid is not being evaluated. This claim is simply incorrect.
What he means is that there are too few randomized impact evaluations.
There are good reasons for this, though there is scope for greater
use of the approach. We can readily endorse Banerjee’s ends,
but endorsing his means would put an end to many effective things
aid donors do. Instead, the proper use of cost-benefit analysis
would do more to help donors make good choices. <
The views expressed here
are the authors' and do not necessarily reflect those of the World
Bank or the Independent Evaluation Group.
Howard White
is a senior evaluation officer of the Independent Evaluation Group
at the World Bank.
Click here to return to the
New
Democracy Forum “Making Aid Work”
Originally published in the July/August
2006 issue of Boston Review
|