|
Review of Evaluations on UNICEF's Education Activities
1994-2000
Executive Summary
Background
The evaluations examined in this review were completed over the
last six years. The underlying education projects typically had been
underway 2-4 years at the time of the evaluation. Initial design of
these projects occurred in the 1-2 years preceding their start.
Hence, the evaluations reviewed here reflect thinking from 5-10
years ago about what interventions and strategies represent
effective education development. In the rapidly evolving world of
UNICEF, that is an eon in time. Over the last 5-10 years, UNICEF
thinking about the focus of its education work and its strategies
for accomplishing this work has developed enormously. Little of the
current thinking can be expected to show up in a review of
evaluations of those earlier projects. Nonetheless, findings from
the work undertaken during this period can inform future education
and evaluation work within UNICEF.
Purpose / Objective
The objectives of the review were to:
(1) identify the range
of strategies used by UNICEF Country Offices to extend access and to
strengthen the quality of education;
(2) suggest the extent to
which UNICEF-supported activities (projects and programs) were
successful in achieving their intended ends;
(3) identify
implications and lessons from these projects and programs in the
context of strategic planning and programming of UNICEF education
work until 2015;
(4) provide an overview of challenges and key
issues that remain, and offer specific recommendations for the
future; and,
(5) assess the contribution of project and program
evaluation in UNICEF education activities.
Methodology
Overall, 411 studies from 93 countries (or regional offices) were
made available for this study. To be included in this review, each
study had to meet two criteria. First, there had to be a clear
UNICEF involvement in the evaluated activity and/or in the
evaluation itself. Second, the study had to be evaluative in nature
and broadly comply with UNICEF's policies, procedures and
methodologies. Studies that reported baseline data, situation
analyses, and general descriptions of educational needs in a country
were eliminated from further consideration. A preliminary screening
of these studies identified 185 studies as meeting these
criteria.
Interviews were conducted with specialists of the Education
Section at UNICEF Headquarters. Findings were also considered in
light of relevant research and international experience reported in
the international development literature.
Key Findings
There is a wide range of views within UNICEF about how to best
extend access, improve quality, and address the education needs of
children. UNICEF's education activities, designed in the early and
mid-90s, did not reflect a strong consensus about what interventions
were most likely to promote wider education access or improve
quality. In fact, an important strength of UNICEF's education work
has been its ability and willingness to undertake such a variety of
interventions, based in large part on the judgment of those in the
field -- country staff, government officials, and local
collaborators.
It should be noted that, given the experience of the last six
years, a broader consensus about effective practice is now
developing, a topic discussed more fully in the main body of the
paper. Since the time frame in which the activities reviewed in this
study were designed and implemented, UNICEF has progressively
developed more coherent goals and strategies for its support and
advocacy work. This change has been consistent with the whole-child
and rights-based approach to development and in line with the
Convention on the Rights of the Child. UNICEF-supported projects and
programs in education still show a high degree of diversity in terms
of strategies and implementation modalities on the ground which, in
turn, reflect the different realities and choices across regions and
countries. Diversity also results from the fact that UNICEF often
attempts to innovate. However, it should be noted that, given the
experience of the last six years, a broader consensus about
effective practice is now developing.
Early Childhood Development:
Findings suggest that early
childhood education can make an important positive contribution in
the lives of children, but that success is not guaranteed if
projects are not properly designed or implemented. Success depends
on effective implementation of activities and the support of key
stakeholders, particularly parents, teachers, and government
officials. The evaluations provide some evidence that children who
participate in early childhood education activities demonstrate
higher levels of learning in primary grades 1-4, but the differences
in achievement tend to be small. Teacher and parent reactions to
these activities were mixed, supportive in some countries, not
supportive in others.
Excluded Children:
Country Officers supported a wide variety
of activities to reach excluded children reflecting the diverse
contexts. Some approaches worked well, some did not.
- Stipend programs generally were effective in getting more
children in school. One program that provided stipends to offset
lost income to the family due to a child attending school instead of
working was quite successful. Subsidies given directly to individual
children in another country did not seem to improve school
participation rates, but did appear to reduce absenteeism and lower
dropout of students, once they were in school.
- Activities aimed at reaching out-of-school youth were generally
regarded as successful. Sometimes, however, results were either
mixed or hard to determine. In some countries, these activities
encountered resistance from the community.
- Several of the projects were successful in raising community
awareness about the value of enrolling children in school, the
problems children face in gaining access to school, and the risks
associated with child labor. However, increasing community awareness
did not always translate into increased community action and, even
when communities did participate more actively, initial increases in
participation did not necessarily endure.
- Increased children's knowledge of risks (e.g., HIV/AIDS) does
not always lead to the intended changes in behavior.
- The success and eventual impact of UNICEF-supported activities
depends as much on the quality of the implementation as it does on
the initial project design. Even well-designed projects fail if
poorly implemented. However, it appeared that monitoring and
follow-through in ensuring effective implementation was sometimes
weak.
- Seemingly well-designed projects encountered unanticipated
difficulty when they misjudged local factors that worked as
disincentives. Similarly, some activities encountered unanticipated
negative cross-impacts, in which solving one problem created other
problems, some more severe than the initial problem being addressed.
Girls' Education:
Across the key areas of UNICEF education
work, its activities to promote girls' education reflect the most
consistent planning, the most uniform framework of action, and the
most systematic evaluation efforts. UNICEF's multi-country African
Girls' Education Initiative (AGEI) has provided a structure that has
helped Country Offices clarify goals and objectives, select
strategies, and monitor progress.
A recent desk review of UNICEF-supported girls' education
activities provides a useful review and analysis of UNICEF's work in
girls' education. Among the main findings were the following:
- Community participation is recognized in most interventions as
a key to ensuring relevance and acceptability of girls' education.
However, lasting impact on girls' participation is limited where
local initiatives are not accompanied by attention to national
policies in education and related sectors.
- The physical accessibility and safety of the school, its sense
of psychosocial security, and the quality and relevance of its
pedagogy are fundamental elements in determining whether and how any
child, and especially girls, will participate.
- Teachers' attitudes and behavior are central in promoting
girls' participation in education.
- Lowering the financial and personal cost of attending school is
a key factor in promoting education. Programs that reduce the direct
costs of school attendance (e.g. elimination of school fees) or
target subsidies to girls (scholarship programs) seem to be
effective.
While the evaluation of girls' education activities is arguably
the most systematic and cumulative area of UNICEF education work and
girls' education activities appear to have considerable impact,
evaluation findings are, nonetheless, mixed. The main findings of
the present review were that:
- While raising community awareness of the importance of girls'
education was an important component of many of the activities,
changing attitudes and behavior was not easy and not always
successful. The impact of some activities was limited by the failure
of the project designers to adequately anticipate and address the
complexity of incentive systems that operate. For example, in one
country, efforts to increase the proportion of female teachers
encountered unexpected difficulty when women, once trained, did not
want to be separated from their families or be assigned to rural
areas. Community-participation activities in support of girls?
education in another country encountered resistance when parents did
not want to pay for children of other families.
- Efforts to improve girls' education sometimes had negative
cross-impacts. For example, in one country where school fees were
abolished, girls' enrollment increased. However, school budgets
declined when Government did not compensate schools for those lost
fees. The loss of resources at these schools threatened the quality
of instruction.
- In some countries, design and implementation of activities
suffered from lack of coherence and poor communication among
partners. Several evaluations highlighted the need for stronger
coordination among UNICEF, government and NGOs.
- Effective interventions are not necessarily less expensive.
Some activities found to be effective had higher unit costs than
less effective alternatives. As long as the increase in outputs is
disproportionately greater than the increase in cost, such
activities should be encouraged.
Education Quality:
There was substantial diversity in the
strategies that were undertaken to improve the quality of education.
While a broader consensus is now forming, the range of interventions
pursued over the last six years provided a useful opportunity to
test a wide array of strategies.
- Teacher training had a positive impact on changing teachers'
pedagogical practices, raising student achievement, and improving
enrollment, though those impacts were not automatic or assured.
- Multi-grade teaching, in which one teacher teaches several
grade levels in the same classroom, was frequently effective in
raising student achievement.
- Student-centered teaching was a relatively popular intervention
but the impacts on teaching practice and on student learning were
mixed. It was frequently too complicated for teachers to implement
effectively.
Restoring Education in Emergency Situations:
UNICEF's role in
emergency situations has evolved from an emphasis on providing
supplies to displaced persons to an emphasis on preserving and
restoring basic structures of their formal education system and
helping countries make the transition from emergency conditions to
functional schools. Among the findings were that (1) UNICEF was
generally effective in delivering intended inputs (supplies,
materials), (2) UNICEF provided a longer-term continuity that was
much-needed during the emergency, but that (3) UNICEF was somewhat
weaker on contingency planning and disaster preparedness.
Country-wide, Multi-country Studies:
These evaluations were
among the most useful for purposes of this review because they
looked across wider sets of activities and emphasized synthesis.
These studies found that stakeholders and participants usually held
positive views of UNICEF activities, that the actual success of the
activities in achieving their intended objectives was quite mixed,
and that limited success in achieving those impacts often was due to
factors external to activities themselves.
- In several countries, UNICEF activities have clear goals but
lacked well-defined objectives or criteria for success. The
evaluations of these activities reflected the lack of clarity about
what was intended to be accomplished. For example, in one case, the
Mid-Term Review concluded that the success of programs could not be
determined because of lack of clear criteria for success, lack of
initial baseline data, inadequate indicators, and poor measurement.
In another country, ambitious project goals were not well aligned
with the limited resources available to the project. The goals were
not translated into realistic objectives that might have been more
easily addressed.
- The confusion between UNICEF and Government (or NGO) partners
about roles and responsibilities was frequently cited as a
limitation on the overall effectiveness of UNICEF work.
Policy Studies:
Studies highlighted the effectiveness of
school clusters and the need for UNICEF Country Offices to publicize
their accomplishments more broadly within the countries.
Lessons Learned
A. Conclusions on UNICEF-supported Education Activities
Widespread positive regard for UNICEF work:
One of the most
consistent findings across activities was that the individuals and
governments involved in delivering, receiving, and otherwise
supporting UNICEF field work believed that UNICEF projects were
doing good things for their country and for them. Participants,
counterparts, and other stakeholders express widespread positive
regard for projects. They like what UNICEF does, believe it is
important, and support the effort. That positive regard is not
always supported by evidence that the activities are meeting their
intended objectives, but the positive regard serves as one indicator
of organizational success.
Government-UNICEF linkages:
Governmental partner agencies and
project participants liked working with UNICEF and held
UNICEF-supported education projects in high regard. However, where
projects did encounter difficulties during implementation, it was
often attributed to problems in coordination within UNICEF or
between UNICEF and Government. This was not necessarily due to
personalities. More often, it was attributed to incompatibilities
between personnel systems, accounting regulations, procurement
procedures, and information flow.
Design features:
Some of the UNICEF-supported projects and
programs showed weaknesses in their design. In particular, the
components of some activities seemed scattered and of dubious
importance. These components may each have been included in response
to the pressure of some interest group during the design phase, but
distract from the overall coherence of the activity during
implementation. At the same time, outcomes are sometimes
over-promised. Seemingly effective activities appear to fail when
they cannot deliver on over-stated promises. Activity designers may
believe they need to promise over-stated outcomes to justify the
requested investment. Actions taken to 'sell' a project during the
activities' approval process may inadvertently raise the stakes
higher than is reasonable, given actual project activities.
Undocumented design changes:
During implementation, project
staff may introduce alterations in project design and intended
outcomes that they believe are necessary accommodations to get past
unanticipated hurdles or design oversights. These changes are often
the product of subtle negotiations among UNICEF and country partners
and result in trade-offs that are often undocumented. One risk is
that, in their attention to initial objectives, evaluators sometimes
fail to evaluate the activity that was really implemented.
Roles and relationships:
One of the threats to the success of
UNICEF-supported education projects was the confusion that developed
among project partners and key personnel about roles, relationships,
lines of authority, and locus of responsibility. Projects failed
when people in leadership positions did not do what they were
supposed to do or did not do things other people thought they had
agreed to do.
Differences in addressing access and quality:
In general,
activities to extend access have had more success than those aimed
at improving educational quality. Extending access often involves
changing structures (e.g., building schools in new locations,
lowering school fees, providing textbooks) aimed at changing
school-going behavior through the creation of incentives and the
reduction of barriers. Changing school quality often focuses more on
getting people to behave differently through training and
persuasion. Changing individuals? behavior through persuasion is
generally the harder proposition.
Sustainability of results:
Evaluations gave little attention
to the issue of sustainability of effects resulting from
UNICEF-supported projects and programs. Some conclusions do,
however, emerge from this review:
- Evaluations of community participation, teacher training,
continuous assessment, and restoration of education in emergencies
all observed that positive outcomes, once achieved, were difficult
to sustain. This lack of sustainability often seemed to stem from
inattention to incentives. Activities were generally designed to
promote the best interest of the child without sufficient attention
to the best interest of those who were expected to implement the
activities.
- Findings suggest that more thought should be given to what
design characteristics are most likely to promote sustainability.
Some evaluations suggest that actions undertaken to facilitate
implementation in the short-term (anchoring a project in the
community) may, in the long run, work against sustainability.
Sustainability comes from building project activities into a more
durable infrastructure, such as the government bureaucracy.
Going to scale:
The goal of most development projects is that
once they demonstrate their success on a small scale, similar
activities will be developed at a larger-scale, perhaps even at the
national level. This move, from pilot to full-scale implementation,
is described as ?going-to-scale?. Although the evaluations reviewed
in this study do not give much attention to this dimension, there is
evidence that the dynamics of going-to-scale are more complicated
than is widely realized. Pilot projects often operate under highly
advantaged conditions that can never be replicated on a wider scale.
When a project goes to wider implementation in settings that are
less advantaged, participants tend to be more average in their
professional abilities, incentives tend to be less attractive,
materials tend to be less available, and supervision tends to get
stretched.
B. Conclusions on the Conduct of Evaluation
Emphasis on evaluation for mid-course correction at the project
and sectoral program level: UNICEF evaluation practice is mainly
geared toward mid-course corrections at the individual activity or
sectoral program levels. The evaluations that were reviewed seldom
made any explicit reference to how they fit into the Integrated
Monitoring and Implementation Plan (IMEP) or, in a broader sense,
into the overall Country Program. Evaluations seldom indicated how
they were intended to connect to the UNICEF's Mid-Term Review or the
Country Program Evaluation. It appears that evaluations, to a large
extent, were designed and implemented in an ad-hoc manner and not as
a tool for strategic Country Program management.
Confusion in selecting evaluation criteria:
Weaknesses
related to the design of projects and programs (see above) lead to a
lack of clarity concerning evaluation criteria. A number of the
evaluations expressed confusion over what criteria were to be used
in judging the success of an activity. In other cases, program
designers failed to specify the criteria of project success, leaving
it to subsequent activity managers or evaluators who may not have
had as clear an understanding about what the activity was intended
to accomplish.
The dominance of affect as an indicator of project success:
Of
the evaluations reviewed, many were unable to document changes in
behavior or performance. When that happened, there was a pervasive
tendency to diminish the importance of those behavioral changes in
favor of attitudinal data, typically showing that participants held
favorable attitudes toward the activity. When evaluations did report
attitude, the definition and measurement of attitude tended to be
rather casual. Attitude measures in a number of the studies were
little more than measures of generalized affect of questionable
validity or meaning.
Limited data on outcomes and impact:
Attention to overall
outcomes in terms of capacity building and impact of
UNICEF-supported education activities on the lives of children seem
to be underrepresented in the evaluations reviewed in this study.
Relatively few evaluations actually assess the extent to which
education activities achieve their broader goals or objectives. A
number of evaluations report disappointing or no discernable
outcomes or impacts.
The attribution of impact to UNICEF-supported activities is
difficult and often impossible. Improved access and quality of
education are often the result of a variety of factors that are not
directly related to UNICEF support, including implementation
decisions of local partners and the macro-economic and social
environment in which the project operates. Factors such as economic
growth, poverty reduction, monetary and fiscal policies, and
government spending can have an enormous impact on access to and
quality of education and provide a more or less favorable
environment to individual project activities.
Issues not fully addressed in evaluations: With notable
exceptions, little attention was given to issues of cost,
efficiency, sustainability, or going-to-scale.
- Cost: Inadequate funding was cited in several cases as an
important factor contributing to implementation problems. Beyond
that level of observation, cost was seldom reported or discussed.
Few evaluations reported aggregate cost, unit costs, or opportunity
costs associated with the interventions being evaluated. Few of the
studies included a cost-benefit analysis. In only 1-2 studies, was
the distinction made or any attention given to the interplay between
investment costs and capital costs. Research and international
experience suggest that failure to attend to recurrent costs is a
frequent reason that projects are not sustained. External funds are
invested in training systems, buildings, and initial production of
materials without adequate regard for the recurrent costs associated
with keeping the training going, the buildings clean, and the
printing presses running after the external monies end.
- Efficiency: The efficiency of UNICEF-sponsored education
projects was virtually never discussed in the evaluations that were
reviewed. This is probably due to the absence of cost data (a
necessary component in the consideration of efficiency).
- Sustainability: While several of the activities had
sustainability as a goal, it was seldom addressed in any of the
evaluations. This is largely because (a) most of the evaluations
were conducted early in the activity, when the evaluation questions
were concerned more with level of implementation and (b) a
persisting confusion about what aspects of an intervention should be
sustained (e.g., specific activities, organizational capacities, or
positive impacts). One of the threats to sustainability was normal
staff turnover within the Country Office and government ministries
working with the activity. Finally, due to the frequent failure to
clearly document (or even consider) recurrent costs, many projects
lacked the necessary financial support needed to continue.
- Going-to-scale: Given the difficulty many seemingly successful
pilot projects have in replicating their success on a wider scale,
attention to ensuring the preconditions for going-to-scale is widely
considered to be an important element in the evaluation of projects
that are intended for larger delivery. It was surprising, then, that
going-to-scale was not widely addressed in the evaluations reviewed
for this study.
Linking evaluation findings to future agendas:
Many of the
key development challenges of the next decade will be different from
the past decade. Across the UNICEF activities reviewed in this
study, there was little attention to emerging issues such as
HIV/AIDS, the impact of new technologies, globalization, and the
complex issues arising from decentralization. Widespread attention
to these issues has arisen more recently.
Recommendations
Coping with new challenges:
Over the last decade, there has
been important elaboration of needs and issues, new focus, and
shifts in the strategic thinking about UNICEF's role in education
development. It is widely regarded as one of the most effective
international agencies working in the area of education development.
Still, as world conditions change, the struggle to remain relevant
and effective continues.
Consolidation of achievements:
UNICEF will need to balance
its attention to new issues with its commitment to sustaining and
consolidating the gains it has already made. Many of the education
issues of the last decade will continue to be prominent in the next
ten years. A critical issue over the next decade will be countering
the risk of eroding the gains that have already been made in
education access and quality, even as new issues are added to the
agenda. It will be important to ensure that existing programs and
systems are working before shifting investments to new agenda. This
effort is particularly threatened by issue fatigue and the backlash
effect. Issue fatigue occurs when the novelty of a problem wears off
and attention shifts to new and more intriguing issues. The backlash
effect occurs when some constituencies feel that certain issues are
getting too much attention and that the concentration of concern for
those issues is allowing inequities to emerge in other areas.
Design and implementation of education activities:
UNICEF-supported education projects and programs are usually
well-considered by governments, NGOs and other partner
organizations. There is sometimes room for improvement, however, in
how projects and programs are designed and implemented. The role of
each partner involved in these activities often needs to be
clarified. Conditions for sustainability and possible scaling-up
need to be more critically assessed. Greater clarity is often
required about expected outcomes, particularly with respect to
national capacity building and impacts on the lives of children.
Individual project and program activities should be clearly related
to situation analyses and overall Country Programs in a human rights
perspective and in terms of results-based management.
Evaluation challenges for the future:
There is a need to
strengthen the role of evaluation in the Country Program cycle and
in overall strategic planning and programming. The evaluation effort
could be strengthened by more attention to outcomes, behavioral
changes, cost, sustainability, and the conditions necessary for
going-to-scale. Additionally, evaluation could benefit from a shift
away from a focus on individual activities toward more of a focus at
programmatic and policy levels.
| 
