Re-thinking the Impact of Humanitarian Aid

1
Re-thinking the Impact of Humanitarian Aid
24th Biannual Meeting Berlin, 2nd December 2008
2
ALNAP 24th Biannual Aims

• To help clarify key issues around use of
  humanitarian impact assessments
• To move towards a shared understanding of the
  limits and possibilities of humanitarian impact
  assessment
• To use this understanding to outline a practical
  vision for future work in this area

3
Overview

• Humanitarianism transformed and the emergence
  of impact assessment
• Challenges for assessing impact
• Conclusions

4
Humanitarianism Transformed

• Three broad trends since the 1980s contribute to
  the current interest in and debate around
  humanitarian impact assessment
• Humanitarian aid expanded and politicised
• Humanitarian aid institutionalised and
  professionalised
• Changing nature of vulnerability and human
  suffering

5
Increase in initiatives, across and within
agencies

• Cross agency efforts include SMART HNTS TRIAMS
  Fritz Humanitarian Impact Project the CDA
  Listening Project ALNAP HPP SPHERE
• Quality Compass ECB Good Enough Guide to
  impact measurement DECs new Accountability
  Framework etc
• WFP, ECHO and UNICEF all include impact in their
  evaluation guidelines ActionAids ALPS Save the
  Children UKs GIM

6
But despite considerable progress, problems
remain. The Biannual background paper has
identified six key broad challenges

1. Defining impact assessment
2. Diverse stakeholders and interests
3. Indicators, baselines and data
4. Methodologies
5. Collective interpretation and analysis
6. Capacities and incentives

7
1. Defining impact and impact assessment
8
Ideal picture of impact the MDG Goal of
universal primary education by 2015
9
Reality of impact is rather different

• Universal primary education would be achieved
  at present rates of progress in 2079 in
  sub-Saharan Africa and in 2036 in the Middle East
  and North Africa
• Social Watch 2007

10
Real world impact is complex
Activity
Output
Outcome
Impact
11
and hard to discern, even a long time after the
fact
12
Attribution or Contribution?
Private Sector
Other NGOs
A.N. NGO
Community and Family
Local partners
Religious organisations
Developing Country Govmts
Civil Society
13
A widely recognised definition of impact
assessment

• Impact assessment is the systematic analysis
  of the lasting or significant changes positive
  or negative, intended or not in peoples lives
  brought about by a given action or series of
  actions.
• Novib/Oxfam research project, reported in C.
  Roche, Impact Assessment for Development Agencies

14
Humanitarian action has its own challenges

• Lack of clarity on IA definition and purpose
• Rapidly changing humanitarian contexts
• No consensus on objectives of humanitarian aid
• Intended impacts of interventions often unclear
  or overambitious

15
Contextual differences between normal development
aid and humanitarian aid situations

• Development
• Considerable Lead Time
• Deliberate pro-active
• Will take time, be thorough, extensive with
  comprehensive data collection
• Location chosen
• Duration planned
• Beneficiary population identifiable and static
• IA goals may be made compatible with
  socio-economic ones

• Humanitarian
• Sudden onset
• Reactive
• May need to be partial in coverage
• Unpredictable location
• Uncertain duration
• Beneficiary population heterogeneous and dynamic
• Priority given to life saving activities
  sometimes difficult to reconcile with IA goals

16
2. Diverse stakeholders, interests and objectives
17
(No Transcript)
18
The challenge of stakeholders (illustrative)
UN agencies
National and local partners
Donors
Media
Affected Population
International NGOs
Private Sector
Red Cross / Red Crescent
Military
Political authorities
19
Different stakeholders have different perceptions
of and interests in impact

• Enabling different stakeholders to express
  divergent views of impact is crucial to
  successful impact assessment
• IA findings more likely to be used they meet the
  interests of the end users

20
Different needs may not be reconcilable and
achievable in a single impact assessment
Accountability or Learning?

• The purpose of most impact assessments is to
  demonstrate past impact and to improve future
  practice, and there may be tension between the
  two.
• Often, too much is expected If we continue to
  expect evaluation to cover most of the
  accountability needs of the sector, we will be
  disappointed (Sandison, P. 2006).

21
3. Indicators, baselines and data
22
Indicators, Baselines and Data for humanitarian IA

• Identifying impact indicators involves value
  judgements about what kinds of changes are
  significant for whom (Roche, C. 2000)

23

• The familiar adage you can lead a horse to
  water, but you cant make it drink illuminates
  the challenge of committing to outcomes. The
  desired outcome is that the horse drinks the
  water. Longer-term outcomes are that the horse
  stays healthy and works effectively. But because
  program staff know they cant make a horse drink
  water, they focus on the things they can control
  leading the horse to water, making sure the tank
  is full, monitoring the quality of the water, and
  keeping the horse within drinking distance of the
  water. In short, they focus on the processes of
  water delivery rather than the outcome of water
  drunk. Patton, M. 1997157-8

24

• Reports were so consistent in their criticism
  of agency monitoring and evaluation practices
  that a standard sentence could almost be inserted
  into all reports along the lines of It was not
  possible to assess the impact of this
  intervention because of the lack of adequate
  indicators, clear objectives, baseline data and
  monitoring. (ALNAP, 2003)

25
Issues include

• Weak or non-existent baselines
• Data is often unavailable or unreliable
• Data collected is mainly quantitative
• Monitoring systems focus on process and outputs

26
4. Methodologies
27
Wealth of tools, techniques and approaches are
available

• Documentary analysis
• Interviews
• Questionnaires (including recipient perceptions
  surveys)
• Monitoring
• Ex-post evaluation
• Case studies
• Participatory Rapid Appraisal (PRA)
• Experimental / quasi-experimental

28
Qualitative versus quantitative

• Any research design is shaped by both
  opportunities and constraints
• Quantitative methods able to tackle what and
  where questions
• Qualitative methods able to answer why and
  how questions, and are good at capturing process

29
(No Transcript)
30
Mixed methods approaches can take into account,
rather than dismiss, the complexity of assessing
humanitarian impact
31
5. Collective interpretation and analysis
32
Improved interpretation and analysis of data
through engagement with affected populations and
other stakeholders

• Humanitarian impact should not only be about
  providing more and better information, but also
  about making sure that findings are used in ways
  that improve the lives of affected populations
• Wider stakeholders should also be engaged in this
  process
• Learning partnerships for impact assessment

33
To date, participation by and accountability to
affected populations has not been a key feature
of impact assessments

• Attempts to improve this include
• ECB Good Enough Guide
• The Quality Compass
• Feinstein International Center Participatory
  Impact Assessments (PIA)

34
6. Capacities and incentives
35
Capacities and Incentives for improved
humanitarian impact assessment

• Lack of individual and organisational capacity to
  do good impact assessments
• TORs are often unclear
• objectives are not defined clearly within the
  context of the intervention
• stakeholder analysis is limited
• timing relates to institutional priorities rather
  than humanitarian need
• skills relating to impact assessment
  methodologies are lacking
• Contributing factors high staff turnover lack
  of a learning culture inadequate investment and
  resources

36
There is a considerable lack of incentives

• Institutional incentives can override
  humanitarian ones too few incentives to conduct
  good impact assessments results-based approaches
  can create perverse incentives
• A number of cultural barriers and biases that
  hinder good quality humanitarian impact
  assessment

37
Recap six challenges

1. Defining impact assessment
2. Diverse stakeholders and interests
3. Indicators, baselines and data
4. Methodologies
5. Collective interpretation and analysis
6. Capacities and incentives

38
Conclusions

• Taken as a whole, the humanitarian system has
  been poor at measuring or analysing impact, and
  the introduction of results-based management
  systems in headquarters has yet to feed through
  into improved analysis of impact in the field it
  is arguable that there has been significant
  under-investment in evaluation and impact
  analysis (Hoffman C.A. et al, 2004)
• Our review gives little indication that there has
  been much movement from the position above,
  articulated in 2004
• Can we move forward? How?