The Tsunami Evaluation Coalition:

1
The Tsunami Evaluation Coalition

• What Worked and What Did Not?
• European Evaluation Society 2006

2
Background of Tsunami Evaluation Coalition

• The TEC is a new sector wide learning and
  accountability initiative constituted in February
  2005
• It is made up of about 40 UN agencies, donors,
  NGOs, a non-for-profit and the Red Cross/Crescent
  Movement.
• Participating agencies have worked within a
  framework that encourages sector-wide information
  sharing, lesson learning, accountability and
  transparency.
• Focus on cross-cutting themes (coordination,
  needs assessment, local capacities, donor
  response, LRRD) and sector-wide performance
  rather than on individual agency performance

3
TEC Timeline

• February 2005 Geneva Meeting
• April 2005 First TEC teleconference
• June 2005 ALNAP Meeting in the Hague
• July August - planning phase for all
  evaluations
• September November field visits
• October 05 Copenhagen Meeting Comm/Diss
  strategy
• November- May 2006 Report Writing
• December 8 ALNAP meeting/TEC meeting Brussels
  Presentation of early findings and early findings
  report
• December 25 publication of early findings
  report
• February 2006 Teamleader validation meeting
  London
• February June 06 Production of the Synthesis
  Report
• July 06 Launch of Synthesis Report during
  ECOSOC
• July 06 April 07 TEC Follow up

4
Getting started (1)

• This was a voluntary initiative started by a few
  actors who felt the time was right for a major
  inter-agency initiative
• The first meeting in 02/05 did not immediately
  provide clarity about roles and responsibilities,
  nor the actual nature of the various studies
• Many actors stayed on the fence .
• Much time was initially spent on gaining mutual
  confidence and building relationships
• Key initial actors busy with other things and TEC
  workload was significant for all key actors
• There needed to be dedicated time and resources
  at the beginning of the process jump started by
  ALNAP - the f/t researcher played pivotal role to
  keep the TEC going during the early days

5
Getting started ..(2)

• Key tipping points were the appointment of the
  researcher, the appointment of the coordinator
  and the ALNAP Biannual Meeting in the Hague in
  June 2005
• ALNAP meeting in particular brought the necessary
  buy-in and funding
• Funding, however, came in slow and had adverse
  impact on timeliness of the TEC
• Some agencies had to wait for full funding before
  the evaluation process took off delayed
  start-up of TEC missions as they were to be
  undertaken simultaneously
• TOR preparation not coordinated - duplication
• Fishing in the same pond

6
Fundraising

• Getting commitments from some major donors
  brought in others and gave wide buy-in
• Down-side multiple donors with short time-frames
  lead to short contracts for consultants,
  shortened field visits, increased admin costs
• Raising funds for the core of the TEC and between
  studies should have been better coordinated
• Fundraising for all five studies and the TEC
  Secretariat was extremely time-consuming and
  cumbersome this should have been part of the
  appeal or a special trust fund established
• Yet, excellent results BUT can this be replicated?

7
Implementation Modalities

• Set-up with a core management group and a broader
  working group worked well
• Strong commitment by CMG and sub-groups with
  very harmonious way of working together
• 3/5 studies had similar set-ups
• Good mix between face-to-face meetings and
  teleconferences
• Good use of technology telecon, shared
  documents, mapping, the resource CD
• Backing of ALNAP, a network with a natural fit to
  the TEC and an interest in joint evaluations
  was critical
• Complex arrangement

8
Theme Coordination led by OCHA
Theme International Communitys Funding Response
led by Danida
ALNAP Secretariat Hosts the TEC and manages the
writing of the Synthesis Report. TEC staff
include Evaluation Advisor Coordinator (EAC),
Researcher Deputy Coordinator (RDC), and TEC
Administrator
Theme Needs Assessment Led by WHO, SDC FAO
Theme Impact Assessment led by IFRC with the
Global Consortium
Theme Impact on Local National Capacities Led
by UNDP by DMI
Theme LRRD Led by Sida
Key Messages Report written by the EAC
Longer term Studies (from 06)
Individual Agency Evaluations (TEC Members)
Synthesis Report Written by the Synthesis
Primary Author with contributions from the EAC
and the RDC.
TEC Online Forum (includes the Evaluation Map)
9
Working through the mandate

• Mandate was assumed by the TEC but had no broader
  clientele, including those not working in
  evaluation units of the respective TEC members
• No real involvement of regional and local actors
• Five cross-cutting themes in principle a good
  idea but resulted in overlap, uncertainties
  between the teams and a confused and overloaded
  recipient community
• Did not consider alternative and possibly more
  cost effective approaches, e.g. one team per
  country
• Missed out on impact although an attempt was
  made to cover this through an IFRC-planned
  initiative that took almost a year to materialize

10
Some TEC shortcomings

• Not all teams worked well together
• Some critical expertise was missing
• Not enough time spent in the field
• Weak on hard data
• Little information on Impact
• Lack of local ownership/buy-in
• Reports of varying quality much work needed to
  bring some of them to acceptable levels
• Country reports in some cases not very strong
  underestimated time needed to do them well
• Many cooks teamleaders not fully on board
• Did not reduce evaluation overload

11
Some TEC Achievements

• First major system-wide humanitarian evaluation
  since Rwanda
• TEC approach can work and lessons from setting up
  the TEC will make the next time easier
• Timing of TEC products was well planned and
  critical (initial findings report for 12/25 and
  the synthesis report for ECOSOC)
• TEC is beginning to influence humanitarian reform
  debate
• Clinton Initiative is moving on critical TEC
  issues in relation on NGOs
• Much more follow up ahead but will need dedicated
  attention and a sustained effort at various
  levels

12
What should we do differently next time?

• Include system-wide mechanism as part of the
  appeal
• Get early in-country stakeholder buy-in
• Establish a local support/reference group(s)
• Organize regular in-country discussion/follow-up
  meetings (through a focal point organization)
• Promote the early establishment of performance
  indicators and ME systems
• Develop an evaluation framework with agreed-to
  performance benchmarks
• Reduce complexities (funding, multi-team etc)
• Identify good practice, not just what did not work