IFCs Evaluation Framework

1
Post-evaluation in IFCNovember, 2003
William E. Stevenson Roland Michelitsch Operations
Evaluation Group (OEG)
2
Overview

• OEG in the World Bank Group
• Project evaluation in IFC
• Beyond project evaluation

3
The World Bank Group
Board
President World Bank Group
International Finance Corporation
Multilateral Inv. Guarantee Agency
World Bank
4
International Finance Corporation
Board
President World Bank Group
International Finance Corporation
Multilateral Inv. Guarantee Agency
World Bank
5
IFCs purpose

• Article 1
• further economic development
• by encouraging the growth of
• productive private enterprise

6
IFCs instruments

• IFC loans equity for viable projects
• Capital mobilized from other sources
• Advisory services

7
IFCs mission

• Promote sustainable
• private sector investment
• in developing countries,
• helping to reduce poverty
• and improve peoples lives

8
IFCs strategy sustainable development

• Support enterprises that
• can prosper in the marketplace
• dont rely on distortions
• are environmentally benign or beneficial
• tangibly improve peoples lives

9
WBG independent evaluation
Board
Director-General, Operations Evaluation
President World Bank Group
International Finance Corporation
Multilateral Inv. Guarantee Agency
World Bank
Operations Evaluation Group
Operations Evaluation Unit
Operations Evaluation Department
10
OEG independent evaluation in IFC
Board
Director-General, Operations Evaluation
President World Bank Group
International Finance Corporation
Multilateral Inv. Guarantee Agency
World Bank
Operations Evaluation Group
Operations Evaluation Unit
Operations Evaluation Department
11
OEGs mission

• Contributing to sustainable
• private sector investment
• through excellence in evaluation

12
Evaluation principles

• Learning accountability
• Relevance
• Independence
• Coherence
• Incentives

13
Overview

• OEG in the World Bank Group
• Project evaluation in IFC

14
Project-level evaluation in IFC

• Self-evaluation
• Expanded Project Spvn. Reports (XPSR)
• Other PSRs, portfolio review meetings
• Independent evaluation
• XPSRs
• System
• Sampling
• Framework
• Validation (ratings lessons)
• LRN system (lessons dissemination)

15
Project evaluation system

• Purpose learning accountability
• Design relevance, efficiency, coherence
• Evaluation framework derived from IFCs purpose
• Random, representative sample
• Investment departments write XPSRs
• OEG reviews and validates findings
• IFC management discusses 30 per year
• Lessons easily accessible through LRN
• Incentives for good execution
• Results feed into investment dept. scorecards

16
Evaluation framework

• Derived from IFCs purpose mission
• In place since 1996
• Aligned with front-end sustainability measures
• Best practice among private sector MDBs

17
MDBs portfolios evaluation reports
18
Evaluation framework Performance dimensions

• Development outcome
• IFCs investment outcome
• IFCs effectiveness

19
Evaluation framework Development outcome

• Synthesis of projects overall impact on
  countrys development
• How well a project has contributed to IFCs
  purpose mission
• Compared to what would have happened without the
  project
• Based on 4 indicators

20
Development outcome Synthesis of 4 indicators
( their success standards)

• Business performance ?Financial viability
  (FRR gt WACC)
• Economic sustainability ?Economic viability
  (ERR gt 10, plus qualitative aspects)
• Environmental (ESHS) ?Compliance
  witheffects IFCs safeguards
• PSD ?Demonstration effects (e.g.
  technology, governance), linkages,
  changes in legal and regulatory framework

21
IFCs investment outcome Synthesis of 2
indicators

• Gross contribution loan ?Loan will earn
  expected interest income
• Gross contribution equity ?Return on equity
  (ROE) exceeds fixed rate loan plus
  premium

22
IFCs effectiveness (work quality) Synthesis of
3 indicators

• Screening, appraisal, ?Materially sound
  structuring execution (e.g. operating
  policies and procedures, credit notes,
  etc.)
• Supervision, ?Appropriate monitoring
  administration (e.g. compliance with
  covenants,
• Role, contribution ?IFCs operating
  principles/strategy IFCs added
  value Client satisfaction
  (timeliness, etc.)

23
Project scorecard (sample)
24
Incentives are essential Awards for good XPSR
execution
25
Work quality drives outcomes

• Departments with the best work quality generally
  achieved the best scores for development outcome
• Development outcome is a key ingredient of the
  Scorecard

• Investment outcome similar pattern

26
Incentives are essentialInvestment department
scorecard

• Development Impact
• Profitability
• Client Satisfaction

• Average XPSR development impact scores
• New commitments which meet criteria for high
  additionality

• Department net income
• Department cost/income ratios
• Volume of commitments

• Quality of IFCs services
• Responsiveness and timeliness of IFCs services
• Overall satisfaction with IFCs services

27
Incentives are essentialInvestment department
scorecard

• Development Impact
• Profitability
• Client Satisfaction

• Average XPSR development impact scores
• New commitments which meet criteria for high
  additionality

• Department net income
• Department cost/income ratios
• Volume of commitments

• Quality of IFCs services
• Responsiveness and timeliness of IFCs services
• Overall satisfaction with IFCs services

28
Project evaluation - summary

• Tailored to IFCs needs
• 3 key performance dimensions
• Aggregatable results
• Coherent measures incentives

29
Overview

• OEG in the World Bank Group
• Project evaluation in IFC
• Beyond project evaluation

30
Evaluation learning from experience
OEGs synthesis
Generation
Annual reviews
XPSRs OEGs mini XPSRs
OEG special studies
The basic building blocks
IFCs
Evaluation
Learning
Cycle
Dissemination
Application
Policy and strategy
Screening
Appraisal
Structuring
Board approval
Supervision and evaluation
31
XPSRs contributing to learning in IFC

• Case Studies
• Training
• Publications

Top Lessons (country, sector, subject)
Portfolio Presentations
Annual Review
Special Studies
Lessons Retrieval Network (LRN)
Learning by doing
IOs using XPSRs
XPSR
32
OEGs annual reporting to Board

• Annual Review of IFCs Evaluation Findings
• Draws from all evaluation findings
• Recommendations to improve IFCs results
• Annual Report on Operations Evaluation in IFC
• Functioning of evaluation processes
• Recommendations to improve processes
• Work Program and Budget
• Agreed among Management, DGO, Board

33
OEGs special studies

• Country
• Brazil (2003)
• Russia (2002)
• Bulgaria (2002 mini)
• Vietnam (2001)
• Kazakhstan (2000)
• Tanzania (2000)

34
OEGs special studies

• Sector
• Extractive industries (2003)
• Agribusiness (2003)
• PSD in energy (2003)
• Information infrastructure (2000)
• African financial intermediaries (1999)

35
OEGs special studies

• Program and thematic evaluation
• Project development facilities (ongoing)
• Equity rating reliability (2003)
• Project economic analysis (2000)
• Africa Enterprise Fund (2000)
• Forest strategy in IFC (2000)
• Foreign Investment Advisory Service (1998)
• Problem projects (1997)
• Market assessments (1997)

36
OEG Vision Embedded evaluation

• Supervision provides aggregatable, real-time
  information about entire portfolio
• Financial viability
• Economic viability
• Environmental compliance
• Achievement of PSD objectives
• Management reports portfolio results
• OEG evaluation shifts focus
• Synthesis analysis and dissemination dominate
• Quality control function reduced

37
An evaluation system should provide

• Relevant learning
• Credible accountability
• Coherent measurement
• Learning incentives