Prospective

1

• Session 3
• Prospective proposing changes to the
• Evaluation Quality Standards
• Group Work

2

• Purpose of EQS
• To improve the quality and credibility of
  evaluation in order to improve development
  cooperation.
• These are standards to inform good practice for
  evaluation process product (?)
• They are not mandatory. They are generally
  applicable to the DAC member countries, but can
  be used by partners and all parties.
• They are intended to be generic for wide
  application. Members are expected to adapt them
  to different contexts, operational mandates and
  types of evaluation.
• This is an opportunity to alter the EQS document
  taking into consideration the current development
  context and all our discussions.

3
Session 3 Prospective Proposing changes to the
EQS

• Group Work
• In the light of the discussion on the purpose and
  role of the Evaluation Quality Standards what
  changes (if any) need to be made to the
• Domains (or section headings) of the document
  e.g. rationale, evaluation scope, context etc.
• Structure of the document
• Content of each domain/section.

4

• 1. Group 1 Cop 1.
• 2. Group 2 Cop 1
• 3. Group 3 Cop 2 (2nd floor)
• 4. Group 4 Cop 2 (2nd floor)
• 5. Group 5 Cop 3 (12th floor) Spanish
  speaking group
• 6. Group 6 Cop 4 (12th floor) Francophone
  group.
• Each group choose facilitator reporter.
• 1 ½ hours.
• Each group feedback on PowerPoint/or flip chart
  (up to 3 sheets)
• 5 minutes to feedback (after lunch).
• (Francophone group in French)

5
French Group Discussionon Standards10
February 09Justine Odjoubé, Modibo Makalou,
Amadou Tidiane Dia, Benoît Chervalier, Gerhard
Siegfried, Dominique de Crombrugghe
6
General structure of the text

• The group agrees with the existing structure
• The introduction should mention de purpose of the
  document along the lines drafted in the morning
• The document is meant to be sufficiently flexible
  to be useful and general to be applicable.
• Depending on the users, certain points will be
  highlighted more than other.

7

• 1. Rationale, purpose and objectives
• Importance of the rationale, purpose and
  objectives why and for whom do we do this
• Leave examples and work with definitions
• Use definitions in glossary
• However this will lead to some problems because
  there are quite some remarks about the glossary

8

• 2. Scope
• Importance to mention the 5 criteria
• 3. Context
• A clear reference to content of PD and Accra do
  we proceed in a context of harmonised donors and
  alignment or not?
• New 3.4. reference to the cultural context
  evaluation should examine how the cultural
  context has been considered and also respect that
  context

9

• 4. Methodology
• 4.3. Relevant stakeholders (partners) are not
  only involved they are encouraged to fully join
  the evaluation process, from the very beginning

10

• 5. Information sources
• No comment
• 6. Independence
• No comment
• 7. Ethics
• No comment
• 8. Quality insurance
• No comment

11

• 9. Relevance of results
• New 9.4. Implication of stakeholders in the
  dissemination of the evaluation results
• 10. Completeness
• The notion of lessons learned, as opposed to
  conclusion / recommendations is unclear
• The final document should be internally coherent
• 10.4. In French, use Synthèse rather than
  Résumé and ensure it is a document that can be
  read independently from the report

12

• For further discussion
• Should we better distinguish an approach towards
  high quality evaluation and technical aspects?
• Should we make a cleare distinction between
  methodology and process?
• Recall that the standards will be applied by a
  multiplicity of publics

13

• On average, evaluations do not perform well where
  it comes to efficiency
• Should financial aspects, inluding evaluation
  budget be better dealt with?
• Should the need for quantified data be stressed?
• In the context description, should the evaluation
  address the risks?
• In 6. 2 should we add a reference to a steering
  committee ?

14

• Group 1
• Feedback

15
What changes need to be made to the

• domains
• Introduction Clarify make reference Accra
  Agenda for Action
• Establish an indicative grid ? product
• Take away section 5 (Information sources)
• add 5.1 under Ethics (also add 8.1 under Ethics)
• add 5.2 under Methodology

16
What changes need to be made to the

• structure of the document
• No changes needed

17
What changes need to be made to the

• content of each domain/section
• Review of wording for a better reflection of
  Paris Declaration (partnership perspective)
• 2.3 on criteria Some or all criteria could be
  used

18
What changes need to be made to the

• content of each domain/section
• 4.2 The group question if the indicators really
  always have to be SMART? (M measuring)
• 9.4 Split in two
• Management response
• (9.5) Dissemination disclosure should be ensured
  (not necessarily by management)

19
What changes need to be made to the

• content of each domain/section
• Look at potential overlap between 9.1 and 10.2

20
DAC EVALUATIONS QUALITY STANDARDS WORKSHOP

• GROUP TWO (2)
• PRESENTATION

21
Not Sufficient time for Task!

• General Comments
• Generally OK better editing rather than
  different content often headings dont match text
• Dividing process/product re-editing

22
Introduction

• Include AAA (after reference to Paris
  Declaration)
• A living Document
• More generic beyond DAC Countries

23
Sections 1, 2 3

• Attention needed to order (for instance)
• section 3 before 2
• section 2.4 (Evaluation Questions) after 1.3
  (objectives)
• Section 3.4 (Implementation) in Section 2.9 refer
  to AAA in here
• Section 2.2 should it be in context
• Opportunity under context to refer to capacity
  building which must be country led and owned
• Section 4 (Methodology) needs work
• Relevant Stakeholders (section 4.3) principle
  (not method) sampling why there? Team why
  there?

24

• Section 6
• Independence is a principle - emphasize
  independence unless explain otherwise
• Add comment from task team on joint evaluation
  (6.3)
• Section 7 Ethical Issues
• add conflict of interest
• Disclosure
• Informing participants
• Delete when requested (confidentiality)

25

• Section 8 Quality Assurance - stakeholder
  involvement throughout the standard (not just
  quality assurance)
• Section 10 Completeness no relationship
  between heading and text

26
Group 3Proposed structure

• Planning
• Design
• Conduct
• Product
• Use and dissemination

1. Introduction
2. Principles
3. Standards
4. Annexes to be determined

27
1. Introduction

• Should link with AAA
• Prepared for and approved by DAC members as
  guidance for DAC members optional to others
• Partner/donor consultative process used to
  develop standards
• Constructive nature of evaluations
• Dissemination to make it available to all
  partners in the interest of transparency
• Evaluation major role for mutual accountability
  for results and transparency (task force)
• Make sure purpose of Standards is captured
  (Pennys summary)

28
3. Standards

• Planning - Add point to emphasise partnership,
  limits of the scope. Use from standards 1.1,
  1.2, 1.3, 2.1, 3.1, 3.2, 3.3, 4.5, 6.1, 6.2
  (carry throughout)
• Design - Use from standards 2.2, 2.3, 2.4, 3.4
• Conduct - Use from standards 4.3, 7.1 and add
  more on methodology
• Product - Use from standards 4.1, 4.2, 4.4, 5.1,
  5.2, 7.2, 8.1, 8.2, 9.1, 9.2, 9.3, 10.1, 10.2,
  10.3, 10.4
• Use and dissemination - Use from standards 9.4
  and add more on dissemination

29
What changes are proposed?

• Group 4

30
Introduction

• Focus on process and product, but with a heavier
  focus on product
• DAC standards but must be inclusive and promote
  wider usage/ relevance (including referring to
  other relevant documents)
• Purpose should be explicit (to improve quality)
• Refer to Accra Agenda for Action

31
Section 1 (rationale, purpose, objectives)

• Re-examine whether all 3 required opportunities
  to refine
• A good evaluation will clearly articulate its
  rationale and purpose and/or objectives, will be
  guided by who the reader is and will produce a
  clear, targeted message
• Users to adapt to needs, context, operational
  mandates

32
Section 2 Evaluation Scope

• Change title to Evaluation scope and focus
• Should be section 3 (and vice versa) (context
  before scope)
• 2.2 is this the right place for it?
• Evaluation criteria
• avoid overlap with objectives
• cross-cutting issues should be included (gender,
  environment, human rights)
• criteria should address both the accountability
  dimension (the what) and the learning dimension
  (the why)

33
Section 3 Context

• Include background information about what you
  are evaluating (bring in 2.2 here, i.e. including
  logic framework)

34
Section 4 Methodology

• Re-title Evaluation approach and methodology
• Bring in PD/AAA elements here (i.e. around
  partnership, joint approaches) in a preamble
• 4.1 contents need to be unpacked, resulting in a
  re-framed section 4
• Research design
• attribution/plausibility
• counter-factual
• Data collection strategies
• recognise there are multiple lines of evidence
• stakeholder consultation included here (inc
  partner countries)

35

• 3. Sampling (retain)
• 4. Analytical techniques for treating the data
• assessment of results comes in here
• 5. Data/ information sources (currently section
  5) should be included here
• 6. Limitations/ constraints

36
Sections 6,7,8

• Section 6 retitled governance
• independence
• Transparency and accountability
• Management
• includes QA (section 8)
• Includes 4.5

37
Sections 9 10

• Reports must be structured and organised to make
  messages accessible
• Structure of document should reflect life cycle
  of evaluation
• Should promote clear, logical products

38
Concluding general comments

• DAC should take a more proactive role to
  disseminating guidance, including to partner
  countries

39
GROUP FIVE
40
Key points

• We strongly support the work of the Task Team on
  New Context for Development Evaluation as a
  important starting point for a reviewing the
  standards in light of the PD and AAA
• We also suggest following up with this useful
  process

41
A. DOMAINS OF THE DOCUMENT RATIONALES,
EVALUATION AND SCOPE, CONTEXT

• Add explicit reference to commitments related to
  Paris Declaration and AAA

42
B. STRUCTURE OF THE DOCUMENT

• Explain explicitly relationships between
  standards and evaluation process (design, content
  and results)
• Change the order of the point between 3.3 and 3.2

43
C. CONTENT OF EACH DOMAIN-SECTION

• 1.2 Bullet 1. With the goal of improving
  development results
• 2.3 Inclusion of PD Principles and Commitments
  made under the AAA
• 3.1 Considering the range of the development
  policies of both donors and partner countries in
  a broader sense than aid policy