Diapositivo 1

1
Results of the midterm evaluation exercise on the
Leader programme for Portugal
Special focus on evaluating innovation Pedro
Afonso Fernandes (CIDEC Lisbon -
Portugal) Expert meeting on Guidance for Common
Monitoring and Evaluation Framework Seminar on
monitoring and evaluation of the LEADER
approach DG AGRI, Brussels Room LOI 130B 25
September 2006
2
Innovation on evaluating innovation

• The innovative starting point
• The 3 3 evaluation matrix
• Specific evaluation tools
• Deal with lack of self-evaluation
• Recommendations on evaluation procedures and tools

3
The innovative starting point (I)

• Evaluating a innovative programme such as LEADER
  requires a innovative starting point at two
  levels
• Community level Guidelines produced by STAR
  committee (Doc. VI/43503/02-Rev.1, January 2002)
• Detailed Common Evaluation Questions and Criteria
  that cover a wide range of evaluation topics
• Implementation of the LEADER method
• Implementation of the 3 actions (integrated pilot
  strategies, co-operation and networking)
• Impact Overall objectives of the Structural
  Funds
• Impact Specific objectives of LEADER (Value
  Added)
• Financing, management and evaluation

4
The innovative starting point (II)

• National level the midterm evaluator (CIDEC)
  developed specific evaluation questions in order
  to cover the main concerns of the national
  authority (IDRHa Institute of Rural Development
  and Hydraulics), namely
• Evaluation of the LAGs functioning and practices
• Evaluation of local strategies appropriateness
  and sustainability
• Evaluation of the monitoring and information
  systems
• Note the specific (national) evaluation
  questions complement the DG AGRIs Common
  Questions
• Those additional questions reinforce the
  evaluation of LEADERs value added (e.g.
  development of competences at local level,
  strategies sustainability, diversification
  effects)

5
The 3 3 evaluation matrix

• Evaluating LEADER is a challenge that results
  from the cross of 3 levels of intervention with 3
  types of action

NO TARGETS
6
Specific evaluation tools (I)

• Multiple methodological tools were mobilized in
  order to deal with the 3 3 matrix

7
Specific evaluation tools (II)

• Case studies
• 26 LAG covered (one half of the total)
• Interview with LAGs coordinator
• Contact with 2 projects from Action 1 and
  interview
• Analysis of the administrative processes
  (dossiers) of 10 projects (9 from Action 1 and
  one from Action 2)
• Telephonic contact with some local partners
• Documental analysis (e.g. local strategies,
  criteria to select projects)
• Quantitative treatment of qualitative information
• Detailed report

8
Specific evaluation tools (III)

• Promoters Inquiry
• Main objective find values for a set of
  indicators specially developed by the midterm
  evaluator with the support of the national
  authority (IDRHa)
• The inquiry was applied with 9 different
  questionnaires according to the type of project ?
  Good response rate (51) and reliability of the
  data
• Quality control procedures statistical tests
  that compared the data from the inquiry with the
  data from the case studies
• Impact and results indicators were favoured in
  order to quantify the LEADERs value added

9
Specific evaluation tools (IV)

• Examples of impacts and results measured by the
  Promoters Inquiry for Portugal
• 63 of the projects incorporated technologies
  (43 ITC)
• 40 of the projects are innovative (product or
  process)
• 10 of the projects involved universities or
  polytechnics
• 60 of the productive projects mobilized local
  raw materials and/or services
• 70 of jobs created were for women
• LEADER stimulates the development of competences
  on 40 of the promoters inquired and
• the development of new different activities (no
  financed) on 12 of the projects
  Diversification effects

10
Deal with lack of self-evaluation

• Only 5 on 26 LAG studied by the midterm evaluator
  have permanent self-evaluation practices
• Even if the LAG has this kind of practices, an
  independent evaluator are not mobilized typically
  
• The midterm evaluator deals with those problems
  by involving LAGs representatives and (selected)
  promoters and partners in the case studies works
• In the future, national (or even European)
  authorities should develop a vademecum or guide
  to favour the self-evaluation by the Local Action
  Groups

11
Recommendations on evaluation procedures and
tools (I)

• At European level, it is important to develop the
  Common Evaluation Questions and Criteria, that
  is, The Starting Point by
• Incorporating new questions/criteria, namely,
  suggested by midterm evaluators
• Simplifying some topics (e.g. the CEQ 4.1 To
  what extent has LEADER contributed to promote
  and disseminate new integrated approaches to
  rural development () was already covered by the
  action-specific CEQ)
• Developing a Common Set of Indicators that would
  facilitate the impact and results evaluation at
  European and National levels (the Portuguese set
  already developed could be a starting point for
  that huge task)

12
Recommendations on evaluation procedures and
tools (II)

• At National level, it is important
• To establish targets (quantified goals) for local
  strategies, co-operation and networking
• To develop Complementary Evaluation Questions,
  Criteria and Indicators in order to reinforce the
  evaluation of LEADERs value added and
  incorporate national concerns
• To develop a mix of evaluation tools that could
  solve the puzzle of the 3 3 Matrix
• Case studies should be privileged because of the
  innovative local-based nature of LEADERs
  projects
• Project inquiries are essential to collect
  quantitative data, namely impact and results
  indicators