Professionalization of evaluation

1
Professionalization of evaluation recent
activities within DeGEval

• Dr. Jan Ulrich HenseDepartment Psychologie,
  Empirische Pädagogik und Pädagogische
  Psychologiejan.hense_at_psy.lmu.de
• Jahreskongress SEVAL 2012, Workshop 2 Freitag,
  7. September 2012Université de Fribourg

2

• (1) Some comments and questions on the
  professionalization of evaluation

3

• Two uses of professional
• Referring to a job or designation (a
  professional painter)
• Is evaluation developing into a full-blown
  profession?
• Referring to performance on a task (a
  professional repair)
• How to carry out evaluations professionally?

4
Evaluators?

• (Mieg, 2010, p. 16)

5

• Two Classes of Professions (Mieg, 2010)
• Care and relationship orientation
• Clients or patients trust professionals
  expertise and integrity
• Readiness to work cost free in emergency
  situation
• e.g. Doctors, Lawyers, Psychotherapists
• Market orientation
• Price competition
• Income orientation
• e.g. Engineers, Consultants

6

• Professionalization as regulation of autonomy
• Definition of education and training demands
• Control of market entry
• Determination of fees and charges
• Monopoly of performance evaluation

(Mieg, 2010)
7

• Conditions for the development of a new
  profession
• Combination of
• a problem relevant for society and
• knowledge on how to explain and deal with the
  problem
• Relation to central values of society (e.g.
  health, justice)
• (Mostly) academic education and training
• Professional societies

(Mieg, 2003)
8

• Phases of development of a profession (Wilensky,
  1964)
• Some are working in a new area (mostly) full time
• Education and training programs are developed
• Professional societies emerge
• Shaping profile by delegating lower tasks
• Conflict between established and graduates of new
  training programs
• Competition with neighboring professions
• Professional Codes und Standards
• Move to formal closure
• Rules and conditions for working in the new
  profession
• Protection of designation
• Declaring non licensed work illegal

9

• (2) Recent DeGEval activities in the context of
  professionalisation
• Competencies for evaluators
• Didactics of Evaluation
• Further projects and plans

10

1. Competencies for evaluators

11

• Why develop "Recommendations on Education and
  Training in Evaluation"?
• Growing demand for evaluation services
• Lack of qualified personnel
• Lack of a professional profile
• Guidelines as contribution to the
  professionalization of evaluation

12

• A bit of history...
• 2004 First German edition
• Authors A. Caspari, M. Hennen, D. Scheffler, U.
  Schmidt O. Schwab
• Published by the DeGEVal working group
  "Education and Training in Evaluation
• 2008 English translation

download/order on www.degeval.de
13

• Goals and intended uses
• Standards for
• Training programmes
• Direction for programme developers and teachers
• Evaluation of existing programmes
• Assessing evaluators
• Choice of an evaluator by clients
• Self-evaluation of evaluators

14

• "A competency is more than just knowledge and
  skills. It involves the ability to meet complex
  demands, by drawing on and mobilising
  psychosocial resources (including skills and
  attitudes) in a particular context."
• (OECD/OCDE DeSeCo-Project)
• Which knowledge, skills and attitudes are needed
  for conducting professional evaluations?

15
(No Transcript)
16
(No Transcript)
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21

• Development of the recommendations
• Theoretical approach
• Based on DeGEval standards of evaluation
• Practical procedure
• Expert panel
• Feedback cycles from DeGEval members
• Officially adopted by DeGEval conference

22

• Who uses the guidelines?
• Usage in advanced study programmes
• Berne ?
• Saarbrücken ?
• Bonn ?
• Further education courses ?
• Evaluation courses within domain-specific
  curricula ?

23
Coverage of competencies in psychology diploma
programmes
Soellner, R., Hapkemeyer, J. Scheibner, N.
(2010). Evaluationsausbildung in der Psychologie
eine Bestandsaufnahme. Zeitschrift für
Evaluation, 9 (2), 297-308.
24

• Need for revision (to be discussed)
• Concept of competencies (KSA)
• Field-specific interpretation and differentiation
• Elaboration of methodological competencies
• Compatibility with international models of
  evaluator competencies, e.g.
• King et al. (2001 2005) Mertens (1994)
• Zorzi, McGuire Perrin (2002)
• Russ-Eft et al. (2008)

25

• 2. Didactics of Evaluation

26
http//www.degeval.de/arbeitskreise/aus-und-weite
rbildung-in-der-evaluation/publikationen-und-prod
ukte
27

• Why didactics of Evaluation?
• Competency profiles
• Contents and learning goals of training
  programmes
• ? What to teach
• Didactics of evaluation
• Practical guidelines for teachers and trainers
• ? How to teach

28

• Goals
• Provide guidance for
• training and education programmes
• courses in evaluation
• self-learning materials (textbooks etc.)
• Based on current state of theories on learning
  and instruction
• Supplement existing recommendations on competency
  profiles

29

• 0 Executive Summary
• Introduction
• Learning contents and goals Competence profiles
  for evaluators
• Programmes
• Types of programmes and target groups
• Didactical foundation
• Learning and teaching methods
• Requirements for teachers/trainers
• Outlook

CondensedCookbook guidelines
30

• Important Messages
• Cover whole range of competencies (KSA)
• Apply a mix of construction and instruction
• Emphasize active, self-directed learning
• Allow for learners own practical experiences
• Provide instructional support
• Embed all contents in authentic evaluation
  contexts
• Match method and learning goals
• Adapt to learners individual needs and
  preconditions

31

• Examples
• Chocolate cookie exercise
• Exercise on the logic of evaluation
• Reflection exercise An evaluator is like a ...
• Discuss different evaluator roles and
  responsibilities
• Role play stakeholder resistance
• Awareness for possible sources of resistance to
  evaluation

32

• 3. Further projects and plans

33
Future projects

• Other topics and activities of the DeGEval
  working group Education and Training in
  Evaluation
• Overview of training courses and seminars
  (online-database)
• Textbook overview and review
• Evaluation capacity building
• Certification and credentialing of evaluators

34
Future projects

• Overall DeGEval efforts
• Work group on ethics in evaluation (ethical
  codes)
• Revision of evaluation standards (until 2014)

35

• Thank you for your interest!
• contactjan.hense_at_psy.lmu.de