Exercise Unit 1

1
Exercise Unit 1 Jennifer Maddrell R561
Evaluation Change in the Instructional
Development Process Indiana University Professor
Knuth May 29, 2006
2
Branching OutTo a new Training Evaluation Plan
3
Evaluation is a means to

• Brinkerhoff Dressler
• Increase the value of the training investment
• Kaufman Watkins
• Compare results with intentions
• Morrison, Kemp Ross
• Make judgments about the worth or success of the
  training program
• Phillips
• Capture the contribution of human resource
  development
• Determine customer satisfaction
• Van Tiem, Moseley Dessinger
• Judge the results of performance
• Trigger or support a decision

4
An evaluation plan will provide

• Feedback
• As a quality control measure
• Control
• To assess value and worth to organization
• Research
• For knowledge to improve
• Intervention
• To affect how training is viewed, used and shared
• Power
• To fairly represent results to stakeholders

Source Knuth, 2006
5
Questions we must answer

• How do participants feel about our training
  program?
• Are participants learning?
• Is their learning transferring to the job?
• Does the organization benefit from our training
  efforts?

6
Finding answers

• Kirkpatricks 4 Levels of Evaluation
• Level 1 Reaction
• How do participants feel about our training
  program?
• Level 2 Learning
• Are participants learning?
• Level 3 Behavior
• Is their learning transferring to the job?
• Level 4 Results
• Does the organization benefit?

Source Kirkpatrick,1998
7
Kirkpatricks Model Measures

• Level 1 Reaction
• Customer satisfaction
• Learners motivation to learn
• Level 2 Learning
• Attitude change
• Knowledge improvement and skill gain
• Level 3 Behavior
• Change in behavior
• Transfer of skills
• Level 4 Results
• Financial Impact On costs, production, etc.
• Non-financial Impact On morale, motivation, etc.

Source Kirkpatrick,1998
8
Appeal of Kirkpatricks Model

• Assesses important areas
• Widely known
• Simple framework
• Easy to explain and understand

9
However . . .

• Widely Know ? Widely Used
• Level 1 Often (over 90)
• Level 2 Sometimes (less than 35)
• Level 3 4 Rarely (less than 15)
• Why is this a problem?
• Level 3 and 4 often perceived as
• Difficult to measure
• Time consuming
• Beyond the realm of most trainers
• Level 1 result does not always mean similar
  Learning / Transfer / ROI results

Source Pershing Gilmore, 2004
10
Other problems

• Undermines Management Partnership
• Training ? Silver Bullet
• Training is only one strategy within entire
  Performance System
• Level 3 4 should include evaluations of entire
  Performance System - not just training
• Lacks Performance System Focus
• What about rest of Performance Environment?
• What factors impede / enable usage of training?
• Feedback Goes to Wrong People
• Feedback to training function only is incomplete
• Must include Performance Environment owners

Source Brinkerhoff Dressler, 2002
11
Alternatives to Kirkpatrick

• Numerous alternative evaluation options exist
• Recommend implementation of Brinkerhoff
  Dresslers Success Case Evaluation Model
• This alternative focuses on trainings business
  impact as part of entire performance system.

12
Success Case Model Answers

• What is the business impact of instructional
  program?
• What is the organization doing that is
  facilitating performance improvement?
• What is the organization doing that is impeding
  performance improvement?

Source Brinkerhoff Dressler, 2002
13
Success Case Model Approach

• Brief survey to large sample to assess
• To what extent have you used your recent
  training in a way that you believe has made a
  significant difference to the business?
• In depth small sample review of both
• successful groups
• nature and business value of their application of
  learning, and
• performance context factors (support)
• unsuccessful groups
• performance context factors (obstacles)
• other factors preventing use of learning

Source Brinkerhoff Dressler, 2002
14
In Summary

• An evaluation tool must be integrated into the
  training programs at BIG.
• Kirkpatricks Four Levels of Evaluation is well
  known, but has limitations
• Unlikely completion of all 4 Levels
• Lacks performance system focus
• Brinkerhoff Dresslers Success Case Evaluation
  Model is the recommended next step approach.
• Rapid evaluation / feedback process
• Addresses key business impact issues
• Contemplates entire performance environment

15
References

• Brinkerhoff, R. O. Dressler, D. (in press).
  Using evaluation to build organizational
  performance and learning capability A strategy
  and a method. Performance Improvement.
• Kaufman, R., Keller, J., Watkins, R. (1995).
  What works and what doesn't Evaluation beyond
  Kirkpatrick. Performance Instruction, 35, (2).
  205-209.
• Kirkpatrick, D. L. (1998). The four levels An
  overview. Ch. 3 in Evaluating Training Programs
  The Four Levels, 2nd ed. San Francisco
  Berrett-Koehler.
• Knuth, R. (2006). Unit 1 Introduction to
  Evaluation. Retrieved May 18, 2006, from Indiana
  University R526 Syllabus http//www.indiana.edu/i
  str561/knuth06sum/unit1print.html
• Morrison, Gary R., Kemp, Jerrold E., Ross,
  Steven M. (2001). Chapter 10, The Many Faces of
  Evaluation. In Designing Effective Instruction
  (3rd edition). New York John Wiley Sons.
• Pershing, J., Gilmore, E. (2004). Evaluating
  Training Programs Kirkpatricks Four Levels.
  PowerPoint Presentation Retrieved May 18, 2006,
  from Indiana University R526 Syllabus
  http//www.indiana.edu/istr561/knuth06sum/PPTs/r5
  61d1_kirk_perspective.ppt
• Phillips, J. (1997). Handbook of Training
  Evaluation and Measurement Methods (Improving
  Human Performance Series) 3rd Edition.
  Butterworth-Heinemann. Chapters 1 - 3.
• Van Tiem, Darlene M., Moseley, James L.,
  Dessinger, Joan Conway (2004). Chapter 7,
  Evaluation. In Fundamentals of Performance
  Technology A Guide to Improving People, Process,
  and Performance, 2nd Edition. Washington, DC
  International Society for Performance Improvement.