Levels of Evaluation

1
Levels of Evaluation

• By
• Troy Brady
• ABS Consulting

2
Different Approaches to Evaluation

• Different approaches emphasize different
  questions regarding a specific practice.
• Some evaluations are primarily quantitative.
• Some are primarily qualitative.
• Some may combine both quantitative and
  qualitative methods.

3
Different Approaches to Evaluation

• 1. Objectives-oriented approaches, Specifying
  goals and objectives and determining the extent
  to which they have been attained. 2.
  Consumer-oriented approaches, Developing
  information on educational "products use by
  educational consumers in choosing among competing
  curricula, instructional products and the like.
  3. Expertise-oriented approaches, Direct
  application of professional expertise to judge
  the quality of educational endeavors, especially
  the resources and the processes.

4
Different Approaches to Evaluation

• 4. Decision-oriented approaches, Emphasis is on
  describing and assessing an educational change
  process and resulting outcomes to provide
  information to a decision-maker. 5.
  Adversary-oriented approaches, Planned opposition
  in points of view of different evaluators is the
  focus of the evaluation. 6. Naturalistic and
  participant-oriented approaches, Inquiry and
  involvement of participants are central in
  determining the values, criteria, needs and data
  for the evaluation.

5
Kirkpatricks Premises

• In Kirkpatrick's four-level model, each
  successive evaluation level is built on
  information provided by a lower level.
• Evaluation should always begin with level one,
  and then, as time and budget allows, should move
  through levels two, three, and four.
• Information from each prior level serves as a
  base for the next level's evaluation. Each
  successive level represents a more precise
  measure of the effectiveness of the training
  program, but requires a more rigorous and
  time-consuming analysis.

6
Level 1 Evaluation Reactions

• Measures how participants in a training program
  react to it.
• It answers questions like
• Did they like it?
• Was the material relevant to their work?
• How were materials, facilities and Instruction?
• Every program should be evaluated at this level
  to provide training program improvement.

7
Level 1 Evaluation Reactions

• Participants' reactions have important
  consequences for learning (level two).
• Although a positive reaction does not guarantee
  learning, a negative reaction almost certainly
  reduces its possibility.
• You cant do four levels of evaluation without
  doing the first one.

8
Level 2 Evaluation - Learning

• To assess the amount of learning,level two
  evaluation often use pretest and post tests to
  determine the amount of learning that has
  occurred.
• Attempts to assess the extent students have
  advanced in skills, knowledge, or attitude.
• More difficult and laborious than level one.

9
Level 3 Evaluation - Transfer

• Measures changes in learners' behavior due to the
  training.
• Are the newly acquired skills, knowledge, or
  attitude being used in the everyday environment
  of the learner?
• The truest assessment of a program's
  effectiveness.
• Measuring at this level is difficult since its
  impossible to predict when the change in behavior
  will occur.

10
Level 4 Evaluation- Results

• Level four evaluation assesses training in terms
  of business results.
• Measures the success of the program in terms that
  managers and executives can understand.
• From a business perspective, this is the reason
  for a training program.
• Determining results in financial terms is
  difficult to measure, and is hard to link
  directly with training.

11
(No Transcript)
12
ROI

• It's difficult to assign a dollar figure to
  learning's most important benefits.
• For example can you calculate how much better
  your managers communicate with their direct
  reports since they took that course on online
  communication skills?

13
ROI Models

• Two models for determining the value of training
  experiences.
• Companies can use those formulas to calculate the
  ROI of learning initiatives.
• Developed by Donald Kirkpatrick and enhanced by
  his colleague Jack Phillips, the two form a
  framework to examine ROI from a human and
  business performance perspective.

14
Premises of ROI Models

• The models evaluate training benefits on several
  levels.
• Critical to both is the concept of "chain of
  effect," which links each benefit level to
  others.
• Each level of measurement depends on the previous
  level, as well as the next.
• Without this link, it's difficult to conclude
  that training is responsible for improvements in
  performance.

15
Comparison of ROI Models

• Kirkpatrick's original model considers the value
  of training on four levels.
• Philips expanded on Kirkpatrick's model
  suggesting that another level could calculate a
  company's return on investment.
• You cannot measure ROI at the fifth level without
  taking accurate measurements at the other four
  levels.

16
Level Five Evaluation

• Level V asks, "Did the monetary value of the
  results exceed the cost of training?"
• This is the measurement of ROI, which can be
  calculated in several ways.

17
TACTP - TACNP PNS

• Subtract the total administrative costs of the
  new program (TACNP) from that of the former
  training program (TACTP) gives the projected net
  savings (PNS) for training administration.
• Although cost savings are important, ROI is much
  more than that.

18
TCT of students CPS

• Divide the total cost of training (TCT) by the
  number of students gives the cost per student
  (CPS) of the training.
• This is useful, but again is not a true measure
  of return on investment.
• Both formulas, do not measure what monetary value
  or profit is derived from a training investment.

19
TB (in ) x 100 / TTC ROI

• Multiply the total benefits (TB) of training in
  dollars by 100 and divide by the total training
  program cost (TTC).
• The true percentage of ROI in a new program.
• This formula is the most accurate of the three.

20
Conclusion

• It's often difficult to show increased value of a
  company's human capital.
• But if you dont your training investment ends up
  on the expense side of the balance sheet, ripe
  for budget slicing.
• By measuring the results of training and tying
  training to strategic business success, a company
  can increase and demonstrate the increase in its
  return on training dollars.

21
References

• Assessing Training Effectiveness by Donald
  Kirkpatrick (1959).
• Accountability in Human Resource Management by
  Jack Phillips