Training Evaluation

1
Training Evaluation
2
Evaluation of Training

• Since huge sums of money are spent on training
  and development, how far the programme has been
  useful must be determined.
• The question of what to evaluate is crucial to
  evaluation strategy. The answers depend on the
  type of training programme, the organization and
  the purposes of evaluation.

3
Objectives of evaluation

1. To determine success in accomplishing Programme
   Objectives
2. To identify the strengths and weaknesses in the
   training process
3. To compare the costs to the benefits of a
   Training Programme
4. To decide who should participate in future
   programmes
5. To test the clarity and validity of tests, cases
   and exercises

4

• 6. To identify which participants were the most
  successful with the programme
• 7. To reinforce major points made to the
  participant
• 8. To gather data to assist in marketing future
  programmes
• 9. To determine if the programme was the
  appropriate solution for the specific need
• 10. To establish a database that can assist
  management in making decisions

5
Criteria of Evaluation

1. Training validity- did the trainees learn during
   training?
2. Transfer validity- what has been learnt in the
   training, has it been transferred on the job?
3. Intra-organizational validity- is performance of
   the new group of trainees, consistent with the
   performance of the original training group?
4. Inter- organizational validity- can a training
   program validated in one organization can be used
   successfully in another organization?

6
(No Transcript)
7

• Feedback It helps in giving feedback to the
  candidates by defining the objectives and linking
  it to learning outcomes.Research It helps in
  ascertaining the relationship between acquired
  knowledge, transfer of knowledge at the work
  place, and training.

8

• Control It helps in controlling the training
  program because if the training is not effective,
  then it can be dealt with accordingly.Power
  games At times, the top management (higher
  authoritative employee) uses the evaluative data
  to manipulate it for their own benefits.
  Intervention It helps in determining that
  whether the actual outcomes are aligned with the
  expected outcomes

9
(No Transcript)
10

• Before Training
• The learner's skills and knowledge are assessed
  before the training program.
• During the start of training, candidates
  generally perceive it as a waste of resources
  because at most of the times candidates are
  unaware of the objectives and learning outcomes
  of the program.
• Once aware, they are asked to give their
  opinions on the methods used and whether those
  methods confirm to the candidates preferences and
  learning style.

11

• During Training It is the phase at which
  instruction is started. This phase usually
  consist of short tests at regular intervals

12

• After Training It is the phase when learners
  skills and knowledge are assessed again to
  measure the effectiveness of the training.
• This phase is designed to determine whether
  training has had the desired effect at individual
  department and organizational levels.
• There are various evaluation techniques for this
  phase.

13

• The various methods of training evaluation are
• Observation
• Questionnaire
• Interview
• Self diaries
• Self recording of specific incidents

14
The Four Levels

• Reaction
• Learning
• Behavior
• Results

15
All about Kirkpatrick

• In 1959, Kirkpatrick wrote four articles
  describing the four levels for evaluating
  training programs. He was working on his
  dissertation for a Ph.D. when he came up with the
  idea of defining evaluation.
• Evaluation, as according to Kirkpatrick, seems
  to have multiple meanings to training and
  developmental professionals. Some think
  evaluation is a change in behavior, or the
  determination of the final results.

16
KirkPatricks four levels of evaluation of
training impact

• Level
• Reaction
• Learning
• Behaviour
• Results

• Questions
• Were the employees pleased with the programme?
• What did the participants learn in the programme?
• Did the participants change their behaviour based
  on what was learned?
• Did the change in behaviour positively affect the
  organization?

17

• The Four Levels represent a sequence of ways to
  evaluate (training) programs.As you move from
  one level to the next, the process becomes more
  difficult and time-consuming, but it also
  provides more valuable information.

18

• Reaction is the measuring of the reaction of
  the participants in the training program.
• what participants thought of the program,
  including materials, instructors, facilities,
  methodology, content etc.

19

• Learning is the change in the participants
  attitudes, or an increase in knowledge, or
  greater skills received, as a result of the
  participation of the program.
• learning evaluation is concerned with measuring
  the extent to which principles, facts, techniques
  and skills have been acquired

20

• Behavior how much transfer of knowledge, skills,
  and attitude occurs after the training.
• behavioral change is measured to determine the
  extent to which principles, facts, techniques,
  and skills have been acquired.

21

• Results is the most important and difficult of
  all - determining final results after training
• evaluation of results involves monitoring
  organizational improvement such as cost savings,
  work output changes, and quality changes.

22
Kaufmans five levels of evaluation of training
impact

• Some researchers, recognising some shortcomings
  of Kirkpatricks four-level approach, have
  attempted to modify and add to this basic
  framework.
• Kaufman has expanded the definition of level 1
  and added a fifth level addressing societal
  issues.

23
Level Evaluation focus
5. Societal outcomes Societal and client responsiveness, consequences and payoffs
4. Organizational output Organizational contributions and payoffs
3. Application Individual and small group (products) utilization within the organization
2. Acquisition Individual and small group mastery and competency
1b. Reaction Methods, means, and process acceptability and efficiency
1a. Enabling Availability and quality of human, financial, and physical resources input
24

• At level 1, the factor of the concept of enabling
  addressees the availability of various resource
  inputs necessary for a successful intervention.
• Level 5 is the evaluation of societal and client
  responsiveness. This moves evaluation beyond the
  organization, and examines the extent to which
  the performance improvement programme has
  enhanced society and the environment surrounding
  the organization.

25
The CIRO Approach

• Developed by War, Bird, and Racham, is a unique
  way to classify evaluation process.
• Four general categories of evaluation are
  described, which form the letters CIRO
• 1. Context evaluation
• 2. Input evaluation
• 3. Reaction evaluation
• 4. Outcome evaluation

26
Context evaluation

• Context evaluation involves
• collecting information about a performance
  deficiency,
• assessing the information to establish training
  needs and,
• on the basis of those findings, setting
  objectives.
• Context evaluation involves obtaining and using
  information about the current operational
  situation (or context) to determine training
  needs and objectives.

27

• Three types of objectives may be evaluated-
• Ultimate objectives (the particular deficiency in
  the organization that the programme will
  eliminate or overcome)
• Intermediate objectives (the changes in
  employees work behavior that will be necessary
  for the ultimate objectives to be attained)
• Immediate objectives (the new knowledge, skills,
  or attitudes that employees must acquire to
  change their behavior and reach the intermediate
  objective)

28
Input Evaluation

• Input Evaluation involves
• obtaining and using information about possible
  training resources to choose between alternative
  inputs to training.
• It involves analysing the resources available
  (both internal and external) and determining how
  they can be deployed so that there is a maximum
  chance of achieving the desired objectives.

29

• Input evaluation refers to the process of
  collecting evidence and using it to decide on the
  training methods

30
Reaction Evaluation

• Reaction evaluation involves
• obtaining and using information about
  participants reactions to improve the HRD
  process.
• The distinguishing feature of this type of
  evaluation is that it relies on the subjective
  input of the participants.
• Their views can prove extremely helpful when
  collected and used in a systematic manner.

31
Outcome Evaluation

• Outcome Evaluation
• involves obtaining and using information about
  the results or outcomes of training, and is
  usually regarded as the most important part of
  evaluation.
• If outcome evaluation is to be successful, it
  requires careful preparation before the programme
  begins.

32

• Stages of outcome evaluation-
• Defining trend objectives
• Selecting or constructing some measures of those
  objectives
• Making the measurements at the appropriate time
• Assessing the results and using them to improve
  later programmes.

33

• A successful training programme produces some
  initial change in a participant, an immediate
  outcome, which is reflected in changes of
  knowledge, skills, or attitudes. These changes
  can be measured during or at the end of the
  programme.

34
The Philips Five Level ROI Framework

• The ROI process adds a fifth level to the four
  levels of evaluation developed by Kirkpatrick.

35
Level Brief Description
1. Reaction and planned action
2. Learning
3. Job applications
4. Business results
5. Return on investment
36
1.Reaction and planned action

• Programme participants satisfaction is measured,
  along with a listing of how they plan to apply
  what they have learned.
• This level of evaluation is important as a
  participant satisfaction measure, a favorable
  reaction does not ensure that participants have
  learned new skills or knowledge.

37
2. Learning

• Measurements focus on what participants learned
  during the programme using tests, skill
  practices, role plays, simulations, group
  evaluations, and other assessment tools.
• A learning check is helpful to ensure that
  participants have absorbed the material and know
  how to use it.

38
3. Job application

• A variety of follow up methods are used to
  determine if participants apply what they learned
  on the job. Level 3 evaluations are important to
  gauge the success of the programmes application,
  it still does not guarantee that there will be a
  positive impact in the organization.

39
4. Business results

• The measurement focuses on actual results
  programme participants achieve as they
  successfully apply the programme material.
  Typical level 4 measures include output, quality,
  costs, time, and customer satisfaction.

40
5. Return on Investment

• Comparing the monetary benefits from the
  programme with its costs. It is usually presented
  as a percent or cost/benefit ratio.
• Very few organizations actually conduct
  evaluations at ROI level, perhaps because ROI
  evaluation is often characterised as a difficult
  and expensive process.

41
(No Transcript)
42
Steps to calculate return on investment (ROI),

1. Identify outcome(s) (e.g., quality, accidents)
2. Place a value on the outcome(s)
3. Determine the change in performance after
   eliminating other potential influences on
   training results.
4. Obtain an annual amount of benefits (operational
   results) from training by comparing results after
   training to results before training.

43
Steps to calculate return on investment (ROI),
(continued)

• Determine training costs (direct costs indirect
  costs development costs overhead costs
  compensation for trainees)
• Calculate the total savings by subtracting the
  training costs from benefits (operational
  results)
• Calculate the ROI by dividing benefits
  (operational results) by costs.
• The ROI gives you an estimate of the rupee return
  expected from each rupee invested in training.

44

• Although business results and ROI are desired, it
  is important to evaluate the other levels. A
  chain of impact should occur through the levels
  as the skills and knowledge learned (level2) are
  applied on the job (level3) to produce business
  results (level4).

45
Outcomes Used in Evaluating Training Programs
Cognitive Outcomes
Skill-Based Outcomes
Affective Outcomes
Results
Return on Investment
46
Outcomes Used in Evaluating Training Programs
(continued)

• Cognitive Outcomes
• Determine the degree to which trainees are
  familiar with the principles, facts, techniques,
  procedures, or processes emphasized in the
  training program.
• Measure what knowledge trainees learned in the
  program.
• Skill-Based Outcomes
• Assess the level of technical or motor skills.
• Include acquisition or learning of skills and use
  of skills on the job.

47
Outcomes Used in Evaluating Training Programs
(continued)

• Affective Outcomes
• Include attitudes and motivation.
• Trainees perceptions of the program including
  the facilities, trainers, and content.
• Results
• Determine the training programs payoff for the
  company.

48
Outcomes Used in Evaluating Training Programs
(continued)

• Return on Investment (ROI)
• Comparing the trainings monetary benefits with
  the cost of the training.
• Direct costs
• Indirect costs
• Benefits

49
How do you know if your outcomes are good?

• Good training outcomes need to be
• Relevant
• Reliable
• Discriminate
• Practical

50
Importance of Training Cost Information

• To understand total expenditures for training,
  including direct and indirect costs.
• To compare costs of alternative training
  programs.
• To evaluate the proportion of money spent on
  training development, administration, and
  evaluation as well as to compare money spent on
  training for different groups of employees.
• To control costs.