Effective Training: Strategies, Systems and Practices, 3rd Edition

1
Effective Training Strategies, Systems and
Practices, 3rd Edition

• P. Nick Blanchard and James W. Thacker
• Evaluation of Training
• Chapter Nine

2
Purpose of Training Evaluation

• Determine if accomplished objectives
• Identify strengths and weaknesses
• Cost-benefit analysis
• Who should participate and who benefited most
• Determine if program appropriate
• Database for decision making

3
Evaluation Phase
4
Kirkpatricks Levels of Criteria

• Reaction did trainees like the program
• Learning demonstration of learning at end of
  program
• Behavior actual transfer to job
• Results impact on bottom line including
  efficiency, productivity, cost, etc.

5
Data Collection Methods

• Interview
• Questionnaire
• Direct observation
• Tests and simulations
• Archival performance data
• Types of data individual, group, system wide

6
Potential Questions to Be Addressed in a Process
Analysis   (Before Training) Part 1 of 2
Were needs diagnosed correctly?    What data
sources were used?    Was a knowledge/skill
deficiency identified?    Were trainees
assessed to determine their prerequisite KSAs?
Were needs translated into training objectives?
   Were all objectives identified?    Were
the objectives written in a clear, appropriate
manner? Was an evaluation system designed to
measure objectives?
7
Potential Questions to Be Addressed in a Process
Analysis (Before Training) Part 2 of 2
Was the training program designed to meet all the
training objectives?    Was previous learning
that supports or inhibits training
identified?    Were individual differences
assessed/factored into training design?    Was
trainee motivation to learn assessed and
addressed?    Were processes built into
training to facilitate recall and transfer?
   Were steps included to call attention to key
learning events?   Were aids for symbolic
coding and cognitive organization
designed? Were opportunities included for
symbolic and behavioral practice?    Were steps
included to ensure transfer of learning to the
job? Are the training methods appropriate for
the learning objectives?
8
Potential Questions to Be Addressed in a Process
Analysis (During Training)
Was there a match between trainer, training
techniques, and training/learning
objectives?    Were lecture portions of the
training effective? Was involvement
encouraged/solicited?   Were questions used
effectively?   Did the trainer appropriately
conduct the various training methodologies
(case study, role play, etc.)? Were they
explained well?        Did the trainer use the
allotted time for activities?        Was
enough time allotted?        Did trainees
follow instructions Was there effective
debriefing following exercises?    Did the
trainer follow the training design and lesson
plans? Was enough time given for each of the
requirements? Was time allowed for questions?
9
Who Is Interested in the Process Data
Training Department   Trainer Yes, to
determine what does and does not work
well.  Other trainers To the extent that
process generalizes to other
training.   Training manager Only if training
is not successful, or if there is a problem
with a particular trainer. Customers
Trainees No Trainees supervisor No
Upper management No
10
Reaction Questionnaire for the Trainer Part 1
of 2
Please circle the number that reflects the degree
to which you agree or disagree with the following
statements. 1 Strongly disagree, 2 Disagree, 3
Neither agree nor disagree 4 Agree 5 Strongly
agree

• 1. The trainer did a good job of stating the
  objectives
• at the beginning of training
  .....1 2 3 4 5
• 2. The trainer made good use of visual aids
  (easel, white
• board) when making the presentations
  ....1 2 3 4 5
• 3. The trainer was good at keeping everyone
  interested
• in the topics .... 1 2
  3 4 5
• 4. The trainer encouraged questions and
  participation
• from trainees .1 2 3
  4 5

11
Reaction Questionnaire for the Trainer Part 2
of 2
12
Steps to Consider in Developing a Reaction
Questionnaire Part 1 of 2
13
Steps to Consider in Developing a Reaction
Questionnaire Part 2 of 2

• Allow space for Additional Comments in order to
  allow participants the
• opportunity to mention things you might not
  have considered.
• a. If right after training, have someone other
  than the instructor administer and
  pick up the information.
• b. If some time later, develop a mechanism for
  obtaining a high response rate (e.g.,
  encourage the supervisor to allow trainees to
  complete the questionnaire on company time).

14
Course Reaction Questionnaire

• Develop a questionnaire to measure student
  reactions to COBA courses
• About 10 questions
• What would you want to know to help you decide
  what courses and instructors you would want to
  take?

15
Procedures for Developing a Multiple Choice Test

1. Examine objectives to determine the content area
   you wish to test.
2. Write the questions in a clear manner. Shorter
   is better.
3. Choose alternatives to the correct response from
   typical errors made during training. Make
   alternatives realistic.
4. Avoid making the correct response always longer
   than incorrect responses.
5. Four alternatives are usually enough.
6. Pretest items. Ask for feedback on clarity. Note
   questions that most get wrong.

16
Example of an Attitudinal Measure
Attitudes Toward Empowerment Please indicate the
degree to which you agree or disagree with the
following statements. 1 Strongly disagree 2
Disagree 3 Neither agree nor disagree 4
Agree 5 Strongly agree
1. Empowering employees is just another way to
get more work done with fewer
people. (reverse scored)..1 2 3 4 5
2. Empowering of employees allows everyone to
contribute their ideas to the betterment
of the company.1 2 3 4 5 3. The
empowerment program has improved relationship
with my supervisor1 2
3 4 5 4. Empowerment has brought more
meaning to my life at this
company..1 2 3 4 5
5. Empowerment interventions should be
introduced in other plants in this
company..1 2 3 4 5 6.
The empowerment process has been a positive
influence in labor-management
relations.1 2 3 4 5
17
Guidelines for Writing Effective Questionnaires
Part 1 of 4
1.  Write simply and clearly, and make the
meaning obvious. Bad  To what extent do
supervisors provide information regarding the
quality of performance of people at your level?
Good How often does your boss give you feedback
on your job?
2.  Ask one question at a time. Bad  Both
the organizations goals and my role within the
organization are clear.
Good The organizations goals are clear. My
role within the organization is clear.
18
Guidelines for Writing Effective Questionnaires
Part 2 of 4
3.  Provide discrete response options. Bad  Duri
ng the past three months how often did you
receive feedback on your work? 1
2 3 4
5 rarely
occasionally
frequently
Good  During the past three months how often
did you receive feedback on your work? 1
2 3 4
5 not once 13
times about once 2-4 times
once a a month a week
a week day or more
19
Guidelines for Writing Effective Questionnaires
Part 3 of 4
4.  Limit the number of response
options. Bad  What percent of the time are you
sure of what your compensation will be? 1
2 3 4 5
6 7 8 9
10 010 1120 2130 3140 4150
5160 6170 7180 8190 91100
Good  What percent of the time are you sure of
what your compensation will be? 1
2 3 4 5 020 2140
4160 6180 81100
20
Guidelines for Writing Effective Questionnaires
Part 4 of 4
5.  Match the response mode to the
question. Bad  To what extent are you satisfied
with your job? 1 2 3 4 5 strongly
disagree agree
strongly disagree agree
Good  To what extent are you satisfied with
your job? 1 2 3 4 5 not at all a
little bit some quite a lot
very much
21
Example of 360-Degree Feedback Results
Skills being Assessed Very Low Low Average High Very High
Listening Skills S P, B E
Managing Conflict S P, B E
Organizing S, P B, E
Written Communication S, P B E
S Subordinates P Peers B Supervisor (Boss) E Employee being assessed S Subordinates P Peers B Supervisor (Boss) E Employee being assessed S Subordinates P Peers B Supervisor (Boss) E Employee being assessed S Subordinates P Peers B Supervisor (Boss) E Employee being assessed S Subordinates P Peers B Supervisor (Boss) E Employee being assessed S Subordinates P Peers B Supervisor (Boss) E Employee being assessed
22
Scripted Situation Item for Evaluation of a
School Superintendent
The following is a scenario about a school
superintendent. Read the scenario and place an X
next to the behavior you believe your
superintendent would follow. The administrator
receives a letter from a parent objecting to the
content of the science section on reproduction.
The parent strongly objects to his daughter
having exposure to such materials and demands
something be done. The effective administrator
would most likely (check one)
____  Ask the teacher to provide handouts,
materials, and curriculum content for
review. ____  Check the science curriculum for
the board-approved approach to reproduction,
and compare board guidelines to course
content. ____  Ask the head of the science
department for his or her opinion about the
teachers lesson plan. ____  Check to see if the
parent has made similar complaints in the past.
23
Types of Outcomes and Examples of Factors
Affecting those Outcomes
24
Cost Savings for Grievance Reduction Training
Pretraining Post training
Management time (for those going to 3rd step) 10 hours per grievance 10 hrs. X 63 grievances 630 hrs. 10 hrs. X 8 grievances 80 hrs.
Union Reps time (paid by management) 7.5 hrs per grievance 7.5 X 63 grievances 472 ½ hrs 7.5 X 8 grievances 60 hrs.
Total Cost Management time Union reps time Total 630 hrs X 50 per hr. 31,500.00 472 ½ hrs X 25 per hr. 11,812.50 43,312.50 80 hrs. X 50 per hr. 4,000.00 60 hrs. X 25 per hr. 1,500.00 5,500.00
Reduction in cost of grievances going to the third step 43,312.50 5,500.00 37,812.50 Cost of training 32,310.00 Cost saving for the first year 5,502.50 Reduction in cost of grievances going to the third step 43,312.50 5,500.00 37,812.50 Cost of training 32,310.00 Cost saving for the first year 5,502.50 Reduction in cost of grievances going to the third step 43,312.50 5,500.00 37,812.50 Cost of training 32,310.00 Cost saving for the first year 5,502.50

25
Calculation of the Utility of the Grievance
Training Part 1 of 2
Formula U (N)(T)(DT)(SDY) - C N
30 T 1 year (This is probably an overly
conservative estimate) DT .2 DT Xt Xu
SD (r yy) Xt average
job performance of the trained supervisors Xu
average job performance of the untrained
supervisors SD standard deviation of job
performance for the untrained supervisors r yy
reliability of job performance measure
26
Who Is Interested in the Outcome Data
Outcome Data Outcome Data Outcome Data Outcome Data
Reaction Learning Behavior Results
Training Department
Trainer Yes Yes Yes No
Other Trainers Perhaps Perhaps Perhaps No
Training Manager Yes Yes Yes Yes
Customers
Trainees Yes Yes Yes Perhaps
Trainees Supervisor Not really Only if no transfer Yes Yes
Upper Management No No Perhaps Yes
27
Reaction to Training Part 1 of 2
Answer the following questions about the training
in Active Listening skills using the scale
below 1 Strongly disagree 2 Disagree 3
Neither agree nor disagree 4 Agree
5Strongly agree    1. The training met the
stated objectives 1 2 3 4 5   2. The information
provided was enough so I understood the concepts
being taught 1 2 3 4 5   3. The practice
sessions provided were sufficient 1 2 3 4 5 to
give me an idea of how to perform the skill   4.
The feedback provided was useful in helping
1 2 3 4 5 me understand how to improve. 
28
Reaction to Training Part 2 of 2
5. The training session kept my interest
throughout 1 2 3 4 5 6. The pace of
the Active Listening session was 1. Way too
fast 2.   A bit fast 3.   Just right 4.   A
bit slow 5. Way too slow   7. What did you
like best about this part of the training    8.
What would you have changed   Additional
Comments    Note A similar scale would be used
for each of the other components of training that
were taught.
29
Calculation of the Utility of the Grievance
Training Part 2 of 2
DT is a measure of the improvement (in standard
deviation units) in performance that trained
supervisors will exhibit. SDY 14,000 This
is based on .40 X 35,000 14,000 The
above assumes average salary of 35,000 dollars.
The .40 comes from the 40 rule, which is a
calculation based on 40 of the average salary of
trainees. So based on the above information,
the utility of the training based on this formula
is 30 X 1 X .2 X 14,000 - 32,020 51,980
30
Paper Pencil Test for Evaluation of Learning
Evaluation of learning   There is no specific
time limit on this test, but most should be able
to finish in about one hour. Answers to the
questions should be written in the booklet
provided.   Please read each question carefully
as some of the questions have more than one part
to them. 1. List four types of active listening
and provide an example for each. 2. List the
steps in the Conflict resolution model. After
each step, provide a relevant example of a
phrase that would represent that step. 3. And so
forth..  
31
Training Evaluation Documents for Active
Listening Part 1 of 8
Scenarios for evaluating learning of Active
Listening Skills Part 1 Initiators Role   Role
Scenarios to Test Trainees (The initiator to be
played by the same actor for all trainees)   Test
Scenario 1 You have just been asked by your
supervisor (the trainee) to serve on the same
committee again. You are angry that they always
ask you. You Start. Say Angrily "OH NO,
YOU DON'T. I'VE BEEN ON THAT COMMITTEE THREE
YEARS IN A ROW AND IT TAKES UP TOO MUCH
TIME".  
32
Training Evaluation Documents for Active
Listening Part 2 of 8
Part 1 Initiators Role   Test Scenario 2 Your
supervisor has just talked to you about why you
dont follow procedures. You say
angrily "WHY ARE YOU PICKING ON ME ALL THE
TIME. I'M NOT THE ONLY ONE WHO DOESN'T
FOLLOW THESE STUPID PROCEDURES".    You have
just been asked by a your supervisor second time
today whether you will be attending the weekly
meeting. You say Angrily "I ALREADY TOLD YOU,
I CAN'T ATTEND THE WEEKLY MEETING BECAUSE I
HAVE TO COMPLETE THE STAFF REPORTS FOR And so
forth (for a total of 5) TOMORROW".  
33
Training Evaluation Documents for Active
Listening Part 3 of 8
Scenarios for evaluating learning of Active
Listening Skills Part 2 Trainees
instructions   This test will require you to
respond to 5 different short scenarios where you
are a supervisor, and you say something to a
subordinate that elicits an angry response. You
will be expected to respond using the skills of
active listening. Below is a description of each
of the scenarios and what you have initially said
to the subordinate. When you are ready for each
of the scenarios to begin nod your head to the
initiator. At that time the initiator will say
something. You need to respond to the comment,
and when complete, explain to the evaluator the
rationale for your response.  
34
Training Evaluation Documents for Active
Listening Part 4 of 8
Scenario 1 You have asked a subordinate to
continue working on a particular committee for
another year. Listen then respond using active
listening. Nod your head when ready.   Scenario
2 You have just talked to a subordinate regarding
the importance of following procedures. Listen
then respond using active listening. Nod your
head when ready.   Scenario 3 Today is the day
of your weekly meeting. You asked your
subordinate if he/she would be attending the
meeting, they said no. It is now time for the
meeting and you call once more to check if they
can make the meeting. Listen then respond using
active listening. Nod your head when ready.
35
Training Evaluation Documents for Active
Listening Part 5 of 8
Part 3 General guidelines for evaluating active
listening scenarios Trainee fails the
scenario if his/her response is focused on the
issue instead of reflecting back what the
initiator says.   Passing responses reflect back
what the person is saying without being a mimic
of what was said. Examples of  acceptable and
poor responses are given in the following slides.
36
Training Evaluation Documents for Active
Listening Part 6 of 8
Scenario 1 The supervisor (trainee being tested)
asked the subordinate to continue working on a
particular committee for another year. Listen to
response and grade according to guidelines
  Unacceptable response I am willing to talk
about reducing the work you have to do if you
will be on it Acceptable response "YOU
DON'T WANT TO BE ON THAT COMMITTEE AGAIN
BECAUSE IT INTERFERES WITH YOUR WORK AND YOU
HAVE DONE YOUR SHARE."
37
Training Evaluation Documents for Active
Listening Part 7 of 8
Scenario 2 The supervisor (trainee being tested)
has just talked to a subordinate regarding the
importance of following procedures. Listen to
response and grade according to guidelines.
Unacceptable response You are not the only
one I have talked to about this Acceptable
response "YOU BELIEVE THAT YOU'RE THE ONLY ONE
THAT I AM SINGLING OUT FOR NOT FOLLOWING
PROCEDURES".  
38
Training Evaluation Documents for Active
Listening Part 8 of 8
Scenario 3 The supervisor (trainee being tested)
called first thing in the AM, and asks the
subordinate if he/she would be attending the
weekly meeting and the subordinate said no they
were busy. The supervisor has just called again
at meeting time to check if they could make the
meeting. Listen and grade. Unacceptable
response The meeting will only be an
hour. Acceptable response "YOU'RE NOT
ABLE TO ATTEND THE MEETING BECAUSE YOU ARE
FINISHING STAFF REPORTS THAT ARE DUE
TOMORROW".
39
Research Design

• Internal Validity
• Did a change occur?
• Was the change a result of the training?
• External Validity
• Will the change occur in other situations with
  different trainees?

40
Threats to Internal Validity

• History events occurring during training
• Maturation natural improvements with
  development
• Testing effects on pre-tests on changes
• Instrumentation different measures at different
  point in time

41
Threats to Internal Validity (continued)

• Statistical regression select trainees measured
  at extremes in abilities/KSAs and regress to mean
• Reactive effects of research situation
  motivation (Hawthorne effect)
• Multiple treatment effects previous training

42
Experimental Designs

• Control group random assignment to treatment
  and control groups so trainees have similar
  characteristics
• Two-group posttest only (used most often)
• Two-group pretest/posttest
• Four group design control for effects of
  pretest and prior knowledge

43
Non-experimental Designs

• Case study intensive, descriptive study with
  after only measures
• One group pretest/posttest design
• One group posttest only design

44
Quasi-experimental Designs

• Nonequivalent control group design analyze for
  equivalence or use multiple regression to control
  for demographic factors
• Time series establish base line, then training,
  then series of measures to determine if change
  has occurred

45
Ethical Issues Concerning Evaluation Research

• Confidentiality
• Informed consent
• Withholding training can provide later
• Use of deception
• Pressure to produce positive results

46
Assessing the Impact of HRD Programs

• Cost-benefit monetary costs in relation to
  nonmonetary benefits
• Cost-effectiveness monetary costs in relation
  to monetary benefits
• Return on investment results/costs