Effective Training: Strategies, Systems and Practices, 2nd Edition

1
Effective Training Strategies, Systems and
Practices, 2nd Edition

• P. Nick Blanchard and James W. Thacker
• Chapter Eight

2
Evaluation Phase
3
Potential Questions to Be Addressed in a Process
Analysis   (Before Training) Part 1 of 2
Were needs diagnosed correctly?    What data
sources were used?    Was a knowledge/skill
deficiency identified?    Were trainees
assessed to determine their prerequisite KSAs?
Were needs translated into training objectives?
   Were all objectives identified?    Were
the objectives written in a clear, appropriate
manner? Was an evaluation system designed to
measure objectives?
4
Potential Questions to Be Addressed in a Process
Analysis (Before Training) Part 2 of 2
Was the training program designed to meet all the
training objectives?    Was previous learning
that supports or inhibits training
identified?    Were individual differences
assessed/factored into training design?    Was
trainee motivation to learn assessed and
addressed?    Were processes built into
training to facilitate recall and transfer?
   Were steps included to call attention to key
learning events?   Were aids for symbolic
coding and cognitive organization
designed? Were opportunities included for
symbolic and behavioral practice?    Were steps
included to ensure transfer of learning to the
job? Are the training methods appropriate for
the learning objectives?
5
Potential Questions to Be Addressed in a Process
Analysis (During Training)
Was there a match between trainer, training
techniques, and training/learning
objectives?    Were lecture portions of the
training effective? Was involvement
encouraged/solicited?   Were questions used
effectively?   Did the trainer appropriately
conduct the various training methodologies
(case study, role play, etc.)? Were they
explained well?        Did the trainer use the
allotted time for activities?        Was
enough time allotted?        Did trainees
follow instructions Was there effective
debriefing following exercises?    Did the
trainer follow the training design and lesson
plans? Was enough time given for each of the
requirements? Was time allowed for questions?
6
Who Is Interested in the Process Data
Training Department   Trainer Yes, to
determine what does and does not work
well.  Other trainers To the extent that
process generalizes to other
training.   Training manager Only if training
is not successful, or if there is a problem
with a particular trainer. Customers
Trainees No Trainees supervisor No
Upper management No
7
Reaction Questionnaire for the Trainer Part 1
of 2
Please circle the number that reflects the degree
to which you agree or disagree with the following
statements. 1 Strongly disagree, 2 Disagree, 3
Neither agree nor disagree 4 Agree 5 Strongly
agree

• 1. The trainer did a good job of stating the
  objectives
• at the beginning of training
  .....1 2 3 4 5
• 2. The trainer made good use of visual aids
  (easel, white
• board) when making the presentations
  ....1 2 3 4 5
• 3. The trainer was good at keeping everyone
  interested
• in the topics .... 1 2
  3 4 5
• 4. The trainer encouraged questions and
  participation
• from trainees .1 2 3
  4 5

8
Reaction Questionnaire for the Trainer Part 2
of 2
9
Steps to Consider in Developing a Reaction
Questionnaire Part 1 of 2
10
Steps to Consider in Developing a Reaction
Questionnaire Part 2 of 2

• 6. Allow space for Additional Comments in order
  to allow participants the
• opportunity to mention things you might
  not have considered.
• a. If right after training, have someone other
  than the instructor administer and
  pick up the information.
• b. If some time later, develop a mechanism for
  obtaining a high response rate (e.g.,
  encourage the supervisor to allow trainees to
  complete the questionnaire on company time).

11
Procedures for Developing a Multiple Choice Test

• Examine objectives to determine the content area
  you wish to test.
• Write the questions in a clear manner. Shorter
  is better.
• Choose alternatives to the correct response from
  typical errors made during training. Make
  alternatives realistic.
• Avoid making the correct response always longer
  than incorrect responses.
• Four alternatives are usually enough.
• Pretest items. Ask for feedback on clarity. Note
  questions that most get wrong.

12
Example of an Attitudinal Measure
Attitudes Toward Empowerment Please indicate the
degree to which you agree or disagree with the
following statements. 1 Strongly disagree 2
Disagree 3 Neither agree nor disagree 4
Agree 5 Strongly agree
1. Empowering employees is just another way to
get more work done with fewer
people. (reverse scored)..1 2 3 4 5
2. Empowering of employees allows everyone to
contribute their ideas to the betterment
of the company.1 2 3 4 5 3. The
empowerment program has improved relationship
with my supervisor1 2
3 4 5 4. Empowerment has brought more
meaning to my life at this
company..1 2 3 4 5
5. Empowerment interventions should be
introduced in other plants in this
company..1 2 3 4 5 6.
The empowerment process has been a positive
influence in labor-management
relations.1 2 3 4 5
13
Guidelines for Writing Effective Questionnaires
Part 1 of 4
1.  Write simply and clearly, and make the
meaning obvious. Bad  To what extent do
supervisors provide information regarding the
quality of performance of people at your level?
Good How often does your boss give you feedback
on your job?
2.  Ask one question at a time. Bad  Both
the organizations goals and my role within the
organization are clear.
Good The organizations goals are clear. My
role within the organization is clear.
14
Guidelines for Writing Effective Questionnaires
Part 2 of 4
3.  Provide discrete response options. Bad  Duri
ng the past three months how often did you
receive feedback on your work? 1
2 3 4
5 rarely
occasionally
frequently
Good  During the past three months how often
did you receive feedback on your work? 1
2 3 4
5 not once 13
times about once 2-4 times
once a a month a week
a week day or more
15
Guidelines for Writing Effective Questionnaires
Part 3 of 4
4.  Limit the number of response
options. Bad  What percent of the time are you
sure of what your compensation will be? 1
2 3 4 5
6 7 8 9
10 010 1120 2130 3140 4150
5160 6170 7180 8190 91100
Good  What percent of the time are you sure of
what your compensation will be? 1
2 3 4 5 020 2140
4160 6180 81100
16
Guidelines for Writing Effective Questionnaires
Part 4 of 4
5.  Match the response mode to the
question. Bad  To what extent are you satisfied
with your job? 1 2 3 4 5 strongly
disagree agree
strongly disagree agree
Good  To what extent are you satisfied with
your job? 1 2 3 4 5 not at all a
little bit some quite a lot
very much
17
Example of 360-Degree Feedback Results
18
Scripted Situation Item for Evaluation of a
School Superintendent
The following is a scenario about a school
superintendent. Read the scenario and place an X
next to the behavior you believe your
superintendent would follow. The administrator
receives a letter from a parent objecting to the
content of the science section on reproduction.
The parent strongly objects to his daughter
having exposure to such materials and demands
something be done. The effective administrator
would most likely (check one)
____  Ask the teacher to provide handouts,
materials, and curriculum content for
review. ____  Check the science curriculum for
the board-approved approach to reproduction,
and compare board guidelines to course
content. ____  Ask the head of the science
department for his or her opinion about the
teachers lesson plan. ____  Check to see if the
parent has made similar complaints in the past.
19
Types of Outcomes and Examples of Factors
Affecting those Outcomes
20
Cost Savings for Grievance Reduction Training
21
Calculation of the Utility of the Grievance
Training Part 1 of 2
Formula U (N)(T)(DT)(SDY) - C N
30 T 1 year (This is probably an overly
conservative estimate) DT .2 DT Xt Xu
SD (r yy) Xt average
job performance of the trained supervisors Xu
average job performance of the untrained
supervisors SD standard deviation of job
performance for the untrained supervisors r yy
reliability of job performance measure
22
Calculation of the Utility of the Grievance
Training Part 2 of 2
DT is a measure of the improvement (in standard
deviation units) in performance that trained
supervisors will exhibit. SDY 14,000 This
is based on .40 X 35,000 14,000 The
above assumes average salary of 35,000 dollars.
The .40 comes from the 40 rule, which is a
calculation based on 40 of the average salary of
trainees. So based on the above information,
the utility of the training based on this formula
is 30 X 1 X .2 X 14,000 - 32,020 51,980
23
Who Is Interested in the Outcome Data
24
Reaction to Training Part 1 of 2
Answer the following questions about the training
in Active Listening skills using the scale
below 1 Strongly disagree 2 Disagree 3
Neither agree nor disagree 4 Agree
5Strongly agree    1. The training met the
stated objectives 1 2 3 4 5   2. The information
provided was enough so I understood the concepts
being taught 1 2 3 4 5   3. The practice
sessions provided were sufficient 1 2 3 4 5 to
give me an idea of how to perform the skill   4.
The feedback provided was useful in helping
1 2 3 4 5 me understand how to improve. 
25
Reaction to Training Part 2 of 2
5. The training session kept my interest
throughout 1 2 3 4 5 6. The pace of
the Active Listening session was 1. Way too
fast 2.   A bit fast 3.   Just right 4.   A
bit slow 5. Way too slow   7. What did you
like best about this part of the training    8.
What would you have changed   Additional
Comments    Note A similar scale would be used
for each of the other components of training that
were taught.
26
Paper Pencil Test for Evaluation of Learning
Evaluation of learning   There is no specific
time limit on this test, but most should be able
to finish in about one hour. Answers to the
questions should be written in the booklet
provided.   Please read each question carefully
as some of the questions have more than one part
to them. 1. List four types of active listening
and provide an example for each. 2. List the
steps in the Conflict resolution model. After
each step, provide a relevant example of a
phrase that would represent that step. 3. And so
forth..  
27
Training Evaluation Documents for Active
Listening Part 1 of 8
Scenarios for evaluating learning of Active
Listening Skills Part 1 Initiators Role   Role
Scenarios to Test Trainees (The initiator to be
played by the same actor for all trainees)   Test
Scenario 1 You have just been asked by your
supervisor (the trainee) to serve on the same
committee again. You are angry that they always
ask you. You Start. Say Angrily "OH NO,
YOU DON'T. I'VE BEEN ON THAT COMMITTEE THREE
YEARS IN A ROW AND IT TAKES UP TOO MUCH
TIME".  
28
Training Evaluation Documents for Active
Listening Part 2 of 8
Part 1 Initiators Role   Test Scenario 2 Your
supervisor has just talked to you about why you
dont follow procedures. You say
angrily "WHY ARE YOU PICKING ON ME ALL THE
TIME. I'M NOT THE ONLY ONE WHO DOESN'T
FOLLOW THESE STUPID PROCEDURES".    You have
just been asked by a your supervisor second time
today whether you will be attending the weekly
meeting. You say Angrily "I ALREADY TOLD YOU,
I CAN'T ATTEND THE WEEKLY MEETING BECAUSE I
HAVE TO COMPLETE THE STAFF REPORTS FOR And so
forth (for a total of 5) TOMORROW".  
29
Training Evaluation Documents for Active
Listening Part 3 of 8
Scenarios for evaluating learning of Active
Listening Skills Part 2 Trainees
instructions   This test will require you to
respond to 5 different short scenarios where you
are a supervisor, and you say something to a
subordinate that elicits an angry response. You
will be expected to respond using the skills of
active listening. Below is a description of each
of the scenarios and what you have initially said
to the subordinate. When you are ready for each
of the scenarios to begin nod your head to the
initiator. At that time the initiator will say
something. You need to respond to the comment,
and when complete, explain to the evaluator the
rationale for your response.  
30
Training Evaluation Documents for Active
Listening Part 4 of 8
Scenario 1 You have asked a subordinate to
continue working on a particular committee for
another year. Listen then respond using active
listening. Nod your head when ready.   Scenario
2 You have just talked to a subordinate regarding
the importance of following procedures. Listen
then respond using active listening. Nod your
head when ready.   Scenario 3 Today is the day
of your weekly meeting. You asked your
subordinate if he/she would be attending the
meeting, they said no. It is now time for the
meeting and you call once more to check if they
can make the meeting. Listen then respond using
active listening. Nod your head when ready.
31
Training Evaluation Documents for Active
Listening Part 5 of 8
Part 3 General guidelines for evaluating active
listening scenarios Trainee fails the
scenario if his/her response is focused on the
issue instead of reflecting back what the
initiator says.   Passing responses reflect back
what the person is saying without being a mimic
of what was said. Examples of  acceptable and
poor responses are given in the following slides.
32
Training Evaluation Documents for Active
Listening Part 6 of 8
Scenario 1 The supervisor (trainee being tested)
asked the subordinate to continue working on a
particular committee for another year. Listen to
response and grade according to guidelines
  Unacceptable response I am willing to talk
about reducing the work you have to do if you
will be on it Acceptable response "YOU
DON'T WANT TO BE ON THAT COMMITTEE AGAIN
BECAUSE IT INTERFERES WITH YOUR WORK AND YOU
HAVE DONE YOUR SHARE."
33
Training Evaluation Documents for Active
Listening Part 7 of 8
Scenario 2 The supervisor (trainee being tested)
has just talked to a subordinate regarding the
importance of following procedures. Listen to
response and grade according to guidelines.
Unacceptable response You are not the only
one I have talked to about this Acceptable
response "YOU BELIEVE THAT YOU'RE THE ONLY ONE
THAT I AM SINGLING OUT FOR NOT FOLLOWING
PROCEDURES".  
34
Training Evaluation Documents for Active
Listening Part 8 of 8
Scenario 3 The supervisor (trainee being tested)
called first thing in the AM, and asks the
subordinate if he/she would be attending the
weekly meeting and the subordinate said no they
were busy. The supervisor has just called again
at meeting time to check if they could make the
meeting. Listen and grade. Unacceptable
response The meeting will only be an
hour. Acceptable response "YOU'RE NOT
ABLE TO ATTEND THE MEETING BECAUSE YOU ARE
FINISHING STAFF REPORTS THAT ARE DUE
TOMORROW".
35
Sources of Invalidity Part 1 of 2
36
Sources of Invalidity Part 2 of 2
Note In the tables, a minus indicates a definite
weakness, a plus indicates that the factor is
controlled, a question mark indicates a possible
source of concern, and a blank indicates that the
factor is not relevant. It is with extreme
reluctance that these summary tables are
presented because they are apt to be too
helpful and to be depended upon in place of the
more complex analysis. No or - indicator should
be respected unless the reader comprehends why it
is placed there. In particular, it is against the
spirit of this presentation to create fears of,
or confidence in, specific designs without an
clear understanding of its strengths and
weaknesses for a particular purpose.