Measuring the Effectiveness of Training

1
Measuring the Effectiveness of Training
ASQ Section 105 February 21, 2008 Presented
By Susan L. Reynolds CQE, CSSBB, CQA, CMQ/OE, CIA
2
What we will cover

• Why training measurement systems may need to be
  changed.
• Can it work? Says who?
• Overview of best practices with an example of one
  set of tools techniques.
• Next steps - where to go from this brief
  introduction.
• References/sources of the information being
  presented.

Disclaimer
Attempting to cram a weeklong course into 1520
minutes could possibly mean that we have to skip
a detail or two
3
Historically training often is

• Measured from the perspective of the participants
  (Happy Sheets)
• Not required to demonstrate an effective transfer
  of learning to the work setting the impact on
  key measures
• The sole responsibility of line-management or of
  the HR/training department
• Costs accounted for in ways that contribute to
  indifference from management (true costs are
  often unknown)
• Managed by staff who themselves believe that the
  business effects of training cannot be measured
  credibly

4
Is Credible Measurement Feasible? Selected
quotes from the references

• The organizational impact of training can be
  measured with credibility and reasonable
  allocation of resources (How to Measure Training
  Results)
• When organizations ask for my help in evaluating
  training, they usually want to know whether the
  training was efficient. This question can be
  quite simply answered by assessing the changes
  achieved during the training. (Evaluating
  Training Effectiveness)
• This Six Sigma based model will help
  organizations forever answer the question, What
  value am I getting from my investment in
  training? (Developing and Measuring Training the
  6 Sigma Way)

5
Productivity Range by Job Complexity
From Running Training Like a Business, Trolley
van Adelsberg, page 4
6
Generally Accepted Best Practices Common
themes in each of the resources investigated

• The overall training program must have goals
  aligned with the organizations business goals
• Individual training modules must have goals and
  objects aligned with those of the overall program
• Training goals must meet the stakeholders needs
• Baseline or control data is needed to create the
  most valuable measurements
• The difficulty of measuring increases with the
  increased value of the measure

7
Decide where youre going first
Would you tell me, please, which way I ought to
go from here? Alice speaks to Cheshire Cat
That depends a good deal on where you want to
get to,' said the Cat.
I don't much care where--' said Alice.
Then it doesn't matter which way you go,' said
the Cat.
8
Tools techniques from one reference source

• The next several slides give an overview of the
  methods covered in How to Measure Training
  Results, by Phillips Stone. Though there are
  common themes, each reference book listed at the
  end, presents a unique approach.
• Organizations have unique business needs and
  start from different points. This particular
  book and the specific techniques covered here,
  may not be the best choice for your organization.
  However they should give you an idea of where
  your own measurement system stands and some idea
  of where to start.

9
5 levels of evaluation
from How to Measure Training Results, Phillips
Stone
10
5 levels of evaluation - Level 1

• Participants' reactions to the training, and its
  design and delivery are measured.
• Most training programs are evaluated at Level 1,
  usually by means of generic questionnaires or
  surveys.
• The value of Level-1 evaluation may be improved
  by asking participants how they plan to apply
  what they have learned
• This level of evaluation is important as a
  measure of participant satisfaction, but it does
  not measure participants ability to implement
  facts, skills, etc on the job.

from How to Measure Training Results, Phillips
Stone
11
5 levels of evaluation - Level 2

• Generally is a skills or knowledge test done
  during or directly after the training.
• Focuses on what the participants learned during
  the training.
• Helpful in determining whether participants have
  absorbed new knowledge and skills and know how to
  use them as a result of the training.
• This level of evaluation is important as a
  measure of knowledge and skills in a classroom
  environment, but it does not measure participants
  long term ability to implement facts, skills, etc
  on the job.

from How to Measure Training Results, Phillips
Stone
12
5 levels of evaluation - Level 3

• A variety of follow-up methods are used to
  determine whether participants apply the training
  to their work settings.
• Frequency and effectiveness of their use of new
  skills are important measures at Level 3.
• Although this level evaluation is important in
  determining the application of the training, it
  still does not guarantee that there will be a
  positive impact on the organization.

from How to Measure Training Results, Phillips
Stone
13
5 levels of evaluation - Level 4

• Business Impact, measurement focuses on the
  actual business results achieved as a consequence
  of applying the knowledge and skills from the
  training.
• Typical Level-4 measures are output, quality,
  cost, time, and customer satisfaction. .
• However, although the training may produce a
  positive measurable business impact, there is
  still the question of whether the training may
  have cost too much, compared to what it achieved.
  

from How to Measure Training Results, Phillips
Stone
14
5 levels of evaluation - Level 5

• Return on Investment - this measurement compares
  the monetary value of the benefits resulting from
  the training with the actual costs of the
  training program.
• ROI usually is presented as a percentage or
  benefit-cost ratio.
• At this level the stakeholders can get an
  understanding of the value of what they are
  paying for.

from How to Measure Training Results, Phillips
Stone
15
5 levels of evaluation (slide repeated)
from How to Measure Training Results, Phillips
Stone
16
Where are you starting from?

• Define the who your training customers are
• Distinguish participants from end customers
  whos paying who benefits vs. whos being
  trained
• Evaluate what level you are currently using to
  measure training
• Dont try to jump from level 1 to level 5
  immediately.
• While implementing a higher level begin
  preparations for the next level (i.e. baseline
  measurements, and any other prerequisites)

17
The ROI Model Process from How to Measure
Training Results, Phillips Stone
18
Overview of the ROI Model and Process

• The ROI Process has been used in hundreds of
  business and government organizations to
  demonstrate the impact and return on investment
  of training programs, human resource programs,
  major change initiatives, and performance-improvem
  ent programs.
• The four features valued most by clients are
  simplicity, cost-effectiveness, flexibility, and
  robust feedback useful for informing senior
  management about performance on the job and
  impact on business measures.

from How to Measure Training Results, Phillips
Stone
19
Develop Objectives of Training ROI step 1

• This initial step develops an understanding of
  the scope of the program and the business
  measures that it should influence. If the program
  is an existing program being evaluated, the
  objectives and content of the program are
  reviewed to guide the development of evaluation
  strategies.
• If it is a new program, needs assessment data are
  used to develop objectives at levels 1 through 4.
  The purpose of the evaluation study is then
  determined.

from How to Measure Training Results, Phillips
Stone
20
Develop Evaluation Plans and Baseline Data - ROI
step 2

• The Data Collection Plan is developed and
  measurements (4 levels), methods of data
  collection, sources of data, and timing of
  collection are identified to collect baseline and
  follow-up data.
• The ROI Analysis Plan is developed and the
  methods of isolation, conversion of data to
  monetary values, cost categories, communication
  targets, and other steps are determined.
• The nature of the training intervention and the
  rollout schedule will dictate the timing of the
  data gathering. The purpose of the study and
  appropriate evaluation strategies are verified
  before beginning the process.

from How to Measure Training Results, Phillips
Stone
21
Collect Data During Training ROI step 3

• The training is implemented, and data is
  collected at level 1 and level 2.
• The evaluator may not always be involved in
  collecting data at these two levels, but should
  require evidence from others (especially at level
  2) that provides sufficient data to satisfy the
  needs of the study at the level in question.

from How to Measure Training Results, Phillips
Stone
22
Collect Follow-up Data After Training ROI step
4

• Applying the methods and timing from the Data
  Collection Plan described earlier, follow- up
  data is collected.
• Depending on the program selected for evaluation
  as described in the Data Collection Plan, data
  collection may utilize questionnaires,
  interviews, data from company records, or other
  methods as appropriate.
• The cost of the training (7) is tabulated per
  the guidelines on the ROI Analysis Plan and will
  be used later in the ROI calculation.

from How to Measure Training Results, Phillips
Stone
23
Isolate the Effects of the Training ROI step 5

• One or more strategies are used to isolate the
  effects of the training. Examples are use of a
  control group arrangement, trend line analysis,
  estimates by participants, estimates by managers,
  and estimates by in-house experts.
• If a control group arrangement is feasible,
  performance data will be collected on the trained
  group and on another group with similar
  characteristics that does not receive the
  training. The pre- and post-training performance
  of the two groups will be compared to determine
  the extent of improvement influenced by the
  training.
• At least one isolation strategy will be used to
  determine the extent of influence the training
  intervention has on key business measures

from How to Measure Training Results, Phillips
Stone
24
Convert Data to Monetary Values ROI step 6 - 9

• Certain business impact data influenced by the
  training will be converted to monetary values to
  allow comparison of training benefits to costs to
  determine the return on investment (calculate the
  ROI, 8).
• Fully loaded costs (7) must be captured in order
  to complete the calculation. If some data cannot
  be converted to a monetary value, that data will
  be reported either as business impact results
  (e.g., improvements in customer or employee
  satisfaction) or as intangible benefits when the
  business impact cannot be expressed as a hard
  value ( 9).

from How to Measure Training Results, Phillips
Stone
25
Generate an Impact Study ROI step 10

• At the conclusion of the study, two reports are
  usually developed for presentation.
• One report is brief and intended for presentation
  to executive management. The other report is more
  detailed and is suitable for other stakeholders.

from How to Measure Training Results, Phillips
Stone
26
Closing the Corrective Action Training Loop
Problems with corrective action responses that
give a human error cause indicate that
Training is the corrective action which will
correct the error, include

• The response may not get to a true root cause
  (topic for another session) the training is at
  best a containment action.
• Without solid Level 3 measurement methods in
  place, the only way for you learn whether the
  training worked or not is by watching for another
  instance of the error
• Lack of goals objectives for training that are
  tied back to business results means the
  training may be viewed as non value added and
  done on the cheap. It may be limited to specific
  individuals or work centers when the cause
  actually involves more of the work force.

27
References