Developing Appropriate Metrics for Process and Outcome Measures

1
Developing Appropriate Metrics for Process and
Outcome Measures

• Amy K. Holtzman, David P. Baker, and Robert F.
  Calderón
• American Institutes for Research
• Kimberly Smith-Jentsch, University of Central
  Florida
• Paul Radtke, NAVAIR Orlando TSD
• IMTA Annual Conference
• November 3-6, 2003
• Pensacola, FL

2
Overview

• Background
• Research on Performance Measures
• Purpose of Current Study
• Method
• Results
• Conclusions

3
Background

• Scenario-based training
• Definition Systematic process of linking all
  aspects of scenario design, development,
  implementation, and analysis (Oser et al., 1999).
• Training is based on an exercise or scenario.
• The military often uses these exercises to
  evaluate if individuals or teams have acquired
  skills for specific missions.
• Performance measures can be used to assess
  performance in a particular scenario.

4
Background

• Performance measures
• Vary by level of analysis
• Vary by types of measures (process or outcome)
• Vary by the purpose of training

5
Research on Performance Measures

• Much of performance measurement research has been
  conducted on improving instruments used to
  evaluate performance in the civilian performance
  appraisal area.
• Most of this research has focused on rating
  scales.
• Rating scales are numerical or descriptive
  judgments of how well a task was performed.
• Civilian team performance and training literature
  has addressed checklist and frequency.
• Checklist contains items with dichotomous
  answers.
• Frequency provides indication of the number of
  times an action occurs.

6
Research on Performance Measures

• Military instructors have primarily used the
  checklist format.
• There is no clear guidance in the literature that
  shows when to use which rating format to best
  measure specific processes and outcomes (e.g.,
  for measuring procedural taskwork, use the
  checklist format).

7
Purpose of Current Study

• This study sought to develop guidance for
  identifying and developing appropriate metrics
  for measuring human performance during
  scenario-based training.
• It was part of a larger study for the Navy
  designed to develop the business logic for an
  automated tool that military instructors can use
  to create performance measures for scenario-based
  training exercises.
• This study was qualitative in nature and involved
  conducting interviews with ten human performance
  measurement experts.

8
Outcome Measures

• Accuracy
• Precision with which a task is performed
• Timeliness
• Length of time in which actions are performed
• Productivity
• Rate at which actions are performed or tasks are
  accomplished within a given situation
• Efficiency
• Ratio of resources required to those expended to
  accomplish a given task

9
Outcome Measures

• Safety
• Degree to which a task is accomplished in a way
  that does not unduly jeopardize human and capital
  resources
• Effects
• Degree to which the desired effect was achieved

10
Process Measures

• Procedural Taskwork
• Requirements specific to a position that follow a
  step-by-step process
• Non-procedural Taskwork
• Requirements specific to a position that do not
  follow a step-by-step process
• Teamwork
• Processes individuals use to coordinate activities

11
Rating Formats

• Checklist
• Checklist contains items with dichotomous
  answers.
• Frequency
• Frequency provides indication of the number of
  times an action occurs.
• Distance and discrepancy
• Numerical indices of how actual performance on
  the task differs from optimum performance
• Rating scales
• Rating scales are numerical or descriptive
  judgments of how well a task was performed.

12
Method

• Participants were given definitions and examples
  of the outcome and process measures and
  definitions of the rating formats.
• They ranked their first, second, and third
  choices for measuring each outcome and process.
• If they felt other formats were necessary, they
  added them.
• They explained their rational for making their
  first choices.
• They also provided demographic information.

13
Method
14
Participants

• Ten experts in human performance measurement
• All had PhDs in Psychology (most in I/O
  Psychology).
• Participants collective experience included
  test development and assessment, performance
  model development, job analysis, and performance
  appraisal measure development.

15
Results First Choice
16
Results Second Choice
17
Results First Choice
18
Results Second Choice
19
Conclusions

• DD was a popular choice for many outcome
  measures.
• Frequency and rating scale formats were judged
  most suitable for measuring safety and effects.
• Checklist was judged most suitable for measuring
  procedural taskwork.
• Rating scales were popular choices for measuring
  non-procedural taskwork and teamwork.

20
Next Steps

• This study was a first step in linking rating
  formats to outcomes and processes.
• Further research is needed.
• A further study with measurement experts that
  takes the purpose of training and the level of
  analysis into account is planned.
• Purposes
• Diagnose performance deficiencies/ provide
  feedback
• Assess Fleet readiness

21
Next Steps

• Results will help us link rating formats to
  processes and measures.
• We will develop business rules for the automated
  tool which will assist military instructors in
  developing performance measures for
  scenario-based training exercises.

22
Contact Information

• Amy K. Holtzman
• American Institutes for Research
• 1000 Thomas Jefferson St., NW
• Washington, DC 20007-3835
• 202-298-2643
• aholtzman_at_air.org