AICC Management

1
AICCManagement ProcessesSubcommittee
ActivitiesOrlando, FLJune 2009Bruce Perrin
2
Management Processes Subcommittee

• Charter Provide recommendations and guidelines
  to the Computer-based Training community that
  identifies the attributes of "Good CBT" processes
  and product.
• Topics for this Meeting
• AICC-sponsored survey
• Development, fielding, and response
• What evaluation practices are we using?
• How will current technologies and approaches
  impact training?
• Findings where do we find a polarization of
  opinion on a technology or approach?
• Recommendations on the use of 3D models (e.g.,
  virtual reality, virtual environments, etc.) in
  training
• Initial taxonomy based on subject of 3D content
• Relevant research
• Discussion and a request

3
Survey Development, Fielding, Response

• Introduced (and edited) survey at AICC meeting in
  San Jose
• What are our typical evaluation practices?
• Are there CBT issues where recommendations might
  help?
• Refined survey over several AICC Executive
  Committee teleconferences
• Worked with QuestionMark to put survey online
• Announced on AICC News Blog
• Publicized on AICC Website
• Discussed at AICC Meetings in Hamburg, Germany
  Louisville, KY
• Hosted by QuestionMark from 5/18/2008 to
  11/18/2008
• Thirty-two responses representing approximately
  25 organizations

4
Findings - Fields Represented in Sample
Survey What are your major fields of
training/learning interest (Choose one or more)?
5
Formative Summative Evaluation Use
Survey On what percentage of your training
systems do you conduct any formative evaluation,
e.g., measurement of training methods/processes,
so that needed changes or modifications can be
made in the early stages of development? On what
percentage of your training systems do you
conduct any type of summative evaluation, e.g.,
measurement of final training system outcomes or
results?
"When the cook tastes the soup, thats formative
when the guests taste the soup, thats
summative." (Robert Stakes)
6
Types of Evaluation Criteria Used

• Survey On what percentage of your training
  systems do you use each of the following types of
  evaluation criteria?
• Reaction how much the trainee liked the program
  or thought it would benefit him/her on the job
• Learning - how much knowledge and skill changed
  in the training setting
• Behavior - how much behavior changed in the work
  place
• Results - how much organizational factors were
  affected

• Most training is evaluated
• Reaction measures still predominate
• Compared to national studies
• Use of reaction and behavior measures similar
• Use of learning measures slightly higher
• Use of results measures slightly lower

7
Opinions on Current Trends/Claims

• Survey sought opinions on 9 trends/claims in the
  Industry
• The current generation learns differently than
  older adults did when they were that age.
• Gaming technology for training is applicable
  across a wide range of training tasks.
• Nearly everyone can learn effectively from gaming
  technology.
• Providing training in a format that is consistent
  with an individuals learning style will
  significantly increase learning performance.
• Computerized methods to adjust training content
  according to performance (e.g., scores on
  embedded tests, actions taken in a simulation)
  will significantly increase learning performance.
• Three-dimensional environments (virtual reality,
  virtual environments) represent an important
  extension to current training technologies, i.e.,
  they are effective and applicable in a variety of
  training
• Effective training cannot be built from
  context-independent, re-usable (sharable)
  learning objects.
• The need for maintenance training will subside
  over time as self-testing equipment and
  job-aiding technology becomes better.
• The disciplined use of meta-data will end up
  saving the training community substantial costs
  in development compared to the cost of developing
  the meta-data initially given current technology.

8
What Are Our Concerns?

• Of most interest (in my opinion) are technologies
  that elicit polarized beliefs
• Almost as many think the statement is true
  (definitely or probably true) as think that it is
  false (definitely or probably false)
• Few people have no opinion (unsure do not know)
• Examples
• The need for maintenance training will subside
  over time as self-testing equipment and
  job-aiding technology becomes better
• Three-dimensional environments (virtual reality,
  virtual environments) represent an important
  extension to current training technologies,
  i.e., they are effective and applicable in a
  variety of training

9
Use of 3D Models in Training

• Why develop recommendations for the use of 3D
  models in training?
• Somewhat polarized opinions on utility (40
  unsure or do not believe 3D models have a
  widespread role in training)
• Considerable interest level at AICC meetings and
  in the training community in general
• Significant promise lower development and
  lifecycle cost greater throughput easier
  distribution
• Modest research base

10
Taxonomy and Studies Reviewed

• Taxonomy of the use of 3D models in training
• 3D content trains a task performed within a
  single visual scene or across independent scenes
• 3D content trains parts of a task in separate
  visual scenes and knowledge/skill must be
  integrated across them
• Desktop (includes all 2D cues to depth, e.g.,
  motion parallax, texture, interposition, linear
  perspective, etc.)
• Immersive (all above plus stereopsis)
• 3D environment in which training occurs (e.g.,
  Second Life)
• Difference between 1 2 is continuous

Study Type 1 Type 2 Type 3 Criteria
Buck, Perrin, et al. (1997-2003) - Virtual Maintenance Training X X Knowledge test and behavior demonstration
Waller (1999) Individual differences in learning spaces in a VE X Knowledge test and behavior demonstration

11
Virtual Maintenance Training Research

• Domain Maintenance training involving re-use of
  3D CAD models
• Tasks
• Remove install, in single visual scene
• Remove install, integrated across visual scenes
• Troubleshooting, in several independent visual
  scenes
• Interventions
• Low and high detail desktop
• Active vs. passive involvement
• Immersive (head-mounted displays) desktop
• Training on physical mockup provide control
  condition
• Method
• Over 200 participants (Boeing US Navy)
• Several independent replications of effects
• Criteria knowledge test, performance accuracy
• Measured experience with computers, 3-D games,
  hands-on activities
• Measured spatial visualization aptitude (ETS
  paper-folding test)

12
Virtual Maintenance Training Research (cont.)
Training Type Findings
Single or Independent Scenes Modest drop in overall learning performance compared to control Similar variability in performance among trainees
Scenes that must be integrated - Desktop Significant drop in learning performance compared to control High variability among trainees in the amount learned (variance often 5 times greater or more)
Scenes that must be integrated - Immersive Extreme drop in learning performance compared to control Extreme differences among trainees in the amount learned (variance often 10 times greater or more)
13
Virtual Maintenance Training Research (cont.)

• Background factors examined to explain
  variability in learning
• Prior experience with tools, repairs generally
  faster performance, but effect on VE and hardware
  training is the same
• Exposure to 3-D computer/video games no
  significant effect
• Immersive tendencies no significant effect
• Extended practice with3-D interface no
  significant effect
• Spatial visualization aptitude, ETS
  paper-folding test

14
Virtual Maintenance Training Research (cont.)

• Tested hypothesis that extreme variation
  following 3D model-based training results from
  lack of visual access
• Changed the location of the part being removed
• Repeated the study
• Findings consistent with other tasks trained in
  the visual field
• Interventions that expanded visual access also
  improved learning performance

Original Modified
15
Spatial Navigation Training Research

• Domain Training spatial knowledge in a virtual
  environment
• Tasks
• Pointing, mapping, or navigating the real-world
  or virtual environment
• Included both small (room size) and large (campus
  wide) settings
• In all cases, separate scenes must be integrated
  to form survey map of the environment (e.g.,
  all type 2 situations)
• Interventions
• Desktop VE
• Compared to training in the physical environment
• Method
• Series of studies on individual differences
• Experimental-control group, post-test only design
• Correlation (latent structural) designs

16
Spatial Navigation Training Research (cont.)
Study Description Findings
Maze training in VE (experimental) physical maze (control) Error in pointing to unseen locations more than 18 times greater after VE-based training
Factors correlating with spatial learning in a VE Strongest - Spatial ability (ETS paper folding was primary measure) Second - Practice time and maneuvering speed Factors not significantly correlated Verbal ability Computer use Gender Spatial accuracy of real world learning (measured as pointing, map making, and navigating Univ. of Washington campus)
17
Initial DRAFT Recommendations
Type of 3D-Based Training Recommendations
3D content trains a task performed within a single visual scene or across independent scenes Follow standard design, development, and evaluation procedures Variability in trained performance may be compared against a control
3D content trains parts of a task in separate visual scenes and knowledge/ skill must be integrated across them Variability in trained performance should be compared against a control (even an untrained group) Correlation between criterion and a measure of the visualization aptitude should be examined Immersive environments should be avoided unless validated Techniques that increase visual access (e.g., transparent a/c skins) should be considered
3D environment in which training may occur TBD
18

• Discussion
• And A Request
• Please forward any published research on use of
  3D models in training that have a learning or
  behavior measure
• Studies that show impacts on speed, cost,
  throughput, etc., without equivalent or better
  learning/performance are not of interest

19
(No Transcript)