The Man-Machine Integration Design

1
The Man-Machine Integration Design Analysis
System (MIDAS) Recent Improvements

• Sandra G. Hart
• Brian F. Gore
• Peter A. Jarvis
• NASA Ames Research Center
• Moffett Field, CA 94035
• Sandra.G.Hart_at_NASA.gov/650 604 6072
• 10/19/04

2
Outline

• Human Performance Modeling
• MIDAS Phase 1 Initial design
• Early applications
• MIDAS Phase 2 Move from Lisp to C
• Recent applications
• MIDAS Phase 3 PC Port/Integrate Apex

3
Human Performance Models Components
Psychological Models
Timeline
Sensory Models
Task Network
Anthropometric Models
Performance WL

Biodynamic Models
Performance Time
Model Architecture, Library, Tools
Team/Org Models
Performance SA
Vehicle Models
Performance Errors
Equipment Models
Visualization
Environment Models
FoV/Reach Envelope
Procedural Models
4
Human Performance Models Architectures
Anthropometric/Biodynamic Physical
characteristics of human body static dynamic
population characteristics limitations RAMSIS,
JACK
Psychological theories, mathematical models,
descriptive functions

• Task network Top-down, based on sequences of
  human/system tasks (derived from task analysis

MicroSaint WinCrew Crewcut IPME IMPRINT
Cognitive Bottom-up, combine theory-based
models of memory, decision making, perception,
attention, movement, etc
ACT-R MIDAS/ AirMIDAS D-OMAR
Soar APEX SAMPLE
Vision Computational representation of the way
the human visual system processes an image to
predict performance given image characteristics
ORACLE NASA Standard Visual Observer NASA Text Visibility Optimetrics Visual Perf Model Georgia Tech Vis Model
5
Human Performance Models can

• Generate hardware, software, training
  requirements for tasks that will involve human
  operators
• Depict operators performing tasks in prototype
  workspaces and/or in remote or risky
  environments
• Perform tradeoff analyses among alternative
  designs and candidate procedures, saving time and
  money
• Identify general human/system vulnerabilities to
  estimate overall system performance and
  reliability
• Provide dynamic, animated examples for training
  and developers
• Generate realistic schedules and procedures

6
Phase 1
7
Overview
A comprehensive suite of computational tools - -
3D rapid prototyping, models of perception,
cognition, response, real- and fast-time
simulation, performance analysis, visualization -
- for designing and analyzing human/machine
systems was developed primarily in Lisp on a
fleet of SGIs
Data Analysis
Run Time Visualization
8
Features

• Pioneered the development of an engineering
  design environment with integrated tools for
  rapid prototyping, visualization, simulation and
  analysis
• Advanced the capabilities and use of
  computational representations of human
  performance in design including a state of the
  art anthropometric model (Jack)
• Flexible enough to support a range of potential
  users and target applications
• But.
• Component models written in Lisp, Fortran, C, C
• Required a suite of SGI machines
• Modeled a single operator
• Time based rather than event based scheduler
  established optimal inter-leaving of task
  components
• No emergent behaviors

9
Richmond, CA Police 911 Dispatch

• Goal Upgrade the facilities and procedures used
  in the 911 dispatch facility
• Accomplished
• Modeled control console and dispatch activities
  in MIDAS
• Evaluated prototype graphical decision aid

10
US Army Air Warrior

• Goal Establish baseline performance measures for
  crews flying Longbow Apache with and without MOPP
  gear
• Accomplished
• Modeled copilot/gunner with Jack (95th male ltgt
  5th female)
• Rendered cockpit using CAD files from
  manufacturer
• Simulated performance of more than 400 activities
• Measured reach, FoV, workload, timelines

11
Short Haul/Civil Tiltrotor

• Goal Evaluate crew performance/workload issues
  for steep (9º), noise abatement approaches into a
  vertiport
• Accomplished
• Constructed MIDAS models of normal and aborted
  approaches
• Contrasted impact of manual vs automated
  nacelle control modes

12
NASA Shuttle Upgrade

• Goal Support development of an advanced orbiter
  cockpit with an improved display/control design
• Accomplished
• Created virtual rendition of current shuttle
  cockpit
• Conducted simulation of first 8 min of nominal
  ascent
• Provided quantitative measures of workload/SA,
  timing

13
Phase 2
14
Features

• Decreased model development from months to weeks
• Increased run-time efficiency from 50x RT to near
  RT
• Multiple operators
• Modeled external vision, audition, situation
  awareness
• Conditional behaviors emerging from interaction
  of top-down goals and environmentally driven
  contexts
• Option of non-proprietary head hands model
• But
• The interface still user un-friendly
• SGI platform
• Cognitive models no longer state of the art
• Performance moderating functions not integrated

15
Overview of Architecture
16
User Interface

• The interactive graphical user interface is used
  to create models, specify and run simulations,
  and view data. It is organized into a
  hierarchical series of screens or editors that
  are navigated with tabs
• Different views of the simulation are offered
  Structure, Geometry, Outline, Animation,
  Real-time/post-hoc data

17
Vehicle Models

• A modeled vehicle represents the combination of a
  guidance/ dynamics model and a visual
  representation
• The guidance/dynamics model moves the vehicle
  along a prescribed route. MIDAS provides two
• NoE helicopter model
• Simple point mass model (used to model arbitrary
  vehicles in a generic way)
• The visual representation is CAD geometry chosen
  from the MIDAS library or developed by the
  modeler.

18
Environment Model

• Tools are provided to model the environment
  outside the crew station (e.g., terrain, weather,
  etc)
• Terrain is modeled as a single object
• Features are simple objects that have no inherent
  behavior and do not move
• One weather condition may be applied to the
  environment by specifying lighting/visibility
  (these are used by the visual perception model)

19
Crew Station/Equipment Models

• The crew station is a collection of equipment
  with which operators interact
• Crew station models may be given a graphical
  representation for animation
• Multiple crew stations per vehicle and multiple
  operators per crew station possible

20
Anthropometric Models

• Anthropometric models provide an animated, 3D
  graphical representation of one or more modeled
  human operators for visualization
• Jack (developed at U Penn/distributed by UGS)
  full-body figure realistic movements
• Head and Hands model government-developed
  representation adequate for many purposes for
  users without a Jack license

21
Vision Models

• Visual attention modeled as single cone,
  varying from 3-15º based on task type.
• External vision
• Peripheral 160 degrees
• Foveal 2.5 degrees
• Perceivability f(visibility, size, distance,
  local contrast ratio)
• Perception level f(dwell time, perceivability)
• Levels of Perception
• Detection
• Recognition
• Identification
• Internal vision
• Symbolic (check read)
• Digital (exact value)
• Text (character string)

22
Auditory Model

• Only within crew station
• External sounds are represented only if channeled
  through equipment
• Two Stages of Processing
• Detection
• Comprehension
• Content
• Verbal strings
• Signals
• All or none processing (Interruptions disrupt
  entire message)

23
Symbolic Operator Models

• Significant advance over earlier version, which
  required specification of all activities at
  primitive task level
• High-level scripting language, Operator Procedure
  Language (OPL) serves as front-end to a reactive
  planning system (RAPS)
• User-supplied procedures are instructions for
  accomplishing tasks
• Manages knowledge and beliefs, integrates human
  actions with scenario events

24

Memory Model

• Simpler model than in MIDAS 1.0
• Distinction between long-term/short-term memory
  was lost
• Memories are represented as a database of
  assertions or beliefs that are symbolic
  expressions describing the property of objects
• Memory can be examined by powerful tools in a
  querying language built into OPL

25
Attention Model

• Based on Wickens Multiple Resource Model.
• Acts as a mediator that maintains an account of
  attention resources in six different channels
• Necessary attention resources must be available
  before primitive tasks are initiated
• Task onset may be delayed if insufficient
  resources

26
Output Behavior Models

• If required resources are available an activity
  that corresponds to a primitive procedure is
  created
• Physical actions and their effects on equipment
  or environmental objects are modeled, regulated
  by a motor control process
• 60 primitive tasks are available in a Procedure
  Library with pre-defined load values easy to
  add more

Visual Auditory Cognitive Spatial Cognitive Verbal Manual Vocal
Estimate Time 0 0 0 2.0 0 0
Visual monitor 5.4 0 6.8 0 0 0
Type (1 hand) 5.9 0 0 5.3 7.0 0
Say message 0 0 0 5.3 1.0 4.5
Move object 5.0 0 1.0 0 2.6 0
27
Simulation System

• Engine/executive (uses discrete-event, fixed-time
  increment approach for advancing the simulation)
• Data collection mechanisms for generating runtime
  data that is graphically displayed which the
  simulation runs and is saved for post-run
  analyses
• Event generation mechanism provides a way for
  timed events to occur on schedule or with
  stochastic variance
• Provisioning system allows users to change the
  simulation and re-run without re-loading/re-star
  ting

28
Workload Situation Awareness

• Workload calculations based on McCracken
  Aldrich (1988)
• Load levels for Visual, Auditory, Cognitive, and
  Psychomotor dimensions are defined for task
  primitives on a scale of 1-7
• Momentary load based on aggregation
• SA calculations based on
• Ratio of operators relevant knowledge/required
  knowledge
• Distinguishes actual SA from perceived SA
• Situational elements can be objects in the crew
  station or the environment that define a
  situation or are in the operators memory and
  are operationally relevant.
• WL and SA values offer a powerful way to
  simulate realistic errors

29
Validation Search Rescue Mission
30
Comparison of Models to Simulator Data
ACT-R/PM U of Illinois Rice University
D-OMAR BBN Technologies
Air MIDAS San Jose State University
IMPRINT/ACT-R MAAD Carnegie Mellon
A - SA U of Illinois
Goodness-of-fit of individual model outputs to
empirical data
Preliminary timeline, SA, attention, wkld,
analysis,task execution times error
vulnerabilities
MIDAS NASA-ARC/Army
31
Nominal Approach Landing Simulation

• PF scanning for TFX, runway
• PNF monitoring PFD, Nav
• PF/PNF monitoring radio
• Flaps 30º/set confirm
• PF requests before landing checklist
• PNF checks/responds hear down
• PF confirms visually/verbally
• PNF checks/responds flaps 30
• PF confirms visually/verbally
• PNF checks/responds speed brakes set
• PF confirms visually/verbally
• PNF declares checklist complete
• PF sets/declares DA at 650
• PNF visually confirms DA set
• Note passing FAF
• Confirms final descent initiated

32
Traffic Call During Approach

• Final approach checklist is complete
• ATC call with traffic advisory
• Both pilots scan for traffic I dont see it
• Neither pilot notices as the decision altitude is
  passed
• After the fact, the First Officer notices
  Were past FAF and not descending
• Crew must decide whether to continue with the
  approach or abort

33
Life Sciences Glove Box
Virtual Glovebox

• Challenges
• Astronauts must follow detailed instructions
  within strict time constraints failure to do so
  introduces risk of science mission failure
• Role of Computational Modeling
• Predict interactive influences of microgravity
  (posture, bracing, precise movements, placing,
  moving, stowing) to develop/evaluate procedures
• Watching an animated dry run enables efficient
  communication among scientists, implementers,
  astronauts more effective training

Life Sciences Glovebox Payload Development Unit
received at Ames from the National Aerospace
Development Agency of Japan (NASDA)
Onboard KC-135
MIDAS rendering
34
Life Sciences Glove Box Simulation

• Goal Predict astronauts performance of complex
  experiments designed to answer questions about
  living organisms adaptation to the space
  environment
• Objectives Evaluate feasibility of following
  proposed procedures within time/performance
  constraints ID factors that will increase risk
  of mission failure e.g., waiting too long to
  photograph slides interruptions task requires
  (unavailable) resource(s)
• The Task
• Turn on experimental equipment (monitor,
  microscope, camera)
• Measure cell density/viability for each of 6
  samples
• Invert sample vial
• Place aliquot of sample on slide
• Place drop of viability stain in sample
• Record time on sample record
• Place cover slip on slide
• Observe on microscope
• Take photographs within specific time window
• Dispose of trash, return vials to containers,
  turn equipment off

35
Cell Staining/Photographing Experiment
36
Phase 3
37
MIDAS v3.0 Features

• Runs on high-end PC
• Simple model of microgravity influence on
  performance
• Physics model of microgravity impact on objects
  available
• Simple within-task fatigue model implemented
• Fatigue state model (U Penn/Astronaut Scheduling
  Assistant) selected
• Notion of task duration - - how long a task
  should take as well as how long it did take
• Grasping, moving, manipulating objects in
  workspace
• Apex will become the heart of the Task Manager
  and enable multi-tasking, task prioritization,
  shedding, deferral, resumption
• Task primitive definitions include failure modes
  (time/quality) that enable the occurrence of
  emergent behaviors
• Mission success/performance measures computed
  vulnerability to error, slipped schedules
  performance degradation

38
MIDAS v3.0 Structure
Task Manager Plans Monitors Remembers Senses Actua
tes
Task Network
List of Tasks/Procedures
Commands
Results
Timeline
Mission Environment
Physical Simulation Perceives Attends Moves/Acts C
hanges
Workstation Geometry
Fit/Reach/Vis envelope
Dynamic models
Dynamic Animation
Task executions
Library Primitive tasks Human model
Task state
Operator state
Mission success
Cognitive simulation Behavior modifiers Situation
Awareness Error, Workload Timeliness
Operator Characteristics
Performance measures
39
Typical Outputs
40
Fresh
41
Tired
42
PC Version Early Simulation
43
Conclusion

• MIDAS 3.0 now operates on a PC platform and will
  soon incorporate significantly enhanced cognitive
  model (Apex)
• MIDAS 3.0 gives users the ability to model the
  functional and physical aspects of a variety of
  operators, systems, and environments.
• It brings these models together in an
  interactive, event-filled simulation for
  quantitative and visual analysis
• The interplay between top-down and bottom-up
  processes and a suite of performance modifying
  functions enables the emergence of un-forseen,
  un-scripted behaviors
• The government has done what it set out to do - -
  spur development of human performance modeling
  tools integrated into a design environment
• Our goal is to continue to add functionality with
  each new application