Title: Approximating Sensor Footprints
1Fall 2005 Simulation Interoperability
Workshop 05F-SIW-039 An Application and
Evaluation of a Methodology for Testing
Operational Level Training Simulations MAJ Karl
D. Neal Virginia Modeling, Analysis, and
Simulation Center Old Dominion University Wesley
N. Colley, Ph.D. and Mikel D. Petty, Ph.D.
Center for Modeling, Simulation, and
Analysis University of Alabama in Huntsville
2Presentation Outline
- Introduction
- Operational-level training simulation
- The VVA problem
- Methodology
- The Missions/Means Paradigm
- Evaluation
- Application
- Terminal Fury 05
- Controlled Testing
- Conclusions
3Operational-level Training Simulations
- Operational Level of War
- Operational-level training simulations
- Simulations used in constructive components of
training exercises for operational level
commanders and their staffs in conducting warfare
at the operational level.
The operational level links the tactical
employment of forces to strategic objectives. The
focus at this level is on operational art the
use of military forces to achieve strategic goals
through the design, organization, integration,
and conduct of strategies, campaigns, major
operations, and battles (DoD Joint Pub 3.0,
2003).
4Verification, Validation, and Accreditation
- Verification
- Determination whether simulation meets design
goals - Validation
- Determination whether simulation reproduces
reality at desired fidelity - Accreditation
- Official certification that a simulation may be
used for a specific purpose
5VVA Problem 1What does does it work mean?
- Stimulates correct training goals?
- Produces valid entity interactions?
- Is sufficiently usable for the training staff?
- Even the concept of correct is tricky.
- How good is good enough?
- Is aggregate behavior okay even if micro-behavior
isnt? - What level of error mitigation by training staff
is acceptable?
6Possible Testing Boundaries
- Between trainees and entire system
- Between operator-controllers and federation
Observers
Observers
Trainers
Trainers
Interface A
Means 2
Interface D
Trainees
Role Players
Simulation
Trainees
Role Players
Simulation
Interface B
Interface F
Operator
Operator
Interface
Interface
Controllers
Controllers
CCWS
-
like
CCWS
-
like
operator
operator
Response Cell
Response Cell
Interface I
interfaces allow
interfaces al
simulation
simulation
command/
msg
command/
msg
Interface E
Technical
Technical
I/O not
I/O not
Interface C
Support
necessarily
Support
necessarily
operational
operational
format
format
Interface J
Interface H
C4ISR
C4ISR
Computer
Computer
Interface G
Systems
Systems
Simulation
Simulation
Real C4ISR
Real C4ISR
1
2
system
system
connected to
connected to
simulation
simulation
Simulation
7The Controller/Federation Boundary
- Advantage Well-defined border
- Machines on one side people on other
- Best place to evaluate simulation alone, rather
than blend of simulation staff - Only logical place to evaluate simulation
usability
8VVA Problem 2
- Vast range of possible scenarios to evaluate
- Environment
- Urban, Mountain, Jungle, Desert
- Geography, Topography, Climate, Day/Night
- Force (Air, Ground, Sea, Special)
- Equipment (sub, fighter, tank)
- Action (support, attack)
- Posture (R.O.E., etc.)
9Presentation Outline
- Introduction
- Operational-level training simulation
- The VVA problem
- Methodology
- The Missions/Means Paradigm
- Evaluation
- Application
- Terminal Fury 05
- Controlled Testing
- Conclusions
10The Use Case Strategy
- Develop a suite of strategically chosen cases
- Each case tests the simulation in a range of
tasks - Problem
- Number of cases difficult to hold down
- Example Air support with F-16 is a different
case from air support with F-18, even with all
other factors identical - Need strategy to organize use cases.
11Structuring the Myriad
- Group related cases hierarchically
- Small changes dont require completely new cases
- Source of Use Cases
- Universal Joint Training List (UJTL)
- Provides a hierarchically organized list of
operational level joint military tasks (missions) - These are the tasks the training audience is
intended to learn from using the simulation.
Missions Decomposition
12UJTL-Based Missions Decomposition
Task 2.3
Task 2.2
Task 2.3
Subtask 2.3.2
13Can the Simulation Support the Task?
- Simulations are not organized by task or case
- Instead, simulations have capabilities or means
- What can the simulation play?
- Forces air, naval, ground
- Equipment tanks, subs, ships, planes, guns
- Action movement, engagement, support
- Hierarchical list of capabilities is the
Means Decomposition
14Means Decomposition
- Much more difficult than missions
- Subject Matter Experts (SME) required
- Consider necessary means for each task
- Need intimate knowledge of military operations
related to the task - Survey requirements documents of simulation
- Ideally, an expert on the simulation, also
- Recognize and develop hierarchy
- Global Hawk ? Air Intelligence ? Air Forces
This is the most difficult and costly part of the
method
15Missions-Means Schematic
16Missions-Means Example
17Evidence Items
- Each low-level subtask, and submean has
- Input and/or Output
- Related missions and means interact through
these inputs and outputs - A task output from the trainees or staff becomes
a mean input to the simulation - A simulation output becomes an input to the staff
or trainees - These matrix elements are Evidence Items
- Low-level interactions to be evaluated
18Evidence Item Purpose
- Provide specific cases for evaluation
- Broad Goals of Evaluation
- Does Simulation provide input to stimulate
trainees performance of UJTL tasks? - Does simulation provide capabilities to execute
decisions made during performance of UJTL tasks? - Method generates a very large set of evidence
items - Specific to operational level training, not
specific to simulation -
19Presentation Outline
- Introduction
- Operational-level training simulation
- The VVA problem
- Methodology
- The Missions/Means Paradigm
- Evaluation
- Application
- Terminal Fury 05
- Controlled Testing
- Conclusions
20Evaluating Evidence Items
- To evaluate evidence items, we created a set of
Test Indicators in four categories
- Capabilities/Execution
- Existence Degree to which E.I. is supported by
the simulation (as is not role-played) - Correctness Degree to which simulation is
accurate and realistic - Completeness Fraction of critical elements
simulated. - Continuity Degree to which simulation produces
logically and temporally consistent results.
- Interface Usability
- Satisfaction How pleasant is the interface to
use? - Currency Timeliness of simulation in providing
needed information to operators. - Modality Appropriateness of push/pull type
used by simulation.
21Test Indicators, cont.
- Operator Errors
- Severe of severe errors that
- Prevent a training goal
- Cause a major pause in the simulation
- Moderate of errors that affect trainees,
multiple operators, or multiple staff members but
are not severe. - Negligible of errors whose effects are
confined to a single operator and have little
effect on training or the simulation as a whole.
- Operator Workload
- Currency Timeliness of simulation in providing
needed information to operators. - Modality Appropriateness of push/pull type
used by simulation.
22Test Indicator Evaluation
- Evaluations made on worksheets for each evidence
item - Most Ratings 05
- 0. Item not supported by simulation
- Minimal support/very difficult for operators
- Incomplete support/cumbersome usability
- Acceptable support/usability
- Good support/usability
- Complete support/ Easy/Excellent
23Presentation Outline
- Introduction
- Operational-level training simulation
- The VVA problem
- Methodology
- The Missions/Means Paradigm
- Evaluation
- Application
- Terminal Fury 05
- Controlled Testing
- Conclusions
24Terminal Fury 05
- Operated by USJFCOM
- HQd at Pacific Command (PACOM)
- Main hub of exercise (70 personnel)
- BLU Naval forces 6 JTLS operators
- RED forces 7 JTLS operators
- 24-hour testing for 10 days
- Hurlburt Field
- BLU air 8 JTLS operators
- 8-hour testing for 3 days
25Joint Theater Level Simulation (JTLS)
26Generating Evidence Items with Method
Effective
- Broad outline of TF 05 training goals allowed
focus of effort on tasks in a subset of UJTL and
corresponding means - Evidence items incumbent to exercise were
identified and indexed - Evaluation sheets were prepared automatically
within the database - Missions/Means Decomposition effective for
producing relevant evidence items for evaluation
27Applying Method at Live Exercise
Ineffective
- Early in the exercise, a handful of evidence
items were evaluated - Operators generally not busy yet
- After exercise ramped up, evidence items could
not be evaluated - Operators were too busy to rate items like
satisfaction for the many evidence items that
corresponded to their ongoing minute-to-minute
work - More to come on TF 05
28Applying Method in Controlled Testing
Effective
- Sets of evidence items were carefully selected
to correspond to single vignette/sub-scenario - Single operator on a JTLS terminal
- Two 4-hour sessions
- Testers guided operator through vignette and
evaluated each evidence item along the way - Very successful the method is very good in
controlled scenarios with no time constraints
29Lessons Learned
- Even in controlled testing, prepared vignettes
were necessary for effective testing - Starting to sound again like use cases
- Future work likely to involve blending the
Missions-Means paradigm with the use-case
paradigm - MM will form an index of available means and
pertinent missions - Use cases will be generated and mapped onto that
index
30TF 05 Observations
- Ancillary observations of note
- Often, federates behaved nominally w.r.t.
requirements, but generated subtly inaccurate
behaviors - Seemingly small issues caused major headaches
- Frequent hand-jams required to maintain
training stimuli - Many operators favored other simulations and
became very frustrated with the JTLS interface
and behaviors - Often the issue was operators desiring more
tactical control than is provided in JTLS. - Such observations would not likely occur under
controlled testing Live exercise observation
still important!
31Presentation Outline
- Introduction
- Operational-level training simulation
- The VVA problem
- Methodology
- The Missions/Means Paradigm
- Evaluation
- Application
- Terminal Fury 05
- Controlled Testing
- Conclusions
32Conclusions
- VVA is complicated by the enormous number of
possible scenarios, forces, equipment and
interactions - The Missions/Means paradigm is a two-dimensional
hierarchical model that brings structure and
organization to the problem - Method effective for generating evidence items
for testing simulation - Our M/M based evaluation proved ineffective in a
live exercise due to intense activity, but did
provide ancillary insights - Evaluation during controlled testing was very
effective when vignettes were prepared carefully - A method with use cases indexed to the
Missions/Means system is recommended
33End of presentation