Approximating Sensor Footprints - PowerPoint PPT Presentation

1 / 33
About This Presentation
Title:

Approximating Sensor Footprints

Description:

Old Dominion University. Wesley N. Colley, Ph.D. and Mikel D. Petty, Ph.D. ... University of Alabama in Huntsville. 2005 FALL SIW. 05F-SIW-039 2. Presentation Outline ... – PowerPoint PPT presentation

Number of Views:24
Avg rating:3.0/5.0
Slides: 34
Provided by: mikel184
Category:

less

Transcript and Presenter's Notes

Title: Approximating Sensor Footprints


1
Fall 2005 Simulation Interoperability
Workshop 05F-SIW-039 An Application and
Evaluation of a Methodology for Testing
Operational Level Training Simulations MAJ Karl
D. Neal Virginia Modeling, Analysis, and
Simulation Center Old Dominion University Wesley
N. Colley, Ph.D. and Mikel D. Petty, Ph.D.
Center for Modeling, Simulation, and
Analysis University of Alabama in Huntsville
2
Presentation Outline
  • Introduction
  • Operational-level training simulation
  • The VVA problem
  • Methodology
  • The Missions/Means Paradigm
  • Evaluation
  • Application
  • Terminal Fury 05
  • Controlled Testing
  • Conclusions

3
Operational-level Training Simulations
  • Operational Level of War
  • Operational-level training simulations
  • Simulations used in constructive components of
    training exercises for operational level
    commanders and their staffs in conducting warfare
    at the operational level.

The operational level links the tactical
employment of forces to strategic objectives. The
focus at this level is on operational art the
use of military forces to achieve strategic goals
through the design, organization, integration,
and conduct of strategies, campaigns, major
operations, and battles (DoD Joint Pub 3.0,
2003).
4
Verification, Validation, and Accreditation
  • Verification
  • Determination whether simulation meets design
    goals
  • Validation
  • Determination whether simulation reproduces
    reality at desired fidelity
  • Accreditation
  • Official certification that a simulation may be
    used for a specific purpose

5
VVA Problem 1What does does it work mean?
  • Stimulates correct training goals?
  • Produces valid entity interactions?
  • Is sufficiently usable for the training staff?
  • Even the concept of correct is tricky.
  • How good is good enough?
  • Is aggregate behavior okay even if micro-behavior
    isnt?
  • What level of error mitigation by training staff
    is acceptable?

6
Possible Testing Boundaries
  • Between trainees and entire system
  • Between operator-controllers and federation




Observers
Observers
Trainers

Trainers

Interface A

Means 2
Interface D




Trainees
Role Players
Simulation

Trainees
Role Players
Simulation

Interface B
Interface F
Operator


Operator


Interface
Interface


Controllers
Controllers






CCWS
-
like
CCWS
-
like




operator
operator
Response Cell
Response Cell
Interface I


interfaces allow
interfaces al


simulation
simulation




command/
msg
command/
msg
Interface E


Technical

Technical

I/O not
I/O not


Interface C
Support

necessarily
Support

necessarily


operational
operational


format
format
Interface J
Interface H


C4ISR

C4ISR

Computer
Computer
Interface G
Systems

Systems

Simulation

Simulation



Real C4ISR

Real C4ISR
1

2


system
system


connected to
connected to


simulation
simulation
Simulation
7
The Controller/Federation Boundary
  • Advantage Well-defined border
  • Machines on one side people on other
  • Best place to evaluate simulation alone, rather
    than blend of simulation staff
  • Only logical place to evaluate simulation
    usability

8
VVA Problem 2
  • Vast range of possible scenarios to evaluate
  • Environment
  • Urban, Mountain, Jungle, Desert
  • Geography, Topography, Climate, Day/Night
  • Force (Air, Ground, Sea, Special)
  • Equipment (sub, fighter, tank)
  • Action (support, attack)
  • Posture (R.O.E., etc.)

9
Presentation Outline
  • Introduction
  • Operational-level training simulation
  • The VVA problem
  • Methodology
  • The Missions/Means Paradigm
  • Evaluation
  • Application
  • Terminal Fury 05
  • Controlled Testing
  • Conclusions

10
The Use Case Strategy
  • Develop a suite of strategically chosen cases
  • Each case tests the simulation in a range of
    tasks
  • Problem
  • Number of cases difficult to hold down
  • Example Air support with F-16 is a different
    case from air support with F-18, even with all
    other factors identical
  • Need strategy to organize use cases.

11
Structuring the Myriad
  • Group related cases hierarchically
  • Small changes dont require completely new cases
  • Source of Use Cases
  • Universal Joint Training List (UJTL)
  • Provides a hierarchically organized list of
    operational level joint military tasks (missions)
  • These are the tasks the training audience is
    intended to learn from using the simulation.

Missions Decomposition
12
UJTL-Based Missions Decomposition
Task 2.3
Task 2.2
Task 2.3
Subtask 2.3.2
13
Can the Simulation Support the Task?
  • Simulations are not organized by task or case
  • Instead, simulations have capabilities or means
  • What can the simulation play?
  • Forces air, naval, ground
  • Equipment tanks, subs, ships, planes, guns
  • Action movement, engagement, support
  • Hierarchical list of capabilities is the

Means Decomposition
14
Means Decomposition
  • Much more difficult than missions
  • Subject Matter Experts (SME) required
  • Consider necessary means for each task
  • Need intimate knowledge of military operations
    related to the task
  • Survey requirements documents of simulation
  • Ideally, an expert on the simulation, also
  • Recognize and develop hierarchy
  • Global Hawk ? Air Intelligence ? Air Forces

This is the most difficult and costly part of the
method
15
Missions-Means Schematic
16
Missions-Means Example
17
Evidence Items
  • Each low-level subtask, and submean has
  • Input and/or Output
  • Related missions and means interact through
    these inputs and outputs
  • A task output from the trainees or staff becomes
    a mean input to the simulation
  • A simulation output becomes an input to the staff
    or trainees
  • These matrix elements are Evidence Items
  • Low-level interactions to be evaluated

18
Evidence Item Purpose
  • Provide specific cases for evaluation
  • Broad Goals of Evaluation
  • Does Simulation provide input to stimulate
    trainees performance of UJTL tasks?
  • Does simulation provide capabilities to execute
    decisions made during performance of UJTL tasks?
  • Method generates a very large set of evidence
    items
  • Specific to operational level training, not
    specific to simulation

19
Presentation Outline
  • Introduction
  • Operational-level training simulation
  • The VVA problem
  • Methodology
  • The Missions/Means Paradigm
  • Evaluation
  • Application
  • Terminal Fury 05
  • Controlled Testing
  • Conclusions

20
Evaluating Evidence Items
  • To evaluate evidence items, we created a set of
    Test Indicators in four categories
  • Capabilities/Execution
  • Existence Degree to which E.I. is supported by
    the simulation (as is not role-played)
  • Correctness Degree to which simulation is
    accurate and realistic
  • Completeness Fraction of critical elements
    simulated.
  • Continuity Degree to which simulation produces
    logically and temporally consistent results.
  • Interface Usability
  • Satisfaction How pleasant is the interface to
    use?
  • Currency Timeliness of simulation in providing
    needed information to operators.
  • Modality Appropriateness of push/pull type
    used by simulation.

21
Test Indicators, cont.
  • Operator Errors
  • Severe of severe errors that
  • Prevent a training goal
  • Cause a major pause in the simulation
  • Moderate of errors that affect trainees,
    multiple operators, or multiple staff members but
    are not severe.
  • Negligible of errors whose effects are
    confined to a single operator and have little
    effect on training or the simulation as a whole.
  • Operator Workload
  • Currency Timeliness of simulation in providing
    needed information to operators.
  • Modality Appropriateness of push/pull type
    used by simulation.

22
Test Indicator Evaluation
  • Evaluations made on worksheets for each evidence
    item
  • Most Ratings 05
  • 0. Item not supported by simulation
  • Minimal support/very difficult for operators
  • Incomplete support/cumbersome usability
  • Acceptable support/usability
  • Good support/usability
  • Complete support/ Easy/Excellent

23
Presentation Outline
  • Introduction
  • Operational-level training simulation
  • The VVA problem
  • Methodology
  • The Missions/Means Paradigm
  • Evaluation
  • Application
  • Terminal Fury 05
  • Controlled Testing
  • Conclusions

24
Terminal Fury 05
  • Operated by USJFCOM
  • HQd at Pacific Command (PACOM)
  • Main hub of exercise (70 personnel)
  • BLU Naval forces 6 JTLS operators
  • RED forces 7 JTLS operators
  • 24-hour testing for 10 days
  • Hurlburt Field
  • BLU air 8 JTLS operators
  • 8-hour testing for 3 days

25
Joint Theater Level Simulation (JTLS)
  • Main Driver of TF 05.

26
Generating Evidence Items with Method
Effective
  • Broad outline of TF 05 training goals allowed
    focus of effort on tasks in a subset of UJTL and
    corresponding means
  • Evidence items incumbent to exercise were
    identified and indexed
  • Evaluation sheets were prepared automatically
    within the database
  • Missions/Means Decomposition effective for
    producing relevant evidence items for evaluation

27
Applying Method at Live Exercise
Ineffective
  • Early in the exercise, a handful of evidence
    items were evaluated
  • Operators generally not busy yet
  • After exercise ramped up, evidence items could
    not be evaluated
  • Operators were too busy to rate items like
    satisfaction for the many evidence items that
    corresponded to their ongoing minute-to-minute
    work
  • More to come on TF 05

28
Applying Method in Controlled Testing
Effective
  • Sets of evidence items were carefully selected
    to correspond to single vignette/sub-scenario
  • Single operator on a JTLS terminal
  • Two 4-hour sessions
  • Testers guided operator through vignette and
    evaluated each evidence item along the way
  • Very successful the method is very good in
    controlled scenarios with no time constraints

29
Lessons Learned
  • Even in controlled testing, prepared vignettes
    were necessary for effective testing
  • Starting to sound again like use cases
  • Future work likely to involve blending the
    Missions-Means paradigm with the use-case
    paradigm
  • MM will form an index of available means and
    pertinent missions
  • Use cases will be generated and mapped onto that
    index

30
TF 05 Observations
  • Ancillary observations of note
  • Often, federates behaved nominally w.r.t.
    requirements, but generated subtly inaccurate
    behaviors
  • Seemingly small issues caused major headaches
  • Frequent hand-jams required to maintain
    training stimuli
  • Many operators favored other simulations and
    became very frustrated with the JTLS interface
    and behaviors
  • Often the issue was operators desiring more
    tactical control than is provided in JTLS.
  • Such observations would not likely occur under
    controlled testing Live exercise observation
    still important!

31
Presentation Outline
  • Introduction
  • Operational-level training simulation
  • The VVA problem
  • Methodology
  • The Missions/Means Paradigm
  • Evaluation
  • Application
  • Terminal Fury 05
  • Controlled Testing
  • Conclusions

32
Conclusions
  • VVA is complicated by the enormous number of
    possible scenarios, forces, equipment and
    interactions
  • The Missions/Means paradigm is a two-dimensional
    hierarchical model that brings structure and
    organization to the problem
  • Method effective for generating evidence items
    for testing simulation
  • Our M/M based evaluation proved ineffective in a
    live exercise due to intense activity, but did
    provide ancillary insights
  • Evaluation during controlled testing was very
    effective when vignettes were prepared carefully
  • A method with use cases indexed to the
    Missions/Means system is recommended

33
End of presentation
Write a Comment
User Comments (0)
About PowerShow.com