Managing the STORM V

1
Managing the STORM VV Program
Synthetic Theater Operations
Research Model
2
AFSAA/SAA - Analyses Foundations Directorate
3
SAA - Focus Areas
4
USAF Manager
5
Developer
6
Background
7
User Requirements
8
Common Analytical System Architecture (CASA)
9
Studytool GUI for Execution
10
Editable Flat File Input
11
Functional Model
12
Air Mission Functionality
13
ASCII Flat File Output
14
Additional Post-Processing
15
Output Tools
16
The Spectrum of VV
Utility of VV
Situational Complexity

• Low Complexity
• (Engagement Models)
• Few Systems
• 1-v-1 Assessments
• Empirically Verifiable
• VV is Objective
• VV Dominates

• Chaotically Complex
• ( Campaign Models)
• Thousands of Systems
• Force-on-Force Assessments
• Empirically Unverifiable
• VV is Subjective
• Accreditation Dominates

Source LtCols Smither and Langbehn Taking a
Functional Approach to MS Policy in Mar 02
PHALANX
17
Decision Making Perspective Theory and Practice
Source LtCols Smither and Langbehn Taking a
Functional Approach to MS Policy in Mar 02
PHALANX
18
Governing Documents

• DoD Directive 5000.59, DoD Modeling and
  Simulation Management, 1 Apr 94 w CH1, 20 Jan 98
• DoD Instruction (DoDI) 5000.61, DoD Modeling and
  Simulation Verification, Validation,
  Accreditation (VVA), 29 Apr 96
• DoD Recommended Practices Guide (RPG), VVA
  Printable Documents at http//www.msiac.dmso.mil/v
  va/default.htm
• AFPD 16-10, Modeling and Simulation Management,
  30 Jan 95
• AFI 16-1001, Verification, Validation,
  Accreditation (VVA), 1 Jun 96

19
DoDD 5000.59Basic Definitions

• Verification--The process of determining that a
  model implementation accurately represents the
  developers conceptual description and
  specifications.
• Validation--The process of determining the
  degree to which a model is an accurate
  representation of the
• real-world from the perspective of the intended
  uses of the model.
• Accreditation -- The official determination by
  the accreditation authority that the MS is
  acceptable for a specific purpose

20
AFI 16-1001 VV Concepts

• VV is a continuous process throughout a models
  lifecycle
• Different VV activities are sponsored to
  support specific accreditation needs
• A systematic VV plan will be part of MS
  development
• VV activities support model acceptance/accredita
  tion requirements defined by the accreditation
  authority
• Models must meet VV requirements of managing
  MAJCOM, FOA or DRU
• VV agent performs all VV activities and
  prepares a final VV report

21
Putting It All Together
Simulation Credibility
STRUCTURED ACCREDITATION ASSESSMENT
Output
Data
Software
Capability
Usability
Accuracy
MS Requirements
User Capabilities
Problem
Credible Solution
USABILITY
CAPABILITY
Simulation has adequate user support to
facilitate correct operation and interpretation
of its outputs
Simulation possesses all required functionality
and fidelity for the problem being solved
Process of Accreditation Assesses All These
Factors!
Source http//www.xo.hq.af.mil/xoc/8-ToolKit-VVa
ndA-DAMM5 (JASA Brief to DAMM SG 11/00)
22
Major STORM Milestones

• Program Initiation - Late 1996
• Version 0.7 Delivery - May 2001
• User Group Meeting - May 2002
• Version 1.0 Delivery - Dec 2002
• User Group Meeting - May 2003
• AFSAT Entry - Feb 2004
• Retire Thunder at AFSAA - May 2004
• User Group Meeting - May 2004
• Version 1.1 Delivery - TBD

• AFSAT Entry - Feb 2004

Source STORM CRLCMP, 3 Jan 2002
23
Criteria for AFSAT Entry
24
AFSAT Entry Process
Source http//www.xo.hq.af.mil/cgi-bin/dir_list.
pl/XOCA/DAMMSG/AFSAT
25
VVA of New Models
26
VV Process for New MS Development
Source The VV Agents Role in the VVA of New
Simulations, DoD VVA RPG, 5/15/01
27
STORM VV Process
28
Conceptual Model Mapping
29
Design Verification Mapping
30
Design Verification Mapping
31
Implementation Mapping
32
Maptool Output from STORM v0.7
33
Calendar Year 2002 Schedule
Jan Feb Mar Apr May Jun Jul
Aug Sep Oct Nov Dec
Verification, Validation and Accreditation
Activities 1. VV Plan
-------------------- 2. Derived Requirements
------------------------- 3. Conceptual
Model Validation ---------------------------
3. Design Verification -------------------
---------------- 4. Implementation
Verification ------------------------------
----- 5. Data VV --------------------------
--------------------------------------------------
------ 6. VV Testing
---------------------V 0.7-----------------
----------------- 7. STORM to THUNDER
Comparison
---
31
28
31
30
30
31
30
31
30
31
30
31
Unclassified
Classified
30
31
As of 27 Mar 02
34
Calendar Year 2003 Schedule
Jan Feb Mar Apr May Jun Jul
Aug Sep Oct Nov Dec
Verification, Validation and Accreditation
Activities 5. Data VV ---------------------
------ 7. STORM to THUNDER ---------V
1.0------------------------------------- 8.
Evaluation Support Package / Final VV Report
------------------------------------
31
28
Classified
30
31
31
30
As of 27 Mar 02
35
Post AFSAT Entry VV

• VV is a lifecycle MS activity
• VV for new STORM software releases
• Add-on scenarios and databases
• Study-specific accreditation