Using the DoD VV

1
Using the DoD VVA Recommended Practices Guide
(RPG) JWARS VV Program Example 

• Simone YoungbloodDefense Modeling and Simulation
  Office703-824-3236syoungblood_at_dmso.mil Michael
  L. MetzInnovative Management Concepts, Inc.
  [phone] x 210mmetz_at_imcva.com

2
Purpose and Overview

• Purpose To demonstrate a practical application
  of the Department of Defense (DoD) Verification,
  Validation and Accreditation Recommended
  Practices Guide for a new simulation development
  effort
• Overview
• The RPG and its evolution
• the Joint Warfare System (JWARS) simulation
• The JWARS VV Process based on the RPG

3
DoD VVA Guidance

• DoD Recommended Practices Guide (RPG)
• Defines underlying philosophy, principles, and
  methodologies recommended for use in DoD VVA
  efforts
• 1996 Version
• Millennium Edition

4
DOD RPG HISTORY

• DoD Instruction 5000.61
• Established common terminology and defined high
  level roles and responsibilities
• DoD RPG
• Defined underlying philosophy, principles,
  methodologies recommended for use in DoD VVA
  efforts
• First distributed in 1996
• Well-received (welcomed)
• adopted by numerous programs (e.g., JADS, JWARS,
  JSIMS)
• accepted by Services -- used as foundation for
  Service VVA policies

5
Objective
Millennium Edition

• Account for diversity of audience

• Account for diversity of situations

• Focus on practical versus theoretical

6
Vision

• The RPG will be a multi-tiered,
• multi-dimensional document
• Non-linear
• Reader-based

7
Approach

• Produce a web-based document
• Structure using multiple layers and paths
• Advantages
• breadth depth
• facilitation of additional info
• empowerment of the reader

8
DoD VVA RPG
The Millennium Edition is a highly flexible,
multi-tiered, multi-dimensional document which
provides the state of the practice for VVA.
BREADTH
9
(No Transcript)
10
(No Transcript)
11
Build 1.5 15 Jan 2001

• Major Functional Upgrade
• Special Topics
• Fidelity
• Requirements
• Risk and Its Impact on VVA
• Reference Documents
• Human Behavior Representation Literature Review
  
• VV Techniques
• VV Tools
• Technical Editing
• Printable Documents
• PDF files
• Table of Contents
• Page Numbers

12
Build 2 15 Apr 2001

• Legacy Core Documents
• Key Concepts
• Special Topics
• Data VV for Legacy Simulations
• Special Topics
• Human Behavior Representation Validation
• Problem Analysis
• VV Techniques
• Reference Documents
• Credibility (preliminary)

Build 2.5
13
Build 3

• Federation Core Documents
• Special Topics
• VVA and TE
• VVA Tools
• Validation Techniques Elaborated
• Metrics
• Cost
• Environmental VV Issues
• Reference Documents
• Credibility (final version)

14
RPG Status

• Build 1 (focus on new developments) 22 May 00
• Build 1.5 (major functional upgrade) 15 Jan 01
• Build 2 (adds legacy) 15 Aug 01

http//www.msiac.dmso.mil/vva
15
Practical Application The JWARS Example
Joint Warfare System (JWARS) VV directed by
the Program Manager to use DoD VVA RPG as the
basis for integrated simulation development/VV
process
JWARS information is available at
http//www.jointmodels.mil/jwars
16
JWARS Mission

• Mission Develop a state-of-the-art, constructive
  simulation that will
• Provide a multi-sided and balanced representation
  of joint theater warfare
• Be able to assess current and future operational
  concepts to include Joint Vision 2010s
• Dominant Maneuver, Precision Engagement, Focused
  Logistics, and Full-Dimension Protection
• Use C4 and ISR as the foundation for how entities
  perceive and interact with one another

• Applications
• 1. Force assessment
• 2. Planning and execution
• Deliberate planning
• Crisis action planning
• 3. System effectiveness and trade off analysis
• 4. Concept and doctrine development and assessment

• Users
• Joint Staff
• Services
• CINCs
• OSD
• Joint Task Forces
• Other DoD org's
• Industry

17
JWARS Requirements - Key Performance Parameters
(KPPs)

• Traceability
• Cause and effect relationships
• Tracking of data sources
• Verification and validation (VV)
• Correct representation of doctrine, performance,
  and environment
• Balance across joint warfare functions
• Utility
• Study execution
• Includes warfare functionality
• Deterministic and stochastic modes
• Multiple levels of resolution
• Execution speed

18
The DMSO VVA Framework
19
VV Status ReportVV Program Objective

• Risk Reduction
• help the JWARS Office meet the ORD requirements
• identify development problems through the VV
  process and recommend corrections early when
  theyre easier and less expensive to fix
• work with the Operational Test Authority to
  coordinate activities
• work with the JWARS Study Team (JST) to share
  information
• Provide an Accreditation Package
• follow JWARS VV Plan to create JWARS VV Reports
• create body of evidence for use in Accreditation
  process
• Via the use of the RPG VV Techniques

20
JWARS Verification Validation JWARS ORD

• JWARS ORD
• (2) Verification and Validation. JWARS shall
  demonstrate and enable the correct representation
  of doctrine, system and unit performance, the
  environment and balance among joint warfare
  functions for U.S. forces, Allies, coalition
  partners and potential adversaries (KPP).
• Threshold. At each release the objects and
  algorithms in JWARS that represent doctrine,
  system and unit performance, and the environment
  shall be verified and validated in accordance
  with the Verification and Validation Plan.
  Balance shall be maintained by equitable
  representation of joint warfare functions,
  consistent with their impact on theater warfare
  operations, and within the context of the
  functionality described at Appendix C.

21
JWARS VVORD Appendix A Force Assessment
Support Requirements

• Identifying forces required to execute the
  National Military Strategy
• Analyzing capability of existing forces,
  logistics, C4 and ISR for multiple contingencies
  or theaters.
• Analyzing requirements to support building the
  POM
• Assessing the capability of proposed out-year
  force structures required to meet alternative
  future scenarios
• Determining the impacts of readiness and
  training on warfighting performance.

22
JWARS VVORD Release 1 (Limited IOC)Study
Execution Requirements

• JWARS shall include C4, ISR, logistics
  capabilities and essential functionality that
  exists in the current MIDAS and TACWAR models
• This threshold is described by the warfare
  functionality at Appendix C
• Release 1 shall be capable of replacing the use
  of TACWAR to support the Force Assessment
  application at Appendix A

23
JWARS VVORD Appendix C Release 1 Threshold
Warfare Functionality

• 51 Thread descriptions grouped into four areas
• Strategic Logistics
• Theater Logistics
• Perception
• Operations
• Represent functional equivalency for MIDAS and
  TACWAR per ORD Paragraph 4.b.(3)(a)(i)

24
JWARS Integrated Software Development and VV
Process
MS Results
25
JWARS VV Plan

• JWARS VV Teams initial task of developing the
  VV Plan
• Use VV Process group developed
• follow JWARS Office Software Development Process
• dont create artifacts solely for VV use
• identifies JWARS User Subgroup members as the
  Validation Agent for warfighting functionality
  products
• identifies JWARS VV Agent as Verification Agent
• identifies JAMIP SC as approval authority for all
  products
• Based on DMSOs DoD VVA Recommended Practices
  Guide
• Approved by JWARS Oversight Group (Joint Analytic
  Model Improvement Program or JAMIP Steering
  Committee in August 98
• directed to combine processes with TE OTA as
  much as possible
• Identifies Roles, Processes, Reports

26
JWARS VV Plan, Processes, Reports Documents
27
New JWARS VV Process
Proposed Work Packages With Embedded Pre-Design
Artifacts
Legend
Feedback
State of Model
ICRs
Normal Flow
Joint Change Requests
Approved Work Packages
Requirements Analysis
JAMIP/CCB Review
Work Package V V
Internal Change Requests
High Level Design VV
Algorithm Val
HLD Rev
Results Validation
Implementation Verification
Detailed Design Ver
Validated Software Release
Application Software
Code Ver
Feedback from V V testing in the later stages
of the process will increasingly take the form of
Internal Change Requests (ICRs) to be addressed
in subsequent iterations to avoid unnecessary
delays in release dates.
28
JWARS VV Processes

• Pre-Design Artifact VV
• Joint Application Design (JAD) Package VV
• Work Package VV
• Conceptual Model of the Mission Space (CMMS)
  Validation
• Algorithm Validation
• High Level Design (HLD) Verification
• Detailed Design (DD) Verification
• Model Validation Criteria (MVC) Development and
  Validation
• Implementation Verification
• Results Validation (RV)

29
High Level Design Verification Process
HLD Documents
JAD Packets
ORD
Thread Intent Statement
Thread Intent Statement
Satisfier
Required Actions and Interactions
Required Actions and Interactions
Satisfiers
Measures
Measures
Satisfiers
30
Detailed Design Verification Process
Requirements
High Level Design
Detailed Design
Things
Implementing Protocols
JADs
Responsibilities
Messages
31
VV Status ReportResults Validation (RV)

• Last VV step for each release
• Release .5 (Alpha Test Version) no RV
• Release 1 (limited IOC) complete RV
• Releases 2 (full IOC) and 3 (FOC) complete RV
• Compares simulation output against (in order of
  goodness)
• Real World Results
• Test Evaluation Results
• Other MS Results (TACWAR and MIDAS)
• VV Agent predictions (Model Validation Criteria
  - developed prior to HLD complete in each
  Iteration included in the release) as a basis for
  sensitivity analysis
• Expert Opinion (AKA Face Validation)

32
JWARS VV Database

• Customized tool to support entire JWARS VV
  process
• Provides traceability of derived requirements to
  ORD
• Contains all Universal Joint Task List (UJTL)
  tasks and measures
• Produces detailed VV reports
• Provides mechanism to trace design products to
  requirements and implementation
• Contains detailed textual and graphical
  description of algorithms
• Designed to capture all developer and VV Agent
  products to support follow-on accreditation
  activities
• Developed in Microsoft Access 2000
• Provides synchronization of replicated databases
  over the Internet
• Supports multi-user secure access to encrypted
  database

33
(No Transcript)
34
(No Transcript)
35
(No Transcript)
36
(No Transcript)
37
(No Transcript)
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
JWARS VV Accomplishments

• Planning and Procedures
• Developed the JWARS VV Plan with JWARS Office,
  JWARS Developers, OSD, J-8, Service, DMSO, and
  VV community participation
• ensures plan is meaningful and achievable
• ensures plan meets DoD requirements
• based on the DoD VVA RPG
• Developed the JWARS VV Procedures with JWARS
  Office (based on VV Plan) that keeps VV process
  up to date with changing JWARS Software
  Development Process
• User Community Involvement
• Provided User Community review process and input
  method to JADs, CMMS, and initial HLD products
• Oversight and Management Involvement
• Provided VV process and result information to
  Oversight Groups (VV-TE WIPT, JAMIP SC and
  EXCOM)

43
JWARS VV Accomplishments

• Improved Initial JWARS SDP
• Pre-design artifacts
• identified format and content inconsistencies
  between JWARS IPTS that lead to better definition
  and standardization
• identified JAD Session inconsistencies that
  caused process to be better defined and
  standardized
• Conceptual Model of the Mission Space (CMMS)
• identified problems (based on user comments) with
  content and format of original text-based CMMS
  product
• identified internal JWARS problems with utilizing
  CMMS
• led to changes in CMMS products and process

44
JWARS VV Accomplishments

• Improved JWARS SDP (continued)
• High Level Design
• identified problems with initial HLD products
  (JWARS Enterprise Model and associated artifacts)
  and the transitions from tool to tool
• led to changes in the HLD process and products to
  UML Designer
• worked with JWARS Object Analysts to define the
  transition from JAD Packages to UML Designer HLD
  artifacts (method, content, format)
• Detailed Design
• Identified problems with traceability from HLD
  due to reverse engineering process
• Worked with JWARS OOA/OOD software engineers to
  improve process and utilize UML Designer
  capabilities
• JWARS Output
• maintained focus on requirement for JWARS to meet
  requirements to support analytic process with
  proper analytic outputs
• developed process for creation of defined Model
  Validation Criteria for use in final VV Results
  Validation
• helped convince JWARS to use the JST Metrics and
  JST Reports as a form of User Space Analysis
  (USA) and basis for Results Validation

45
JWARS VV Status
C complete I in progress P in
planning/preparation
slide 45
46
JWARS VV Summary

• Implemented concepts of RPG
• RPG provided guiding set of principles for
  original processes and plan
• RPG supported expanded concepts and detailed
  process development
• Used lessons learned from process to feedback
  into the RPG update process
• Risk Reduction
• Achieved most risk reduction goals
• Improved JWARS simulation development process
• Increase Credibility (body of evidence)
• Recorded every artifact and VV event
• provided VV Reports
• provided VV detailed database

47
JWARS VV Results ValidationComparison to OTE
Results

• Operational Test Evaluation (OTE) Results
• Difficult to obtain
• Difficult to make direct comparisons of
  one-on-one tests to a theater level simulation
• Must find acceptable OTE results and then
  attempt to create a snippet or vignette of the
  simulation scenario that is close enough
• Then make multiple runs of the stochastic
  simulation and find the range and mean
• Then compare the simulation output to the OTE
  result
• If significantly different, attempt to understand
  why
• This process is just beginning for JWARS with a
  series of meetings with Pentagon OTE officials

48
JWARS VV ResultsComparison to Other MS Results

• Comparison to other MS Results
• Easier to obtain than OTE results
• Challenging to make direct comparisons of
  different theater level simulations
• Must find comparable simulation scenarios that
  are close enough
• Then make a single run of deterministic referents
  and multiple runs of the stochastic JWARS
  simulation and find the range and mean
• Then compare the simulation outputs to one
  another
• If significantly different, attempt to understand
  and explain why
• This process is in work for JWARS for the TACWAR
  comparison
• Process will continue with MIDAS next and then
  many of the other simulations JWARS will replace

49
Release 1.3 Results ValidationPlanning
TACWAR Comparison

• Objective Conduct the JWARS output comparison
  to TACWAR output as part of the JWARS Release 1.3
  Results Validation process
• Criteria
• Versions - JWARS 1.3 and TACWAR 5.3.1
• Scenarios
• Orders of Battle
• Concepts of Operation
• Identify candidate output for comparison
• Establish comparison conditions
• Process
• Conduct Comparison
• Identify similarities and differences
• Adjudicate with TACWAR and JWARS experts to
  determine (estimate?) reasons for differences
• Schedule
• Resources
• Report To JWARS Office, JWARS VV-TE WIPT,
  JAMIP SC and include in JWARS Release 1.3 VV
  Report

50
JWARS VV Results ValidationSME Review

• Comparison to other Expert Opinion
• Easiest to obtain but must use caution
• Possible but not trivial to compare expert
  opinion to JWARS output
• Two methods being used
• First is the comparison of JWARS output to VV
  Agents MVC
• Has been difficult because JWARS developers did
  not build representation to the MVCs
• Must choose, in an ad hoc manner, what input
  variables can be changed and then make a
  prediction of direction of output movement and
  gross quantity
• Has been successful to date and has been useful
  to the developers
• Second is a more formal Subject Matter Expert
  (SME) comparison planned for JWARS Release 1.4
  and or 1.5 in 2002
• The first part of this process has been used for
  JWARS Release 1.1 and 1.2
• Although not as structured as planned it has been
  successful
• Will continue for all future JWARS Releases

51
JWARS VV Results ValidationSME Opinion

• Formal Subject Matter Expert (SME) comparison
  planned for JWARS Release 1.4 and or 1.5 in 2002
• Now planning for
• identification of SMEs, method (group meetings or
  distributed internet meetings)
• Qualification and weighting of SMEs selected by
  experience
• Development of scenario snippets or vignettes
  with sufficient information for SMEs to make a
  meaningful prediction
• Method for comparing SME predictions to JWARS
  output
• Care must be taken in how the SME opinion is
  collected
• VV Agent must be prepared to explain gross
  differences