M

1
MS Support to DDG 1000 Test and Evaluation

• Libby Unruh Lucia Perkins
• Naval Surface Warfare Center Dahlgren
• May 7-11, 2007

2
Outline

• Purpose
• Background
• MS Approach
• MS VVA Strategy
• MS Approach to TE
• Lessons Learned
• Summary

3
Purpose

• Provide an overview of the approach and strategy
  for using MS to support DDG 1000 Test and
  Evaluation Program
• Developmental Test (DT)
• Live Fire Test and Evaluation (LFTE)
• Operational Test (OT)

4
DDG 1000 Physical Design
Characteristics Length 600 ft Beam 80.7 ft
Draft 27.6 ft Speed 30 kt

• Displacement 14,564 LT
• Installed Power 78 MW
• Crew Size 142 (incl.
  Aviation detachment)

• Sensors
• Dual Band Radar (DBR)
• S-Band Volume Search Radar (VSR)
• X-Band Multi-Function Radar (MFR)
• HF MF Bow Sonar Arrays
• Multi-Function Towed Array
• EO/IR System
• ES System

• Weapons
• (80) Advanced vertical launch (AVLS) cells for
  Tomahawk, ESSM, Standard Missile
• (2) Advanced Gun System (AGS) 155 mm guns
• (600) 155 mm rounds
• (2) 57 mm Close In Guns (CIGS)
• Torpedo Defense (Space Reservation)
• Anti-Terrorism (Space Reservation)

Superstructure Composite structure

• Aviation
• MH60R and (3) VTUAVs
• (Capacity for 2 MH 60Rs)

Integrated Power System (IPS) (2) Main Turbine
Generators (MTG) (2) Auxiliary Turbine Generators
(ATG) (2) 34.6 MW Advanced Induction
Motors Integrated Fight Through Power

• Boats
• (2) 7m RHIBs (sized for (2) 11m RHIBs)

Hull Wave-piercing tumblehome
5
DDG 1000 Program Schedule
Hull
Phase III
Key PDR Preliminary Design Review CDR Critical
Design Review DAB Defense Acquisition Board MS
B Milestone B PRR Production Readiness
Review SR Software Release A Award (C -
Contract, AP - AP) SF Start Fabrication LK Lay
Keel L Launch D Delivery Completed Event
Award
Spiral Development Review
System Design
Prelim Design
CDR
PDR
Phase IV
DAB Review
MS B
DAB Review
Ship PRR
Ship Design
Transition Design
Detail Design
Land-Based Testing
PRR
Production Design
IPS
Build Production Unit
IPS EDM Tested
Deliver to Yard
Build IPS EDM
Land-Based Testing
PRR
Production Design
Build Production Unit
AGS
AGS EDM Tested
Build AGS EDM
Deliver to Yard
At-Sea
FQT / Land-Based
MFR EDM
Land-Based Testing
DBR EDM
Deliver to Yard
DBR CDR
DBR PDR
S-Band Change
PRR
Build DBR Production Unit
DBR Production
TSCE
SR 1
SR 2
SR 3
SR 4
SR 5
SR 6
A-LOE
SF
LK
L
D
A-AP
A-C
OPEVAL
Split Fund in FY07 / 08
DDG 1000
LK
A-LOE
SF
L
D
A-C
A-AP
Second ship delivery TBD pending ongoing contract
negotiations
DDG 1001
LK
L
A-AP
SF
A-C
DDG 1002
LK
L
A-C
A-AP
SF
DDG 1003
A-AP
SF
LK
A-C
DDG 1004
A-AP
SF
A-C
DDG 1005
A-AP
A-C
DDG 1006
6
DDG 1000 Capability Summary with Development
Schedule
7
Test Evaluation Path to IOC
2009
2010
2011
2013
2007
2008
2012
Phase III
Phase IV
Post Delivery Test Trials
PDR
CDR
PRR
PRR
MSA 1 ( PRR BL)
MSA 2 (PVT BL)
MSA 3 (Ship BL)
MSE SHIP
Stages 1-6
Stage 7
EDM Testing
DTB4D (10 Events)
DTB5 TECHEVAL (11 Events)
OPEVAL
H/W FAFT PVTFATEQT
A/B/C Trials
INCO
EME Testing
Inter Element (RV) Testing
DELIVERY
Element to -- Element Integration Testing
IOC
Spiral 1 Upgrade TBD
R1
R4
TSCE EDM System Acceptance Testing (SAT)
R2
R5
R3
R6
DTB4C (4 Events)
DTB2-260 DTB2-520
DTB4A SDTS
DTB3-350 DTB3-360 DTB3-720
DTB2-340 DTB2-720
DTB2-410
Risk Reduction
Requirements Verification Period
Evaluation Period
MSE
SI
SSD
STE
Complete
EQT deferred to Ships 3 4 Assets
8
Simulation-Based Development and System
Integration

• Requirements validation
• Performance assessment
• Hardware / software design integration testing
• Operational concepts development
• Mission effectiveness
• Reduced design cycle time
• Lower risk

System Integration and Engineering Demonstrators
Virtual Design, Integration Test

• Rapid engineering prototyping
• Engineering Development Models
• Subassembly testing
• Hardware / software in the loop
• Operational environment emulation

9
DDG 1000 MS Approach

• The fundamental principles
• Implement and use rigorous software development
  practices in compliance with established policy
• Increase confidence through graduated MS
  management oversight and VVA activities based
  upon criticality of MS intended uses to the
  application (i.e., TE)

10
DDG 1000 MS VVA Strategy

• Required by DDG 1000 contracts
• Directed by DDG 1000 Process Directive 4.004,
  Modeling and Simulation Process
• Outlined in DDG 1000 VVA Master Plan
• MS VVA is a key parameter for both Test and
  Engineering events
• Leadership requires any results of MS to be
  properly accredited before briefed at key program
  engineering milestones (SRR, PDR, CDR, PRR, )
  and prior to any DTE and LFTE event conduct
• Not all MS could / can / need to be assessed for
  accreditation
• A risk-based MS selection process is used to
  designate MS for VVA (given the criticality of
  decision, and MS reliance in decision)

11
MS Selection Process

• MS pedigree (history of prior use)
• MS credibility Gap (lack of pedigree for current
  use)
• MS criticality
• Used to assess the risk of the MS use to the
  application
• Tailored from IEEE Std 1201
• MS Criticality is determined by two factors
• Level of Criticality indicates the importance of
  the decision supported by the MS
• Level of Reliance indicates the dependence on MS
  in making the decision
• Assessed Criticality levels are based on the
  impact of the MS use to the application

MS Criticality, Pedigree, and Credibility Gap
are the Three Main Factors That Determine the
Need to Accredit an MS
12
VVA Process High-Level Overview
13
Accreditation Authority

• Accreditation authority delegated down to
  resource Holder whenever possible
• DT and LFTE events delegated to PMS 500 TE
  Director
• Engineering Events / Milestones (including SRR,
  PDR and CDR) delegated to Design Agent
• Navy Technical Team provides checks and balances
  to ensure VVA properly performed
• Accreditation Authority for OTE will be COTF

14
MS VVA Infrastructure
AA PMS 500 N (LFTE Events)
AA PMS 500N (TE Events) Raytheon/BAE (ET/EE
Events)
AA PMS 500N (TE Events) BIW/NGSS (ET/EE Events)
ARP Chair NGSS / BIW
ARP Chair TBS (DDI/BAE)
ARP Chair (DDI)
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
TRP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
MSP
Systems Requirements and Effectiveness (SRE)
System Integration
Mission Systems Equipment System Software
Development (MSE SSD )
Ship Detail Design Ship Production and Test
Ship Logistics
LFTE
Accreditation Review Panels Focused on Specific
MS Areas
15
MS VVA Summary

• Technology Development Phase
• Almost 1000 DDG 1000 Decisions (Specific Intended
  Uses) across 123 events required MS
• Over 450 individual MS were needed to achieve
  those decisions
• Over 150 MS were determined to require
  accreditation (given the importance / risk of
  decision and MS reliance in decision)
• Detail Design and Construction Phase
• About 323 SIU(s) identified
• About 239 MS needed
• Number of MS needing accreditation is under
  review

Significant Focus on VVA of Models Used at All
Levels of Usage
16
Six Sigma Optimization Project

• Project Vision

• The DDG 1000 accreditation process results in
  only the necessary accreditations being
  performed, with each being performed with minimal
  cost and schedule impact

17
Six Sigma Optimization Project (contd)

• Gathered comprehensive list of stakeholders and
  identified undesirable effects, causes, and
  opportunities
• Identify requirements early from all DDG 1000 MS
  stakeholders to obtain complete and timely
  definition of accreditation needs of Tier 1,
  Tier 2, and Training stakeholders
• Coordinated planning across multiple IPTs
  involved in DDG 1000 MS development and
  accreditation leads to a more efficient
  accreditation process
• Collaboration between DDG 1000 MS, Tier 1 and 2
  and training, promotes effective reuse of MS
  artifacts
• DDG 1000 MS artifacts from the requirements,
  development, integration and test phases can
  directly support the VVA process and all its
  required artifacts, resulting in MS developers
  driving accreditations

18
Six Sigma Optimization Project (contd)

• Key areas for execution improvement
• Reduce number of accreditations
• Improve MS selection designation process (and
  combine accreditations)
• Reduce duplicated development
• Simplify accreditation process
• Design and use risk-focused accreditation and
  other pre-tailoring steps

19
MS Outlook

• Complete VVA on MS with scope and schedule
  defined by impact on the ship
• Mitigation or reduction of a risk
• Design decision, CAIV trade-off, or change
• Shift from element perspective (box, subsystem,
  system, ...) to end-to-end mission perspective
  (IASD, IUSW, LAW, ...)

20
MS Approach to TE for DDG 1000

• MS in TE
• VVA
• MS Usage
• DTE / LFTE / OTE
• TSSS

21
MS in TE

• Models and simulations have become increasingly
  important in acquisition programs
• Increased emphasis in joint war fighting
  capabilities
• Increased emphasis in demonstrating mission
  capabilities
• Emphasis to reduce TE cost
• Higher scrutiny
• MS have a major contribution in TE
• Shorten schedules
• Reduce resources and risks
• Increase the quality of systems acquired
• Improve system performance thru insight and
  understanding
• Make comprehensive TE more affordable
• TE also plays a fundamental role in MS
  development
• Test data is often used to validate MS

22
Policy

• DODD 5000.1
• The conduct of test and evaluation, integrated
  with modeling and simulation, shall facilitate
  learning, assess technology maturity and
  interoperability, facilitate integration into
  fielded forces, and confirm performance against
  documented capability needs and adversary
  capabilities (E1.11)
• DODI 5000.2
• The TE strategy shall provide information about
  risk and risk mitigation, provide empirical data
  to validate models and simulations, evaluate
  technical performance and system maturity,.
  (E5.1.1)
• Appropriate use of accredited models and
  simulation shall support DTE, IOTE, and LFTE.
  (E5.1.4.7)

23
Challenges

• Integration of MS in TE
• How heavily should the TE community rely on MS?
• Are sufficient MS tools available to support TE
  requirements?
• How does the TE community determine that MS
  results are credible?
• Do current practices, initiatives and investment
  strategies promote credible use of MS in TE?

24
MS Strategy in TE

• DTE MS - DDG 1000 TEMP
• LFTE MS - LFTE Management Plan
• OTE MS and VVA - Section IV of the TEMP
• PM commitment to effective MS use
• Test Director takes ownership of using accredited
  MS
• It is the responsibility of the test director to
  determine the use of MS test assets or tools and
  related MS requirements
• It is also the responsibility of the test
  director to see the need for credible MS to make
  credible decisions affecting the verification
  event conduct and outcome
• Understand the relationships between
• The problem being solved
• The role of MS in the problem
• The role of VVA in establishing MS credibility

25
VVA Strategy in TE

• Promote VVA early in the test event planning (1
  to 5 years before event)
• Identify MS needs and candidate MS
• Identify specific intended use(s)
• Define high level MS requirements (what the MS
  is required to do)
• Draft a Selection Memo
• Coordination between MS and TE teams is also
  important
• Test analysts participate in VVA reviews
• MS Validation
• Test Planning Working Groups (TPWG)
• Test Planning Changes

26
MS Usage in Test Events

• Pre-Test Analysis Code A
• Develop Test Scenarios
• Predict System Under Test (SUT) behavior
• Identify performance boundaries
• Determine SUT mechanical design
• Test Execution Code B
• SUT stimulator
• Provide test scenarios
• Post-Test Analysis Code C
• Determines alternative solutions
• Allows for what-if scenario analysis
• Examines alternative hypothesis (i.e., manning
  concepts)
• Supplements test results
• Identify critical areas for subsequent tests
• MS Validation with test data Code D

27
Usage Codes
28
DTE MS and VVA

• 31 DTE events conducted
• 55 MS reviewed for usage, pedigree, and risk
• 24 MS assessed for accreditation
• 24 MS accredited with limitations
• Contributors to success
• Accreditation required by Mission Readiness
  Review
• Required by MRR Convening Letter in contracts
• MS VVA tracked during DT planning
• Use of Test Assessment Reports
• Status Reported regularly at Test Planning
  Working Groups

29
LFTE MS and VVA

• 7 LFTE events conducted
• 15 MS reviewed for usage, pedigree, and risk
• 10 MS assessed for accreditation
• 8 MS accredited with limitations
• 1 MS recommended non-accreditation
• 1 MS was accredited with limitations only for 5
  out of 7 SIUs
• Contributors to success
• MS planning and VVA approach specifically
  stated in the DDG 1000 LFTE Management Plan
• VVA process tailored to LFTE needs

30
Operational Testing

• OTD responsibilities
• Define simulation requirements early
• Intended uses, scenarios
• Fidelity
• Feasible, credible, cost effective
• Capabilities, limitations
• VV expectations
• Incorporate into TEMP (Parts IV and V)
• Monitor VV progress
• New COTF approach to Integrated Testing
• May require additional planning for integrating
  COTF MS requirements within specific DDG 1000
  program test events

31
Total Ship Systems Simulation Release 4
Total Ship System Simulation
ENGAGE
MK57 VLS
TSCEApplications
AGS/LRLAP
MT61
AGS/LRLAP
MT62
Generic Missile (ESSM/SM-2)
Acoustic Adapter
SENSE
Integrated Sonar Simulation - Active
IFF Adapter
Visualization
IFF
DBR Adapter
DBR
Scenario Generator
NAV Adapter
NAV/Ownship
SHIP SYSTEMS
IPS Adapter
Record/Playback
IPS
JVMF Adapter
Exercise Monitor and Control
EXCOMMS and OFFBOARD Resources
JVMF I/F
Intel Link Adapter
Intel Data Link I/F
Data Collection
Link 16 Adapter
Link 16 I/F
HLA LAN
CEC Adapter
CEC
TSCE LAN
R4 TSCE Applications
Updated for R4
R4 Adapter
No updates for R4
32
MS Based Integration Approach
Total Ship System Simulation (TSSS) Evolves
Consistent with S/W Releases
Software Integration Strategy is Incremental
Builds/Capability
VTUAV
VTUAV
LOS/BLOS (VHF, UHF, HF, LINK 11, 16, 22)
SATCOM (EHF, WGS, GBS, UHF)
SATCOM S-Band DDL
UCARS
EO/IR
CDL-N
CDL-N
2.75 LGR
MH-60R
DAP
DAP
MH-60R
DAP
DAP
DAP
Combat System
Sonobuoy
MMR
FLIR
ALFS
ES
Hellfire
JCM
Torpedo
DDS
PAAA
DAP
AGS
D A P
LRLAP
DAP
ESSM
DBR
DAP
SM-2
IFF
DAP
MK 57 VLS
2 D A P
DAP
VLA
HF Array
D A P
TLAM
MF Array
D A P
CIGS 57MM
3P Round
DAP
MF Towed Array
D A P
Chaff
GIANT
Decoy System
Towed Torpedo Countermeasure
D A P
DAP
LEAD
EO/IR
NULKA
DAP
ES Suite
DAP
SUPPORT
SHIP
Nav Radar
IBS
DAP
EXCOMMS
SENSE
Navigation
D A P
ENGAGE
C2I
Electro-Optical Surveillance
DAP
Aux Control
Space Weight
AVIATION
Automated Damage Control
Engineering Control System
DAP
IPS Control
33
DDG 1000 Test Bed Evolution

• DDG 1000 Test Bed Approach builds on DDG 1000
  TSSS and Navy Enterprise Investment

34
MOE Test Bed Scope

• The MOE Test Bed concept covers seven statistical
  ORD metrics
• PRA anti-ship cruise missile
• PRA aircraft
• Probability of first attack against submarines
• Probability of In-Stride Mine Avoidance (ISMA)
• Probability of mission kill against fast attack
  craft raid
• Probability of mission kill against patrol craft
  raid
• Probability of torpedo avoidance
• Implementation in one or two or five
  configurations is an engineering design decision
  to be made during execution of the Test bed

35
DDG 1000 Test Bed Configuration ASCM End State
(Real Time with Tactical HW / SW)
Network Interface Layer
TSCEi
ES Federate
ES Federate
ES Federate
DBR Federate
DBR Federate
DBR Federate
TSCE Core
TSCE Core
TSCE Core
DAPs
TSCE Core
TSCE Core
TSCE Core
Interface Federate
Interface Federate
Interface Federate
ESES
ESES
ESES
RES
RES
RES
(TSSS)
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
Low Med fidelity
A
A
A
ES DP SP
DBR DP SP
ES DP SP
ES DP SP
DBR DP SP
DBR DP SP
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
A
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
IDS compliant
IDS compliant
IDS compliant
IDS compliant
IDS compliant
IDS compliant
IDS compliant
IDS compliant
IDS compliant
system interface
system interface
system interface
system interface
system interface
system interface
system interface
system interface
system interface
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
Network Interface Layers
RTI TSCE
-
I
-
Network Interface Layer
RTI
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
RTI Interface Layer
SIMDIS
SIMDIS
SIMDIS
IDS compliant
IDS compliant
IDS compliant
Ship
Ship
ASCM
Ship
ASCM
ASCM
IDS compliant
IDS compliant
IDS compliant
system interface
system interface
system interface
motion
motion
seeker
motion
seeker
seeker
system interface
Threat, ship tactics
Threat, ship tactics
Threat, ship tactics
RePLAYS
RePLAYS
RePLAYS
high fidelity seeker
high fidelity seeker
high fidelity seeker
Ship
Ship
ASCM
Ship
ASCM
ASCM
high fidelity seeker
conditions of the day

conditions of the day

conditions of the day

signature
signature
airframe
signature
airframe
airframe
Data Collection
environment data
airframe
environment data
environment data
airframe
airframe
airframe
airframe
airframe
Display
ESSM
ESSM
ESSM
Scenario
Scenario
Scenario
ASCM
ASCM
ASCM
Environment Federate
Decoys
Environment Federate
Environment Federate
Decoys
Decoys
SM
SM
SM
autopilot
autopilot
autopilot
RTI Interface Layer
Threat/EA Federate
Threat/EA Federate
Threat/EA Federate
AGILE
AGILE
AGILE
Common Lethality
Common Lethality
Common Lethality
Server
Server
Server
LPD17 / Enterprise PRA Test Bed Re
-
Use
LPD17 / Enterprise PRA Test Bed Re
-
Use
LPD17 / Enterprise PRA Test Bed Re
-
Use
Legacy Model Re
-
Use
Legacy Model Re
-
Use
Legacy Model Re
-
Use
DDG 1000 Re-Use
DDG 1000 New or Accelerated to RT
36
TSSS (VVA) SR4

• Consolidated VVA approach
• SAT, DTE, C2I, WCE
• Designation letters
• Engineering events (Raytheon- AA)
• DTB2-340 (Navy- AA)
• Consolidated plans reports
• Accreditation and VV plan
• VV report
• VV assessment report
• Accreditation letters
• Engineering events (Raytheon- AA)
• DTB2-340 (Navy- AA)

VV Artifacts Maintained in Unclassified DOORS
(Baseline Controlled)
37
TSSS VVA (SR4) (contd)
TSSS Qualification Database
TSSS Requirements folder
Reqts MS
VI for MS
TC for MS
VS for MS
Detailed Level
SIU
High Level
MS_A req1 MS_A req2 MS_A req3
HL_ Req_01 HL_ Req_14 HL_ Req_19 HL_
Req_23 HL_Req_N
SIU_1 .. SIU_2 .. SIU_3.. SIU_N.
DL_Req_01 DL_Req_22 DL_Req_34
MS_B req1 MS_B req2 MS_B req3
Classified DOORS Portal
Unclassified DOORS Portal

• SIU, High and Detailed Level Requirements Modules
  defined by TSSS user
• Arrows show ownership of links Not Direction
• All Modules eventually link to Requirements
  Module

DDX Requirements Database
Classified DOORS Portal
38
Lessons Learned

• Must have MS VVA written into contracts
• Six Sigma is a useful tool for optimization
• Linkages from data into models and from model to
  model should be documented
• MS Master Plan needs to be constantly revised to
  fit current acquisition phase of the program
• Open communication amongst Integrated Product
  Teams is critical

39
Summary

• MS have attained an increasingly important role
  in TE
• The need for accredited MS becomes equally
  important
• MS management and VVA strategy have laid a
  solid yet flexible foundation for the DDG 1000
  TE Program
• Lots of work and challenges lay ahead

40
Questions?