SYSTEM TEST AND EVALUATION

1
SYSTEM TEST AND EVALUATION

• FAA Lifecycle AMS
• Training Forum
• August 10-11, 1999

2
TOPICS OF DISCUSSION

• Overview of System Test and Evaluation Activities
  in the Acquisition Process
• Organizational Roles in System Test and
  Evaluation
• Discussion of Tests Conducted by the Development
  Activity
• Details of the Operational Test Process

3
ACQUISITION MANAGEMENT PHASES
Mission Analysis
Investment Analysis
Solution Implementation
In-Service Management
IOTRD
Mission Need Decision
Investment Decision
In-Service Decision
Contract Award
DT
OT
SAT
SAT
IOTE
SAT
4
APPLICATION OF STE PROCESSES TO ACQUISITION
MANAGEMENT PHASES

• Investment Analysis

• Solution Implementation

In-Service Management

• Testing Needed Modifications and Improvements
• Support the Monitoring of System Performance and
  Capacity
• Support the Conduct of System and Field
  Familiarization Testing
• Collect Data to Support Mission Analysis

• Support Development of APB and RD
• Assess Feasibility of COTS/NDI and Prototype
  Solution
• Assess Testability of Requirements
• Support Development of COIs and CPPs
• Develop High Level Test Strategies

• Support Development of ASP and IPP
• Prototype Evaluation
• Specification Development
• Risk Reduction
• Support Production Decision
• Test Conduct
• Development Test
• Operational Test
• Production Acceptance Test
• Site Acceptance Test

5
ORGANIZATIONAL ROLES

• Product Team - provides the overall guidance,
  direction, resources and funding for all
  development activities including System Test and
  Evaluation.
• ACT - as a member of the Product Team has the
  responsibility for leading the System Test and
  Evaluation efforts. Interfaces with all other
  organizations involved in the STE process.
• AT/AF - initial source of operational
  requirements and provider of operational
  expertise during the conduct of System Test and
  Evaluation.
• ATQ - responsible for the conduct of the
  Independent Operational Test and Evaluation.
  Interfaces closely with System Test and
  Evaluation Team.
• AOS - responsible for testing of upgrades to
  fielded systems and provides operational
  expertise to the System Test and Evaluation Team.
• Prime Contractor - responsible for conduct of DT,
  PAT and SAT and for the implementation of all
  changes as a result of testing to the system
  under development.
• Expertise from other organizations as required.

6
FEDERAL AVIATION ADMINISTRATION
7
TEST DEFINITIONS

• Development Test
• Conducted to demonstrate that all
  technical and performance requirements specified
  in the contract have been met. Conducted by the
  contractor and witnessed by the FAA.
• Operational Test
• Conducted to demonstrate that a new system
  is operationally effective and operationally
  suitable for use in the NAS, and that the NAS
  infrastructure is ready to accept the system.
  Conducted by the FAA.
• Production Acceptance Test
• Conducted to verify the assembly line is
  producing units that have the same quality and
  performance as the first article. Conducted by
  the contractor and witnessed by the FAA.
• Site Acceptance Test
• Conducted to ensure that the system is
  installed and functioning properly in preparation
  for site acceptance. Conducted by the contractor
  and witnessed by the FAA.

8
FAA Activities
Verification of Contractor Activities
NOTE Process Description numbers from STE
Process Document
9
DEVELOPMENT TESTING
VERIFY ALL CONTRACTUAL (SPECIFICATION)
REQUIREMENTS

• Hardware Testing
• Begins at the board/component level and proceeds
  to the system level.
• Tests verify achievement of requirements relating
  to factors such as timing, current leakage,
  thermal stress and EMI.
• Software Testing
• Begins at the unit level and proceeds to the
  segment and Computer Software Configuration Item
  (CSCI) level.
• Tests verify the achievement of requirements
  through a structured bottom up approach.
• Interface Testing using simulated inputs if
  actuals are not available.
• Development testing concludes with system level
  testing to verify all contractual/specification
  requirements are satisfied.

10
OPERATIONAL TEST
VERIFY ALL OPERATIONAL REQUIREMENTS
The major components of Operational testing are

• Integration Testing - verifies that the system
  interfaces to existing elements of the NAS and
  that the new system can operate within the NAS.
• Suitability Testing - evaluates the degree to
  which a product intended for field use satisfies
  its availability, compatibility,
  transportability, interoperability, reliability,
  maintainability, safety, human factors, logistics
  supportability, documentation, personnel and
  training requirements. Suitability also assesses
  COIs.
• Effectiveness Testing - evaluates the degree to
  which a product accomplishes its mission when
  used by representative personnel in the expected
  operational environment.

11
OPERATIONAL TEST PROCESSES

• Develop test plans and procedures to document
  proper conduct of test.
• Prepare OTE laboratory to ensure
• Proper integration of all required hardware.
• Proper integration of developed software
• All required test tools are provided.
• Data collection tools are in place.
• Arrange for key site testing.
• Conduct Dry Run to
• Verify readiness of developed hardware and
  software.
• Ensure proper operation of all test hardware.
• Validate all test procedures and redline/revise
  as required.
• Collect data required for test readiness review.

12
OPERATIONAL TEST PROCESSES (Continued)

• Conduct Test Readiness Review to include review
  of
• Test objectives and schedule.
• Hardware and software configurations.
• Test personnel roles and responsibilities.
• Documentation.
• Known problems or deviations.
• Readiness to proceed.
• Test conduct
• Execution of test in accordance with plans and
  procedures.
• Accomplishment of all test objectives.
• Collection of all required data.
• Analysis of all test data.

13
OPERATIONAL TEST PROCESSES (Continued)

• Generate and track the resolution of Problem
  Reports.
• Regression test conducted to
• Exercise modified areas.
• Test other areas that may have been affected by
  changes in order to verify system performance
  integrity.
• Prepare Quicklook and Final reports to document
  results and recommendations.

14
ACRONYM LIST
APB ASP COTS/NDI COIs CPPs CSCI
DT EMI FAA IOTE IOTRD IPP NAS
PAT OT OTE RD SAT
Acquisition Program Baseline Acquisition Strategy
Paper Commercial Off the Shelf/Non-Developmental
Item Critical Operational Issues Critical
Performance Parameters Computer Program
Configuration ItemDevelopment Test Electro-Mechan
ical Interference Federal Aviation
Administration Independent Operational Test and
Evaluation Independent Operational Test and
Evaluation Readiness Declaration Integrated
Program Plan National Air Space Production
Acceptance Test Operational Test Operational Test
and Evaluation Requirement DocumentSite
Acceptance Test