CARE ASAS Validation Framework System Performance Metrics

1
CARE ASAS Validation Framework System
Performance Metrics

• 10th October 2002
• M F (Mike) Sharples

2
Content
System Performance Metrics

• Aims
• Approach
• Analysis

3
Aims
System Performance Metrics

• Using recognised metrics is fundamental to
  measuring system performance
• The ASAS Validation Framework requires consistent
  metrics to provide comparable results
• The System Performance Metrics work
  demonstrates a method for identifying existing
  metrics for new scenarios

4
Approach
System Performance Metrics

• Considerable existing work in this area
• PRS
• C/AFT
• TORCH
• INTEGRA
• Collating these required a consistent hierarchy
  taxonomy

5
Hierarchy
System Performance Metrics

• OBJECTIVES ? PERFORMANCE ? METRICS
• AREAS

6
Hierarchy
System Performance Metrics

• OBJECTIVES
• Tie in with ATM 2000 Strategy
• High level therefore no direct measure
• PERFORMANCE AREAS
• Tie in with PRC (as this gives greatest
  commonality)
• Lower level therefore easier to measure
• METRICS
• The measurements that can be made

7
Taxonomy
System Performance Metrics
METRICS ? INDICATORS CARE-ASAS PRC Validati
on C/AFT Framework TORCH

METRIC DEFINITION MEASURE Even
t, ratio or unit that is quantifiable
8
Linkage (many to many)
System Performance Metrics
Delay (not capacity) Cost effectiveness Flight
Efficiency Environment regulation Military
Co-operation Military Access Air transport
security
Metrics

• ECONOMICS
• ENVIRONMENT
• SECURITY / DEFENCE

9
Further breakdown
System Performance Metrics

• PERFORMANCE AREAS broken down into ASPECTS where
  appropriate
• Example
• ACCESS (PERFORMANCE AREA)
• Airports
• Sectors (ASPECTS)
• Routes
• Assists use with scenarios that look at specific
  airspace

10
Perspectives
System Performance Metrics

• Different views (perspectives) can be applied to
  the selection of metrics
• Airline perspective as in C/AFT
• ATM perspective as in PRC
• Validation technique
• Permits further breakdown and filtering other
  than purely hierarchical

11
Example of perspective
System Performance Metrics

• Performance Area Flight efficiency
• Airline perspective
• Actual fuel burn .v. planned fuel burn
• ATM perspective
• Efficiency of route structure

12
Characteristics
System Performance Metrics

• Further criteria for selecting metric suitability
• Objectivity Objective/subjective
• Intrusive High / Low
• Cost High / Low
• Reliability High / Low
• Validity High / Low
• Utility High / Low
• Expertise High / Low
• Resource High / Low

13
Analysis
System Performance Metrics

• To illustrate feasibility of approach a
  demonstrator database was created
• 230 System Performance Metrics stored on database
• Derived from recognised sources
• Preliminary metric classification
• Perspectives available
• ATS provider / Operator / ASAS / Analysis
  Type(or any combination of these)

14
Metrics storage
System Performance Metrics
15
Cross link queries
System Performance Metrics
16
Flexible output
System Performance Metrics
17
ASAS case studies
System Performance Metrics

• Time based sequencing in approach
• Airborne self-separation in en-route airspace

18
Metrics selection criteria
System Performance Metrics

• Time based sequencing in approach
• Selected Objectives Safety Capacity Economics
• Selected Performance Areas Safety Delay Cost
  Effectiveness Flexibility Flight Efficiency
• Methodology Each of...
• 1 Analytic or fast-time simulation
• 2 Real-time simulation
• Airspace TMA / Airport
• Perspective ASAS each of...
• 1 Operator
• 2 Service provider

19
Metrics selection criteria
System Performance Metrics

• Airborne self-separation in en-route airspace
• Selected Objectives Safety Capacity Economics
• Selected Performance Areas Safety Delay Cost
  Effectiveness Predictability Flexibility
  Flight Efficiency Equity
• Methodology Each of...
• 1 Analytic or fast-time simulation
• 2 Real-time simulation
• Airspace En-route
• Perspective ASAS each of...
• 1 Operator
• 2 Service provider

20
Metrics selection
System Performance Metrics

• Microsoft Access prototype developed to
  demonstrate the filtering and selection process
• Automated selection process provides guidance
• Identifies metrics used in previous work
• List is not definitive or restrictive
• Once automatic selection process is complete, a
  manual overview can select the most appropriate
  metrics

21
Conclusions
System Performance Metrics

• System performance metrics can be linked to the
  strategic objectives of ATM (and ASAS)
• The work has successfully consolidated metrics
  from a number of sources
• Effective filtering requires effective
  classification - this will necessarily be an
  ongoing and iterative process
• Selection process provides guidance - it is not
  definitive or restrictive

22
(No Transcript)