Development and Validation of Simulation Metrics

1
Development and Validation of Simulation Metrics

• FAA-EUROCONTROL Technical Interchange
  MeetingPerformance Metrics in Air Traffic
  ManagementToulouse, France
• 15-17 May 2002

Amanda Stott Patricia Massimini
2
En Route Congestion
Enroute IntersectionsAt or above FL270
3
Terminal Area Delays
Annual
4
Airspace Redesign Approach
5
Tools for Airspace Redesign
ArcView
ESRI
CRCT
Collaborative Routing Coordination Tool
CAASD
DPAT
Detailed Policy Assessment Tool
CAASD
EACM
Enhanced Airfield Capacity Model
CAASD
IDAT
Intersect Density Analysis Toolset
CAASD
INM
Integrated Noise Model
FAA
MapInfo
MapInfo Corp.
NASPAC
National Airspace System Performance Analysis
Capability
CAASD
NIRS
Noise Integrated Routing System
FAA / Metron
RAMS Plus
Reorganised ATC Mathematical Simulator
ISA Software
RSATS
Recombinant Sector Analysis Tool Suite
CAASD
SDAT
Sector Design and Analysis Tool
FAA
SIMMOD
ATAC
TAAM
Total Airspace and Airport Modeller
Preston
TARGETS
Terminal Area Route Generation, Evaluation and
Traffic Simulation
CAASD
TRACS
Traffic Reroute and Analysis Capability System
CAASD
6
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage

7
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage
• Delays
• Arrival and departure, ground and enroute

Cumulative
8
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage
• Delays
• Arrival and departure, ground and enroute
• User impacts
• Miles traveled

9
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage
• Delays
• Arrival and departure, ground and enroute
• User impacts
• Miles traveled
• Workload
• Sector loading

10
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage
• Delays
• Arrival and departure, ground and enroute
• User impacts
• Miles traveled
• Workload
• Sector loading
• Conflicts

11
Metrics from Simulation

• Throughput
• Peak and average runway and airport usage
• Delays
• Arrival and departure, ground and enroute
• User impacts
• Miles traveled
• Workload
• Sector loading
• Conflicts
• Visualization

12
FAA Operational Data

• Traffic
• Current traffic OAG, ETMS, SAR, ARTS
• Future traffic airport authority, major
  carriers, ATA, FAA TAF, EIS, future traffic
  models
• Operations
• OPSNET, ASPM
• Delays, taxi-out and taxi-in times, throughput,
  interarrival and interdeparture times
• ASPM, ASQP
• Safety
• Operational errors / operational deviations
• Traffic flow management actions
• Miles in trail
• Ground stops, ground delay programs

13
Degree of Correspondence Between Simulation
Results and Operational Data
Simulation

• Times for every aircraft and controller action
• Location and time of loss of separation if no
  controller action

• Altitudes
• Taxi-out times
• Delays
• Interarrival times
• Interdeparture times
• Runway throughput

Arrival delay, taxi out

• Operational errors
• Operational deviations
• Vectoring
• Traffic flow management actions

Operational Data
14
Calibration and Validation

• Typically, will calibrate baseline operations
• Runway arrival and departure rates

15
Calibration and Validation

• Typically, will calibrate baseline operations
• Runway arrival and departure rates
• Interarrival or interdeparture times

16
Calibration and Validation

• Typically, will calibrate baseline operations
• Runway arrival and departure rates
• Interarrival or interdeparture times
• Departure altitudes

• Typically, will calibrate baseline operations
• Runway arrival and departure rates
• Interarrival or interdeparture times
• Departure altitudes
• Taxi-out or taxi-in times
• Delays
• Validation
• Visual with controllers
• Analysis of summary statistics

17
Example Detroit Terminal and Cleveland En Route
Airspace
18
Example Detroit Terminal and Cleveland En Route
Airspace
DTW
MSP
BOS
ORDMDW
CLE
EWR LGA JFK
PIT
PHL
CVG
STL
BWI IAD DCA
CLT
MEM
ATL
DFW
19
Issues at ZOB and DTW

• Use of new runway for triple arrivals

22L
21R
22R
21L
20
Issues at ZOB and DTW

• Use of new runway for triple arrivals
• New arrival and departure routes and relief of
  MIT

21
Issues at ZOB and DTW

• Use of new runway for triple arrivals
• New arrival and departure routes and relief of
  MIT
• Relief of congested enroute airspace

• Use of new runway for triple arrivals
• New arrival and departure routes and relief of
  MIT
• Relief of congested enroute airspace
• Impact on ground operations

22
Traffic Schedule
Arrivals
Departures
22
23
Simulated Arrival Throughput
Triple Arrivals
Dual Arrivals
23
24
Effect of Triple Arrivals on Delay
16
49
Applies only to IMC, MIT enforced
25
Miles in Trail Restrictions on DTW Departures
26
Relieving MIT Departure Restrictions
Add departure routes
27
Effect of MIT Restrictions on Departure Delay
28
En Route Redesign to Support Relief of Departure
Restrictions

• 103 reroute sets defined, more than 1300 flights
  per day

28
29
Approach for Analysis of En Route Redesign
URET and SDAT to generate metrics
Rerouted trajectories Nonrerouted SAR tracks
SDAR--No Reroute
29
30
Results of Analysis of RedesignMax 15-Minute
Sector Counts
31
Results of Analysis of RedesignHourly Sector
Counts
32
Results of Analysis of Redesign Difference in
Sector Counts Before and After Reroutes
33
Results of Analysis of Redesign Individual
Sector Behavior
Before Reroute
Monitor Alert Parameter
After Reroute
Notional results
34
Results of Analysis of Redesign Visualization
of Flows
35
Ground Flow Analysis to Support New Runway and
Triple Arrivals
Identified chokepoints and developed new ground
routes
35
36
Concept Dependencies

• Ability to simulate is concept dependent
• Simulation tools may not be able to deliver all
  required metrics

Alt 1
Alt 3
Alt 2