Line Operations Safety Audit (LOSA): A Practical Overview

1
Line Operations Safety Audit (LOSA) A Practical
Overview
Third ICAO-IATA LOSA TEM Conference Malaysia
Airlines, Kuala Lumpur - September 13-14, 2005

• James Klinect and Patrick Murray
• The University of Texas / The LOSA Collaborative

2
Introduction James Klinect

• University of Texas (UT) LOSA research for the
  industry
• Project Coordinator TEM LOSA
• Director Bob Helmreich
• LOSA Collaborative LOSA implementation for
  airlines
• Role Founder/President
• De-identified data to University of Texas
• Disclaimer Not an commercial for the LOSA
  Collaborative
• Sharing of mistakes made and lessons learned
• Provide a enough information to evaluate LOSA and
  possibly implement it themselves (ex, Varig, Lan
  Chile,.)

3
LOSA Airlines 1996-2005
Since 1996, 4500 observations, 25 airlines 11
countries

• AeroMexico
• Asiana Airlines
• Alaska Airlines
• Air New Zealand
• Braathens
• Cathay Pacific
• China Airlines
• COPA

Continental Airlines Continental
Express Continental Micronesia Delta Airlines EVA
Air Frontier Airlines LACSA QANTAS
Malaysia Airlines Mt. Cook Regional
Express Singapore Airlines Silk Air TACA / TACA
Peru UNI Air US Airways
4
Presentation Outline

• LOSA Rationale / Deliverables
• Why LOSA and what do you get?
• LOSA Operating Characteristics
• What is the definition of LOSA and how does it
  work?
• LOSA E-mail Inbox
• What are the most frequently asked LOSA
  questions?
• Audience Question and Answer

5
LOSA Rationale / Deliverables
6
Aviation Safety Envelope
Accidents
Incidents
Normal Operations
7
Safety Data Coverage
Accidents
Incidents
Normal Operations
8
Aviation Safety Envelope
Accidents
Incidents
Normal Operations
LOSA Rationale Proactive snapshot of system /
crew performance strengths and weaknesses in
normal operations (just like a health checkup)
9
LOSA Measurement Challenge

• Challenge - What type of observational measure
  best captures safety performance data in normal
  operations?
• Concerns
• Observer overload Cant capture everything
• Data analyst overload Must have a roadmap for
  interpretation
• Manager overload Data need face value and point
  to safety issues
• Solution Threat and Error Management (TEM)

10
Primary LOSA Measures TEM

• What do flight crews do to safely fly from A to B?

Wheres CRM and human factors? Non-technical
skills are one of many contributors to
effective/ineffective TEM performance
11
LOSA Deliverables Diagnostic Findings

• Most frequent and mismanaged threats
• Most frequent and mismanaged errors
• Fleet and operational differences
• Comparison to other airlines on LOSA safety
  indices

12
Sample LOSA Analysis
13
Raw Data Narratives
LOSA Flight 23 Error 1 A340 ARPT 401 to
ARPT 204 (Crew 8) Phase of Flight
Takeoff/Climb Pilot Flying CA Takeoff / Climb
Narrative Almost immediately after takeoff, the
Captain started a turn up the river, and did not
wait until 400 feet, per SOP. About 300 feet,
captain reached over to MCP (called FCU on
Airbus) and selected Flight Path Angle. He did
not tell FO of this selection, which is very
unusual of a setting. In fact, the FO later asked
me if I had ever seen anyone use this selection
and stated that he has never seen it in his 2
years of being on the Airbus. Because the Captain
did not tell the F/O what he was doing, when the
FO noticed this setting, he reselected the more
common mode (Heading). Captain then reached over
and then reselected Flight Path Angle. This led
to a brief discussion of whether of not the
aircraft would capture NAV in this setting. As a
result of all of this, the FO missed ATC handoff
to departure control, a call that was not made
until the aircraft leveled off at 5000 feet. Both
Captain and FO were involved in heads-down stuff
as the aircraft leveled off. If the autopilot
would not have leveled off, then the crew would
have not caught it, because they were doing
extraneous things. (FO, programming MCDU
Captain, looking at weather).
14
Raw Data Threats
LOSA Flight 50 Threat 2 B737-700 AUS /
DFW Threat Description Threat runway switch. We
had expected and had briefed 17Left from the ATIS
broadcast. No reason for the switch given by
ATC. Threat Management Description Scramble to
brief new runway plus linked to error of not
finding the proper approach chart and automation
error of not setting up computer for the landing
runway. FO left the approach and arrival route
for 17Left in the FMC instead of the new runway.
Based on this the aircraft reached DIETZ and
started a turn to heading 350 which was still the
active route instead of the 300 heading
assigned. Phase of Flight Des/App/Land Threat
Type ATC Threat Code ATC runway change Threat
Outcome Linked to Flight Crew Error
15
Summary - LOSA

• Primary features
• Purely diagnostic no solutions flight
  operations health check
• Captures system safety and pilot performance
  strengths and weaknesses in normal operations
  using observations
• Analysis
• Aggregated TEM data provides a safety snapshot of
  flight operations on a typical day
• Organizational change
• Measure the effectiveness safety solutions with a
  follow-up LOSA

16
LOSA Case Study One

• Two LOSAs (3 years apart)
• 1st LOSA results An example of targets
  identified
• Undesired number of below standard leadership
  ratings
• Problems associated with checklist protocol /
  usage
• High number of unstable approaches
• Low rate of error trapping (errors detected and
  managed)

17
LOSA Case Study One

• Airline response to data
• Formed committees to address targets for
  enhancement
• Implemented a leadership training module in
  recurrent
• Revised checklist philosophy
• Established bottom lines for unstable approaches
• Implemented error management training course for
  all pilots
• Check airmen were trained in evaluating error
  management

18
LOSA Case Study One

• 2nd LOSA results
• Captain leadership scores significantly improved
• 40 reduction in checklist errors
• 77 reduction in unstable approaches below 1000
  ft.
• FOQA (QAR) showed similar trends
• Two fold increase in error trapping rates
• Change in the check culture to evaluate error
  management
• Increase management buy-in on the safety change
  process

19
LOSA Case Study Two

• 1st LOSA results - Targets identified
• Improve energy management issues during
  descent/approach/land
• 18 of flights with a speed deviation undesired
  aircraft state
• Lower the prevalence of unstable approaches
• 11 of flights with an unstable approach
• Lower prevalence of checklist errors
• 50 of flights with a checklist error
• Improve monitor/cross-checking performance
• 38 of errors went undetected (not responded to
  by the flight crew)
• Improve ATC threat management
• 22 of ATC threats were mismanaged - linked to
  flight crew error

20
LOSA Case Study Two
Proactive safety change . Over one year, approx
182,500 flights, 4 rate 7,300 unstable much
better than 11 rate 20,075 unstable
21
Convergent Validity LOSA and ASAP
Well, James, very nice information. We see a
lot of the same things in our ASAP database
(voluntary incident reporting)
- Quote from an ASAP
Project manager

• Why would an airline do a LOSA when ASAP
  (incident reporting) captures the same things?
• ASAP collected when an event occurs
• LOSA collected independent of event
• By focusing on LOSA data implementing effective
  solutions, airlines can should be able to lower
  their potential for incidents / accidents

22
LOSA Operating Characteristics
23
Secret to LOSA Success
LOSA is not for all airlines - Its success
depends on pilot trust
Regulator
Check Airman
Nobody
Low pilot trust Low quality data because
there will be no differentiation between LOSA and
proficiency checks by regulators and check airmen
24
LOSA How to Gain Pilot Trust
LOSA is defined by 10 operating characteristics

• Jumpseat observations during normal operations
• Anonymous, confidential, and non-punitive data
  collection
• Voluntary crew participation
• Trusted and trained observers
• Joint management / union sponsorship

• Systematic observation instrument
• Secure data collection repository
• Data verification roundtables
• Data-derived targets for enhancement
• Feedback of results to line pilots

25
LOSA Operating Characteristics

• Jumpseat observations during normal operations
• Routine flights only
• No line checks or training flights
• Anonymous, confidential, and non-punitive data
  collection
• No names, flight numbers, or other identifying
  information
• Observer identity kept anonymous
• Data only used for safety purposes no
  disciplinary action
• Voluntary crew participation
• Flight crews have the right to decline a LOSA
  observation
• Typical denial rate is very low 1 per 100
  flights

26
LOSA Operating Characteristics

• Trusted and trained observers
• Trust
• Selection process management / union list of
  candidates
• Mostly regular pilots - Captains and First
  Officers
• Training
• Five days ground school (2), test observations
  (2) recalibration (1)
• Curriculum
• LOSA protocol
• How to write quality narratives
• TEM recognition, recording and coding

27
LOSA Operating Characteristics

• Joint management / union sponsorship
• Steering committee Flight Ops, Training, Safety
  and Union
• Symbolized with a signed agreement and sent to
  all pilots
• Systematic observation instrument TEM based
• Observers only record the TEM events they see
  no judgments
• Secure data collection repository
• Third party or pilot association gate keeper
• Pilots must believe that observations will not be
  misplaced

28
LOSA Operating Characteristics

• Data verification roundtables
• TEM data checked for coding accuracy and
  consistency with SOP
• On completion, data analysis begins
• Data-derived targets for enhancement
• Serve as benchmarks for organizational change
• Measure, change, measure again approach
• Feedback of results to line pilots
• Results and information on how airline management
  intends to respond with organizational change

29
LOSA Defined

• To ensure standardization, LOSA must have all ten
  operating characteristics
• If less than ten, the project needs another
  acronym
• Endorsed by
• ICAO
• FAA (pending advisory circular)
• IATA
• UT
• TLC
• IFALPA
• US ALPA

30
Frequently Asked Questions
31
LOSA Cost?

• Depends on a number of factors indirect and
  direct costs
• Number of observations to be collected
• Number of external and internal observers
• Depth of data analysis
• Insurance discounts?
• External provider fee LOSA Collaborative and
  others
• Some examples of misleading efforts to lower
  costs
• Observations only no roundtables or data
  analysis
• Operating crew as their own observers / self
  report
• Video in the cockpit / analyzed by one retired
  pilot in the airline
• These projects MIGHT have value but they are not
  LOSA

32
Length and Size of LOSA?

• Project Length
• Average 8 months from planning to a final report
• Airline 75 of time spent on planning
• LOSA Collaborative
• 30 of time spent on planning, observer training,
  data collection
• 70 spent on data verification, analysis and
  final report
• Project Size Want to extrapolate to everyday
  operations
• Weighted sample to match daily departure rates
  per fleet
• Minimum number of observations per fleet 40
  observations
• Cap observers to no more than 15 observations to
  prevent observer fatigue

33
Continuous or Snapshot LOSA?

• Recommended Snapshot approach with a sample of
  observations
• Continuous LOSA more cost, questionable value
  over snapshot LOSA
• A snapshot approach is more manageable
• There is a start and end for data collection that
  marks a point in time to derive targets and get
  to the real work of making changes
• Snapshots can complement other continuous safety
  efforts such as QAR or incident reporting

34
How Realistic is LOSA Data?

• Doesnt observers change the way flight crews
  behave?
• If they do, then LOSA results are
  underestimates of what actually occurs in
  normal operations
• If pilots fake good in front an observer,
  explain.
• Average three errors per flight 25 are
  mismanaged
• Average 32 of flights have an undesired aircraft
  state
• Common to observe substandard CRM or procedural
  violations
• Pilots calling girlfriends during taxi-out
• Offering cigarettes to LOSA observers in the
  cockpit

35
What about observer bias?

• Major sources of observer bias
• Sensitivity Observers are more sensitive to
  certain areas of flight operations (ex, human
  factor experts that only see human factor
  problems)
• Proficiency - Lack of TEM knowledge to correctly
  collect data
• Fatigue Observer burnout less data collected
  less precise
• Some data quality controls
• Use many different observers (Captains, First
  Officers, instructors, external)
• Cap the number of observations per observer
• Focus observers on gathering just the facts
  no judgment of causes
• Test for TEM knowledge during training backed up
  with data verification

36
Biggest Threats to LOSA Success?

• LOSA mutations and people calling or thinking its
  LOSA
• Senior management not briefed and/or prepared for
  the possibility of undesirable results
• Results not communicated to pilots affects
  pilot acceptance for the second LOSA
• Unmotivated observers
• No safety change process in place to act on LOSA
  findings

37
More Information about LOSA?

• ICAO LOSA Handbook Document 9803
• ICAO Journal LOSA Special Edition (vol. 57,
  no.4, 2002)
• UT website (www.psy.utexas.edu/HumanFactors)
• FAA LOSA Advisory Circular (Draft) Pending 2006
• Klinect Ph.D. dissertation - Academic
  demonstration of LOSA (available on request by
  e-mail)
• Research papers on TEM and LOSA

38
Audience Question and Answer