Course Schedule (con

1

• Course Schedule (cont)
• two
• Principles of Testing
• Principles of Assessment
• Conducting a PPC
• PPC Tolerances

2

• Course Schedule
• two
• Principles of Testing
• Principles of Assessment
• Conducting a PPC
• PPC Tolerances

3
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
4
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
5
Principles of Testing
General Principles A good test should be 1)
Valid 2) Reliable 3) Comprehensive 4)
Discriminating 5) Objective
6
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Validity
7
Validity
Does test measure what it is supposed to
measure? 1) make the test realistic 2) make
the test complete 3) apply standards
carefully 4) no test of items outside std
8
Validity Example
Which is the most valid? 1) describe action in
briefing 2) simulate action in aircraft 3) do
action in aircraft 4) do action in simulator
9
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Reliability
10
Reliability
How accurately does it measure what it
measures? Must be consistent 1) Avoid
extremes - distractions - aircraft -
weather 2) Plan compensations (eg. V1) -
beforehand - carefully
11
Reliability Example
Which test is most reliable? 1) PPC in
marginal VFR weather 2) PPC in good weather,
light winds 3) PPC in good weather, strong
winds 4) PPC in IFR weather near minima
12
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Comprehensive
13
Comprehensiveness
The test must include all the relevant
sequences - important "need to know"
info - do not focus on details - vary the
program from test to test - change approaches,
sequences, emergencies (where discretion)
14
Comprehensiveness Example
Which test is most comprehensive? 1)
departure, steep turns, hold, NDB, ILS 2) dep,
hold, ILS, ILS, VOR 3) dep, hold, ILS, VOR 4)
dep, steep turns, LOC (BC), PAR
15
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Discriminating
16
Discrimination
Detects different levels of achievement -
separate qualified from unqualified - this is
main goal of test - rank those who are
qualified - use overall performance for
borderline cases
17
Discrimination Example
Which test is most discriminating? 1) no
oral test before flight 2) oral test before
flight with general briefing 3) oral
test and specific test sequences 4) oral
test before and during flight
18
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Objective
19
Objectivity
Test outcome must not be affected by personal
examiner 1) do not impose your own
standards 2) adhere to standards in FM, SOP, AOM
and DCP Manual 3) do not let rating errors
affect your assessment 4) carefully
record events as they occur
20
Objectivity Example
Which written assessment is most
objective? 1) poor judgment during
emergency 2) excessive swing during engine
fail 3) altitude 100ft low inbd to beacon 4)
airspeed unstable on final
21
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
22
Common Rating Errors
1) Error of Standards 2) Error of Central
Tendency 3) Error of Halo 4) Error of
Logic 5) Error of Familiarity 6) Error of
Narrow Criterion
23
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of central tendancy
24
Error of Central Tendency
Rate of performance at centre of scale Hesitant
to give excellent or poor ratings Affects all
assessments by this observer Most common with
inexperienced observers
25
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of Standards
26
Error of Standards
Use personal standards to rate others If good at
it - rate harder If poor at it - rate
easier Affects all assessments by this observer
27
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of Halo
28
Error of halo
Allow general impression to affect rating Can be
favourable or unfavourable Affects only certain
persons rated Can include stereotypes
29
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of Logic
30
Error of Logic
When several similar traits Traits are related in
mind of observer Could relate engine fail at V1
to SE landing Observer generally not aware of
error
31
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of Familiarity
32
Error of Familiarity

• Grading objectivity weakened
• Accustomed to weaknesses and eccentricities
• Occurs over long period of time
• Stop and regain perspective

33
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
Error of Narrow Criterion
34
Error of Narrow Criterion

• Related to assigning rating
• Often occurs in assessing two-crew or multi-crew
  assessments
• One who does the best of perhaps a marginal
  performance obtains a high rating

35
Principles of Testing
Outline General Principles Common Rating
Errors Self-study handout
36

• Course Schedule
• day 2
• Principles of Testing
• Principles of Assessment
• Conducting a PPC
• PPC Tolerances

37
Principles of Assessment
Outline PPC general guidelines Assessment
format Overall assessment
38
Principles of Assessment
Outline PPC general guidelines Assessment
format Overall assessment
39
PPC general guidelines
1) Take note soon after event 2) Terminate test
if failure 3) Allowance for turbulence 4) No
advice, criticisms, training or
demonstrations 5) Create realistic scenarios 6)
Do not simulate multiple, unrelated failures
40
PPC general guidelines (cont)
7) PPC - Use Designated Check Pilot Manual 8)
Simulated weather at or below minimums 9)
Need a "U" to fail 10) Take weather info into
account in scenarios 11) PNF responsible if
gross deviations, violations
41
Performance Assessment
According to the following publications DCP
Manual PPC - AFM - AOM, Pilot Operating
Handbook - Operations Manual - SOP
Manual IFT - Instrument Procedures
Manual - Approach Procedures (Jepp) -
Company Procedures
42
Principles of Assessment
Outline PPC general guidelines Assessment
format Overall assessment
43
PPC Assessment Format
S exercise flown within acceptable
tolerances and procedures correct and
effective SB a brief excursion from the criteria
or a deviation from standard practices. Correc
ted by acknowledging the deviation during
de-briefing or regaining the criteria during
flight U gross excursions from acceptable
tolerances or a violation of airspace or
clearance or a fault in procedure that could
create an unsafe or ineffective situation
44
PPC Assessment Format
S - Minor errors only - Airspeed and
altitude control acceptable - Handling and
knowledge are acceptable and safe considering
the experience of the candidate
45
PPC Assessment Format
U - Endanger aircraft, passenger, crew - It
results in a crash - Multiple errors made in
one exercise - Violates ATC clearance or an
altitude - Major deviation from standard
procedures or practices or safety was
jeopardized - Candidate required continued
prompting or help from other candidate - It
exceeds aircraft limitations or - Candidates
demonstrates unsatisfactory knowledge of
systems, equipment, or procedures
46
PPC Assessment Format
SB - Aircraft handling and knowledge are safe
but of a lower standard that can be corrected by
briefing - Brief excursion from tolerances but
corrected promptly - Deviated from SOP but
did not create a more hazardous situation and is
clarified by candidate - Candidate experienced
some difficulty or required slight prompting to
accomplish the task
47
Principles of Assessment
Outline PPC general guidelines Assessment
format Overall assessment
48
Overall Performance
is it safe and effective? are criteria
broken? are any fail points done? is crew
concept used? are crew in control of
situation? are procedues known and executed?
are limitations adhered to? does an error
create an unsafe situation? are errors
recognized and corrected? would you send your
family with this pilot?
49

• Course Schedule
• day 2
• Principles of Testing
• Principles of Assessment
• Conducting a PPC
• PPC/IFT Tolerances

50
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios Safe checking practices Post-fl
ight briefing Written comments
51
Conducting a PPC
Outline Flight test requirements PPC Pre-fl
ight preparation/briefing Flight test
scenarios Safe checking practices Post-flight
briefing Written comments
52
PPC Requirements
PRE-FLIGHT - Flt planning - weather, fuel,
NOTAMs, WB - IFR x-country exercise for
initials - Cockpit checks and ident of
NAVAIDS - Read back and interpretation of an
ATC clearance - Take-off and passenger
briefings DEPARTURE AND ENROUTE - Transition
to instruments - Climb and level flight - Use
of NAVAIDS and adherence to ATC clearance
53
PPC Requirements (cont)
AIRWORK - Applicant Should Perform Medium
and steep turns Hold Tracking on VOR and
NDB Failure of AI and DI - timed turns,
SL - climbs and descents - optional for
renewals
54
PPC Requirements (cont)
TERMINAL PROCEDURES Transition to approach
should be flown At least 2 approaches to
minima - ILS and NPA required for
initials - different approaches if
possible - NDB recommended Missed
approach Circling (optional) Landing from
an instrument approach EMERGENCIES One
asymmetric approach should be done No
multiple unrelated failures
55
PPC Additional Requirements
Typical Large Type Aircraft 2 T/O's 2
landings (one asymmetric) asymmetric
instrument approach engine fail after T/O
RTO (mandatory in SIM) engine fail on
rejected ldg/MAP circling if company
limitslt1000/3 refer to DCP Manual Chap
7 Schedule I
56
PPC Additional Requirements
Typical Small Type Aircraft 2 T/O's 2
landings (one assymetric) steep turns
approach to stall - ldg and T/O RTO
(mandatory in SIM) refer to DCP Manual
Chap7 Schedule II
57
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios Safe checking practices Post-fl
ight briefing Written comments
58
Pre-Flight Preparation
Before meeting candidate - check weather -
applicant's info - plan IFR x-country
exercise - plan oral questions After meeting
candidate - ensure admission req'ts met -
ensure aircraft is ready for IFR flight - check
accuracy of log, flt plan, WB - test knowledge
of weather and NOTAMs - test IFR knowledge and
aircraft limitations
59
Oral Questions
Valuable part of any flight test Can be used
to identify changes Can provide feedback on
weak training areas Ask "why" questions Cover
only "need-to-know" material Vary questions
from test to test
60
Aircraft Limitations
"Need to know" a/c speeds if not familiar with
type Vref Vr Hold Vmca Engine out
approach Use these briefed speeds to assess
speed control
61
Aircraft IFR Requirements
( general aviation requirements)
- Radio, 2 nav radios, aircraft instruments - C
of A, C of R, - Journey log book - Compass swing
(12 months) - Static pressure check (24 months) -
Altimeter certified (24 months) - Weight and
Balance report - Radio station license
62
Pre-Flight Briefing
private location, calm atmosphere entire
flight is IMC at minimum clear who is PIC
description of major test components
responsibility for radios and clncs cockpit
checks to be done per SOP crew responsibilities
and coordination emergencies - use of
"SIMULATED" how engine failure to be
simulated "FIELD IN SIGHT" or overshoot
candidates QUESTIONS
63
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios - Exercise Safe checking
practices Post-flight briefing Written
comments
64
Flight Test Scenarios
Make efficient use of time Create realistic
scenarios Liaise with ATC early No
overloading Minimize re-positioning in Sim
Use proper ATC terminology
65
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios Safe checking practices Post-fl
ight briefing Written comments
66
Safe Checking Practices
Check Pilot assuming crew duties RTO
Engine fail after V1 Rejected landing with
engine fail Engine out landing Approach to
stall
67
De-Brief After Flight Test
Advise points in order Include location,
atmosphere, tone Quiet location no distraction
68
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios Safe checking practices Post-fl
ight briefing Written comments
69
Post-Flight Briefing
Pass advise applicant of outcome
establish a positive atmosphere
highlight above-average performance
de-brief as per written notes identity
major (possibly minor) errors
Fail advise applicant of outcome
establish a positive atmosphere
highlight above average performance
explain reasons for failure
de-brief other major errors advise
regarding training and re-test
70
Conducting a PPC
Outline Flight test requirements -
PPC Pre-flight preparation/briefing Flight
test scenarios Safe checking practices Post-fl
ight briefing Written comments
71
Written Comments
Should be factual Clear and concise Must
support assessment Required for "U" and "SB"
No sermons or teaching Relate assessments to
PPC tolerances
72

• Course Schedule
• day 2
• Principles of Testing
• Principles of Assessment
• Conducting a PPC
• PPC Tolerances

73
PPC Tolerances
Outline Statistics DCP Manual PPC fail
points DCP Manual tolerances 6.8.8
74
PPC/IFT Tolerances
Outline Statistics ACP Manual IFT fail
points PPC fail points ACP Manual tolerances
75
IFT Failure Items
RENEWAL NDB 28 HOLD 23 ILS
13 TRANSITION 8 LOC 5 VOR
5 DEPARTURE 5
INITIAL NDB 29 HOLD 26 ILS
26 DEPARTURE 7 TRANSITION 6 FLT.
PLANNING 6 HANDLING 4
76
PPC Tolerances
Outline Statistics DCP Manual PPC fail
points DCP Manual tolerances
77
DCP Criteria
from DCP Manual 6.8.8 -assigned headings within
10 degrees -assigned tracks and bearings within
10 degrees -altitude within 100 ft except at MDA
when accurate altitude control is
required -airspeed within 10 knots for holding,
approach, and missed approachand -not more than
half scale deflection, as appropriate to aircraft
type, of course deviation indicators during
instrument approaches -assumes no unusual
circumstances wx,turbulence, malfunction(s), and
type of approach may modify tolerances
78
fail Points
Outside DCP Manual, AFM, IFR Procedures No
check of flt instruments before flight No check
or incorrect use of icing systems No check of
electrical charging system Wrong altimeter
setting Vital actions checklist item missed
Non-compliance with ATC clearance No
communication with ATC when required No ident
or monitor of NAVAIDs Comm fail procedures
Inability to establish cleared holding pattern
79
Tolerance Example 1
Take-off
Loss of directional control Wrong corrective
action for swing Exceeding airframe or engine
limitations Rotation before Vr Failure to
comply with departure clearance Vital action
check missed
80
Tolerance Example 2
Intstrument Approaches
Loss of separation due to failure to follow
clearance No prompt corrective action when DCP
criteria exceeded Unable to complete a
successful landing Exceeding airframe or engine
limitations
81
Tolerance Example 3
Systems Malfunctions Vital action check
missed Missing a check which could create an
unsafe situation Exceeding airframe or
engine limitations Missed Approach Not
following published procedure or clearance
82
Questions?