Raters Training as a Safety Concern

1
Raters Training as a Safety Concern
Associate Member

• Sergey MELNICHENKO
• CompLang Aviation Training Centre (Moscow),
• Deputy Director

ICAO LPR Implementation Workshop Russia,
St-Petersburg, May 24-26, 2011
2
Raters Training as a Safety Concern
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• - RATER AS A DECISION MAKER
• - SELECTION OF PERSONNEL
• - AB INITIO TRAINING
• - RATERS CERTIFICATION
• - INTER-RATER RELIABILITY
• - INTRA-RATER RELIABILITY
• - RASCH ANALYSIS
• - RECURRENT TRAINING
• - EVALUATION OF RATERS
• ADEQUACY

3
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
4
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Aviation English test results are felt around
the globe
Raters Training as a Safety Concern
5
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Inadequacy in Level Evaluation Level 3 instead
of Level 4 Loss of job and/or
occupation Financial losses for AO or
ANSP Market losses Redistribution of air
transportation services Lack of
confidence for test-takers
Raters Training as a Safety Concern
6
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Inadequacy in Level Evaluation Level 4 instead
of Level 3 Aviation safety is
compromised Degradation to Level
2 Overconfidence for test-takers
Raters Training as a Safety Concern
7
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
8
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
ICAO recommendations on interlocutors and raters
qualifications
Raters Training as a Safety Concern
9
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
ICAO recommendations on interlocutors and raters
qualifications (Doc 9835) Familiarity with ICAO
documentation Annex 1 Doc 9835 ICAO Rated
Speech Samples CD ICAO Circ 318
Raters Training as a Safety Concern
10
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
ICAO recommendations on interlocutors and raters
qualifications (Doc 9835) Familiarity with ICAO
documentation Operational and/or language
specialist expertise Level 5 or
Level 6 proficiency Familiarity with Aviation
English RATERS CODE OF ETHICS
Raters Training as a Safety Concern
11
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
12
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• Theory of language testing
• ICAO documents Annex 1, Annex 10, Doc 9432, Do?
  9835, Circ 318
• The ICAO Proficiency Rating Scale and holistic
  descriptors
• Correlating speech samples with the ICAO LPR
  Scale

Raters Training as a Safety Concern
13
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Raters training course should be aimed at
experienced non-native and native English
language teaching professionals and
subject-matter experts who are currently teaching
Aviation English and who want to be able to rate
according to the ICAO Language Proficiency Scale
Raters Training as a Safety Concern
14
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
15
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• Certification by national Civil Aviation
  Authority or by international aviation
  organization is mandatory

Raters Training as a Safety Concern
16
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• Is geographical principle necessary?

Raters Training as a Safety Concern
17
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• Are quotas necessary?

Raters Training as a Safety Concern
18
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
19
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Different raters but same results
Raters Training as a Safety Concern
20
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
21
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Different times but same results
Raters Training as a Safety Concern
22
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
23
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Evidence of the validity of the responses
Raters Training as a Safety Concern
24
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Evidence of the validity of the
responses Correctional refresher course
Raters Training as a Safety Concern
25
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

Evidence of the validity of the
responses Correctional refresher
course Administrative measures to marginal
raters
Raters Training as a Safety Concern
26
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
27
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Annual refresher course for raters
Raters Training as a Safety Concern
28
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Annual refresher course for raters New materials

Raters Training as a Safety Concern
29
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

Annual refresher course for raters New materials
Practical training
Raters Training as a Safety Concern
30
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011

• RATER AS A DECISION MAKER
• SELECTION OF PERSONNEL
• AB INITIO TRAINING
• RATERS CERTIFICATION
• INTER-RATER RELIABILITY
• INTRA-RATER RELIABILITY
• RASCH ANALYSIS
• RECURRENT TRAINING
• EVALUATION OF RATERS ADEQUACY

Raters Training as a Safety Concern
31
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Rater is not a lifetime profession RATER
TRAINING IS A SAFETY CONCERN
Raters Training as a Safety Concern
32
Associate Member
ICAO LPRI Workshop, St-Petersburg, Russia, 2011
Thank you
Raters Training as a Safety Concern