CONOPS Elements

1
Air Education and Training Command
Sustaining the Combat Capability of Americas Air
Force
Test of Basic Aviation Skills (TBAS)
Implementation Status
2nd Lt Kirk Reimer AETC SAS/CS 29 Nov 05
I n t e g r i t y - S e r v i c e - E x c e l
l e n c e
2
Overview

• Risks to TBAS Implementation plan and follow-on
  PCSM (TBAS) operation
• Program management tool to identify/address all
  programmatic risks

3
Plan

• Hardware received Oct 05 - Nov 05
• System config/assembly Nov 05 - Apr 06
• Begin loading s/w on systems Jan 06
• TBAS shipped to sites May 06 - Jun 06
• BAT systems deactivated 17 Jul 06
• TBAS set-up at sites 17 Jul - 13 Aug 06
• TBAS operational at all sites 14 Aug 06
• Boards stop using BAT scores 30 Sep 06

4
Risk 1

• Risk TBAS scores add no incremental significance
  to that of the AFOQT / flying experience baseline
• Impact Inclusion of too many subtest scores
  potentially degrades PCSM model predictive
  ability Drives other efforts (e.g., equating
  studies involving BAT TBAS, experimental
  operational TBAS) that also must be accomplished
  prior to operational implementation Determines
  TBAS test battery content and administration
  order
• Plan Ensure significance of subtest in scoring
  algorithm only in presence of AFOQT and flying
  experience.
• POC 1Lt Nathan Murphy / 1Lt Caleb Earnest

5
Risk 2

• Risk TBAS test taking instructions may be vague
  or misleading
• Impact Candidates may misunderstand
  instructions resulting in scores not reflecting
  their true abilities
• Plan Request Dr Carretta and Dr Ree review the
  test instructions
• POC 2Lt Kirk Reimer

6
Risk 3

• Risk TBAS tests may demonstrate a bias in either
  gender or ethnicity
• Impact PCSM test scores favor certain
  candidates resulting in lawsuits or Congressional
  inquiries
• Plan
• Recommend no minimum for consideration on board
  until enough data is gathered on minority groups
  to test bias?
• Determine collection goals for conducting valid
  bias tests, collect the data, continue testing
  during TBAS operation, update rules as necessary
• Explore equating strategy between BAT and TBAS to
  establish continuity of PCSM database (1993)
• Allows historical (trend) analyses
• Allows us to draw on BAT-based PCSM to address
  subgroup issues (e.g., adverse impact, predictive
  bias)
• Difficulty arises in lack of identical subtest
  continuity---overcome with factor analysis?

7
Risk 3 (Cont)

• Plan
• After deciding on TBAS-based PCSM model,
  re-accomplish BAT-TBAS equating on larger
  existing database
• POC 1Lt Nathan Murphy / SSgt Burkhardt

8
Risk 4

• Risk PCSM-TBAS scoring algorithm (based on
  pre-operational TBAS test) may assign regression
  weights which incorrect for operational TBAS
• Impact Operational PCSM-TBAS give candidates
  scores which are inconsistent with their actual
  test performance which may lead to protest
• Plan Conduct comparison between pre-operational
  and operational TBAS so operational scores can
  use scoring equation (regression weights)
  developed using experimental TBAS
• POC SSgt Burkhardt / 1Lt Nathan Murphy

9
Risk 5

• Risk Future technical support or upgrades may be
  designed for wrong equipment/software
• Impact May lead to upgrade failures or extend
  technical support lead times
• Plan Document TBAS hardware and software
• POC SSgt Burkhardt / Datavise

10
Risk 6

• Risk Testing at different locations may not be
  standardized
• Impact Candidate scores may be unfairly
  helped/hurt by testing application resulting in
  lawsuits/Congressional inquires
• Plan
• Develop TBAS test administration manual to ensure
  standardized test administration procedures
  including instructions to applicants, data
  collection and processing, recovery from system
  failure, etc.
• Develop TBAS Information Pamphlet analogous to
  AFOQT Information Pamphlet, this would provide
  applicants with information about TBAS subtests,
  application procedures, etc.
• POC Datavise

11
Risk 7

• Risk Test-Retest policy may give unfair
  advantage to candidates who take the test twice
• Impact Candidates not taking the test twice may
  bring lawsuits or Congressional inquires
• Plan
• When BAT was implemented operationally, no
  retests were allowed
• AFRL study showed a 6 month retest interval was
  acceptable policy changed to be consistent with
  AFOQT (6 month retest interval)
• Study whether TBAS retests should be permitted
• Analyze existing TBAS retest results to see if
  learning occurred at various intervals keep
  existing 6-month rule unless analysis indicates
  otherwise
• TBAS subtests are less reliable (shorter?) than
  BAT subtests may affect retest performance
• Consider creating TBAS composites (e.g., ATT
  HTT ATTHTT) or lengthening subtests to increase
  score reliability
• POC 2Lt Kirk Reimer

12
Risk 8

• Risk PCSM TBAS test may not be in compliance
  with APA standards
• Impact Candidates may bring lawsuits or
  Congressional inquires on basis of non-compliance
• Plan
• TBAS Test Manual must be completed or under
  development
• Subgroup norms including information about
  adverse impact and predictive bias
• Partially addressed by minority group analysis
  plan
• POC 1Lt Nathan Murphy / 1Lt Caleb Earnest

13
Risk 9

• Risk TBAS test systems may be vulnerable to
  candidate test manipulation
• Impact Candidates may hack or otherwise
  manipulate TBAS test station to influence their
  TBAS scores
• Plan Lock-down the TBAS system so only
  essential test functions can perform and no
  function outside of the test can be performed
• POC SSgt Burkhardt

14
Risk ?

• Document TBAS development process
• Establish a continuing process of IOTE
  throughout life of TBAS
• Establish a plan to develop the follow-on to
  TBAS put on a 7-8 year development cycle similar
  to AFOQT new form development
• Retain BAT software for possible future use

15
Replenishing the Combat Capability of Americas
Air Force
I n t e g r i t y - S e r v i c e - E x c e l
l e n c e