CAOC Performance Assessment System

1
CAOC Performance Assessment SystemCPASInforma
tion Age Metrics Working Group26 October 2005

• JOHNS  HOPKINS U   N   I  V  E   R 
  S   I  T  Y
• Applied Physics Laboratory
• Precision Engagement - Operational Command
  Control
• Technical Manager Glenn Conrad [phone]
  glenn.conrad_at_jhuapl.edu
• Project Manager F.T. Case [phone]
  ft.case_at_jhuapl.edu

2
Background

• CAOC performance assessment system (CPAS) project
  evolved from an the FY05 C2ISRC-sponsored project
  DeCAIF.
• 505th OS leadership identified the need for an
  ACMI-like capability for the CAOC-N instructor
  cadre. The system should be capable of
• Tracking targets through the F2T2EA kill chain
• Highlighting procedural errors that occur during
  execution
• Automatically generating the CAOC-Ns debriefing
  shot sheet
• C2 Battle Lab leadership approved APLs proposed
  CPAS project as an initiative in Jan 05.
• APL was tasked to develop and demonstrate a
  prototype CAOC instrumentation capability that
  would support TST operations at the CAOC-N and
  the TBONE initiative.
• APL worked with CAOC-N subject matter experts
  (SMEs) on the spiral development of CPAS from Feb
  05 Aug 05 to establish the required info items
  and the look feel of how CPAS would display
  them.
• APL engineers demonstrated the functional CPAS
  X-1 prototype at CAOC-N during RED FLAG 05-4.2
  (mid-Aug 05) and the X-2 prototype at the C2
  Transformation Center during TBONE LOE5 (JEFX 06
  Spiral 0) in Sep 05.

3
CAOC PAS
Demonstrates a prototype ACMI-like
instrumentation capability for Falconer
instructor cadre. Enables near-real-time,
end-to-end performance assessment and
reconstruction of TST operations to support
performance assessment.
CAOC PERFORMANCE ASSESSMENT SYSTEM (CPAS)

• CAOC PAS provides a means to collect and store
  time-stamped information in near real-time during
  TST operations.
• Develops and refines filters and algorithms that
  provide key information items required for
  Falconer crewmember situational awareness and
  performance assessment
• Provides an end-user defined, near real-time
  presentation of time-lined activity information
  for presentation to individuals and groups during
  post-mission operations

4
Project Objective

• Demonstrate a capability to reconstruct and
  display events and decisions that occur within a
  TST cell to support post-mission assessment.

• GOALS
• Develop a non-intrusive data capture
  (collection and archiving) capability for
  TST-related information at CAOC-N
• Develop a reconstruction capability that will
  portray event and decision activities that
  occurred during the execution of TST ops
• Develop an information injection capability that
  enables an instructor or assessor to input
  comments/observations into the data archive
• Implement the CAOC Assessment System capability
  at CAOC-N and demonstrate it during Red Flag
  05-4.2 (Sep 2005)
• Develop initial capability to work with the C2B
  TBONE initiative and CAOC processes using TBONE

5
CPAS Operational Architecture for CAOC-N
Individual Presentation
NELLIS Range Operations
CPAS Algorithms
CPAS DR
CPAS Presentation Display
CAOC-N network
O-O-D-A
Instructor Injection Capability
Group After Action Review (AAR)
6
CPAS Prototype Software

• Developed in Microsoft Visual Studio (IDE) using
  Microsoft .NET Software Framework
• 20K lines of C and C code Runtime
  executable 2 Mb
• Built on top of MS Access (DBMS)
• 3 logical constructs of software collectors,
  algorithms, user interface
• Layered software architecture for both CAOC-N and
  TBONE

CAOC-N (X-1)
TBONE LOE 5 (X-2)
7
CPAS Functionality

• Creates and Exports Summary of Target Activity

• Maintains Status Monitor of Targets in Near
  Real-Time

• Summary of Procedural Errors
• ADOCS Status Light Slide Bar
• ADOCS Remarks Search Engine

• Drill-down to F2T2EA Details for any Target

• Elementary search engine for IM chat dialogue

• Calculates TST measures and metrics of interest

8
CPAS X-1 Demo
9
Next Steps in CPAS Evolution

• APL Focus
• Develop additional collectors and
  correlation/fusion algorithms for integration
  into CPAS X-1 and/or X-2 framework(s)
• CAOC-N Focus
• Employ CPAS X-1 at CAOC-N during AWC sponsored
  exercises and training events
• Develop and deliver CPAS V-1.0 capability to
  CAOC-N and evolve V-1 as required
• AFRL/HE Focus
• Incorporate the CPAS X-1 data collection
  instrument with AFRL/HEs (Mesa) subjective
  evaluation capability (Performance Measurement
  System) to provide an enhanced AOC Performance
  Measurement System
• AFEO Focus
• Evolve and employ CPAS X-2 during JEFX Spiral
  2/3/MainEx in support of AFEO assessment team
• Develop and deliver CPAS V-2.0 capability for use
  by AFEO assessment team during JEFX 2006 MainEx
• Work with JFIIT to integrate CPAS V-2.0 with WAM
  to provide an enhanced data collection and
  presentation capability for use during JEFX 2006
  MainEx

10
BACKUPSLIDES
11
Why a CAOC Assessment System ?

• The Problem There is no efficient means to
  collect, store, correlate, and retrieve key
  information items needed to reconstruct a time
  history of activities and events that occur
  during real-time CAOC operations to answer the
  question What happened ?
• The Need (C2-C-N-06-05 C2-A-N-02-03)
• A means to capture and archive time-stamped
  information from multiple CAOC operational
  sources,
• Algorithms to support correlation of events, and
• A means to visually portray both activities
  performed and decisions made for both operators
  and instructors alike.
• The Payoff
• Enable experts to assess what happened during
  training and operational sessions,
• Provide the information required to improve
  process shortfalls, identify training needs, and
  to highlight the things that are being done right
• Support archiving for future after-action review
  (AAR)

12
CPAS Spiral Development

• CAOC-N DR
• CAOC-N Algorithms
• CAOC-N Presentation
• WEEMC Data

Feb 05
KA with CAOC-N Instructor Cadre

• Robust DR
• Final Presentation
• Procedural Algorithms

TBONE Functional Demo

• Prototype Screens

CAOC-N Functional Demo
Sep 05
KA on Presentation Prototype
Mar 05
Aug 05
TBONE Interface Research
Aug 05

• Updated Screens
• Prototype DR

Refine Presentation Prototype
CAOC-N Integration Test 2 (SP2A)
May 05
Aug 05

• Updated DR
• Refine Presentation
• Initial Algorithms

• Completed DR
• Refine Presentation

13
TST Process _at_ CAOC - N

1. White Force (instructors) provide initial input
   to TST cell on potential target
2. Potential target info entered into ADOCS TDN
   manager and Intell develops target
3. Target is classified as a TST and entered into
   ADOCS TDL list
4. Targeting data is developed for target folder by
   Intel and targeting
5. Attack Coordinator searches ATO for available
   weapon platforms to be tasked to engage TST.
6. Nominations are provided to TST cell chief who
   approves pairing.
7. Positive target ID is accomplished by the Intell
   cell and SODO approves tasking order.
8. Tasking order (15-line text msg) is drafted by
   C2DO GTC and transmitted via Link-16 or voice
   to controller (AWACS)
9. AWACS acknowledges receipt and passes info to
   weapon platform who accepts or rejects tasking.
10. Acknowledgment is provided to C2DO with estimated
    TOT from the weapon platform
11. Target is attacked
12. Weapon platform provides MISREP to AWACS from
    fighter with actual TOT and BDA (if available).
13. Target kill chain is complete

CCO
White Force
SIDO
SADO
SODO
SOLE
NALE
MARLO
BCD
TST Ch
GTC
ACF Analysis Coord Fusion
ATK COORD
C2DO
Target Duty OFC (TDO)
IDO ISR SCIF
SIGINT
IMINT
MASINT
Wpneer
Tgteer
14
Reactive Targeting Coordination Card
15
CPAS Near Real Time Status Display
ADOCS
CPAS
16
Generates Summary of Process Errors
17
Drill-down to Kill-Chain Details for any Target
18
Create Shot Sheet Summary of Target Activity
and Export to Excel Spreadsheet
19
IM Chat Correlation and Search

• Correlates milestone events with IM chat dialogue

20
Chat Log Search and Catalogue
21
Measures Metrics

• These were measures of interest for CAOC-N
  instructors at RED FLAG 05-4.2
• Number of targets prosecuted listed by their
  priority number
• Total number of targets prosecuted
• Average time of target prosecution
• Average time a target stayed in each kill chain
  segment

22
TST Measures of Interest

• CPAS X-1 can calculate 34 of these measures of
  interest
• Green can do now (12)
• Yellow could do (7)
• White harder to do - need additional collectors
  (15)
• Red cant do (1)

e-mail correspondence - Bill Euker to F.T. Case
- Friday, September 16, 2005 433 PM
23
CPAS Calculation of the TST MOI
23
14
6
25
24
21
21
22
26
35
9
8
24
CPAS Data Collection Sources

• During JEFX 2002, the following systems in the
  TCT section, on the Combat Ops floor, and on the
  Nellis ranges had the capability to automatically
  record time tagged information for post
  processing
• ADSI. Records link data received at the ADSI.
  Data is written to JAZZ disks for replay on ADSI.
• Sheltered JTIDS System (SJS). Records link data
  received and transmitted at the SJS. Data will
  be translated into ASCII format and provided on a
  CR-ROM on a daily basis.
• Joint Services Workstation (JSWS). Records data
  received and displayed on the JSWS. Data is
  written to 8mm DAT tapes for replay on JSWS.
• Information Workspace (IWS).
• IWS voice will be recorded using VCRS connected
  to primary TCT nodes on the AOC floor.
• IWS text chat will be logged and downloaded at
  the end of each day.
• IWS peg-board activity will be manually captured
  using screen capture and stored for post-event
  analysis.
• Digital Collection Analysis and Review System
  (DCARS). DCARS will record all simulation
  traffic including ground game target locations.
• Tip OFF. A laptop loaded with Tip Off software
  will be connected to the TRS in the JICO cell and
  will log all incoming TIBS and TDDS traffic
  received at that node.
• GALE. The GALE system used by the Threat Analyst
  (SIGINT Tracker) will log all incoming SIGINT
  data and provide post event hard and soft copy
  detection and convolution data.
• WebTAS. WebTAS can be interfaced with the TBMCS
  SAA system and record all track information
  received and displayed on the SAA. Data is
  translated into ASCII format for inclusion into
  the master JEFX 02 TCT assessment database.

From AFC2TIG, JEFX 02 TIME CRITICAL TARGETING
DATA MANAGEMENT AND ANALYSIS PLAN, July 2002,
para 4.2, pg 11..
25
Future Focus Areas

• Automate scenario selection/tailored training
• Scenarios designed to address specific individual
  and team objectives based on performance
• Warehoused vignettes based on desired knowledge,
  skills, and experiences
• Make real-time interventions during training
  event via coaching, tutoring, scenario changes
• Part-Task Training capability for
  planning
• AOC Brief/Debrief System

26
AOC Performance Measurement
27
AOC Performance Measurement

• Research objective
• Design and validate measurement methods,
  criteria, and tools
• Importance to the warfighter
• Enables performance-based personnel readiness
  assessment not box-checking
• Quantitative Qualitative assessment of training
  effectiveness
• Status
• Testing at CAOC-N (Jun 05)
• Approved Defense Technology Objective (HS.52)