George Tadda

1
INFORMATION FUSION FOR CYBER SITUATION AWARENESS

• George Tadda
• Fusion Technology Branch
• Information Directorate
• Air Force Research Laboratory
• E-mail george.tadda_at_rl.af.mil
• Phone [phone]

2
Outline

• Introduction
• Motivation
• Situation Awareness Reference Model
• Metrics
• Application of Lessons Learned

3
Work in Situation Awareness (SA)

• Used reference models to demonstrate/build
  prototype systems for
• Cyber (Defense Security (DS) 05, SIMA 05)
• Tactical (ISIF 02)
• Global (ISIF 04)
• Maritime
• and Many Others
• Developed Metrics (DS 04) to Evaluate Level 2
  Systems and applied them to Cyber (DS 05)
• After much discussion we questioned the
  difference between tracking objects and
  situations and whether the majority of the
  metrics are just another way to measure integrity
  of tracks
• Additional Activities
• Jean Roy, under The Technical Cooperative
  Program, presented a
• definition of situational analysis and included
  in "Concepts,
• Models, and Tools for Information Fusion
• Snidaro, M. Belluz, G. Foresti, Domain
  knowledge for
• security applications, ISIF07 defined types of
  events (simple,
• spatial, and transitive)
• Dale Lambert, formalizing situation awareness
  through mathematics

07-210
4
Motivation(Reality of Todays Environment)
Today WE Have
Objects
Tactical
Alerts
Cyber
Events
Global
and MORE
07-291
5
Motivation
(STEP 1 From Data -gt Complex Relations/Situation
(s))
(STEP 2 From Complex Relations/Situation(s) -gt
Anticipation)
07-291
6
Sharing the Stage (From A Model Perspective)
SITUATION AWARENESS
FUSION - TACTICAL

• Most popular is the Joint Directors of
  Laboratory (JDL) Model (Sensor-based)
• Functional Model
• 5 Levels (Level 0, 1, 2, 3, 4)
• Published By Llinas, Hall, White (1992)
• Most work concentrated on Level 0/1/4 (Dots on
  Map)
• Little definition of Level 2/3 (What do they
  mean?)
• Bottom-up, Data Driven

• Receiving Much Attention Today from the
  Cognitive Community
• Mental Model
• 3 Levels Perception, Comprehension,
  Projection
• Developed by M. Endsley (1995)
• Extended by McGuinness and Foy for Resolution
• Top Down, Goal Driven

07-291
7
Situation Awareness Reference Model (Combining
The Best Of Both Worlds)

• Based on JDL Endsleys Models
• - Plus Initial Data Requirement
• - Textual Inputs (Info Exploit)
• Define Problem/Goal Top Down
• - What/Where/Who
• Processing Flow ( )
• - Projection The Alert(s)
• - Comprehension
• -- Model Analysis
• - Perception
• -- Data Collection
• -- Parsing/Extraction
• -- Data Cleansing
• - JDL Level 0/1
• Process Refinement ( )
• - Missing Data

Level 0/1
Data
Parsing
Sources
Sources
Collection
Extraction
Perception
Data Cleansing
Data Requirements
Evidence
Model
Knowledge
Knowledge
Additional
Analysis
Discovery
Discovery
Info
Tools
Tools
Tools
Comprehension
Matches/ Partial Matches
Target
Missed Questions
Models
ANTICIPATION
Potential New Relationships
Tools
The Problem
Based on Model Unfolding
07-291
8
Situation Awareness Reference Model (Applied to
Cyber SA)
Open Source
Snort
Dragon
Host IDS
Network Stats
Web Logs
Sys Logs
Level 0/1
Post Proc
Data
Parsing
Data
Parsing
Perception
Collection
Extraction
Sources
Sources
Collection
Extraction
Perception
Data Cleansing
Data Cleansing
Data Requirements
Evidence (Alerts)
Evidence
Model
Knowledge
Knowledge
Additional
Analysis
Discovery
Discovery
Info
Comprehension
Tools
Tools
Tools
Comprehension
Matches/ Partial Matches
Target
Missed Questions
Models
ANTICIPATION
Potential New Relationships
Tools
The Goal
Anticipation
07-291
9
Situation Awareness Reference Model(Applied to
Cyber Domain)
Open Source
TBD
07-291
10
Lexicon(Background)

• Evidence
• IDS Alerts (i.e., Snort, Dragon)
• System Logs
• Service Logs (i.e., Apache, IIS)
• Network Flow Data
• Track collection of all evidence available
  against one or more targets originating from one
  or more attackers
• Situation set of tracks at a snapshot in time
• Situation Awareness of a Network analysts
  mental model of the situation
• True Positive successful attack
• False Positive incorrectly identified attack
• Non-relevant Positive correctly identified
  attack that fails or is incomplete (i.e., try to
  exploit a blocked vulnerability)

Valeur et al, A Comprehensive Approach to
Intrusion Detection Alert Correlation, IEEE
Transactions on Dependable and Secure Computing,
Jul-Sep 04
06-081
11
Metrics Overview

• Confidence measures the ability of the system
  to correctly identify the track(s)
• Recall Percentage of tracks detected in relation
  to the total known
• Precision Percentage of correct tracks detected
  in relation to number of detections
• Fragmentation Percentage of tracks reported as
  multiple tracks that should have been reported as
  a single track
• Mis-Association Percentage of tracks that are
  neither correct nor a fragment in relation to the
  number of detections
• Purity characterizes the quality of the
  detections
• Mis-Assignment Rate Percent of evidence
  incorrectly assigned to a given track
• Evidence Recall Percentage of evidence detected
  in relation to the total known evidence
• Cost Utility a single weighted measure of the
  system in identifying important or key tracks
  with respect to a concept of cost
• Timeliness measures the ability of the system
  to respond within time requirements of a
  particular domain

06-081
12
Cost Utility(Weighted Cost and Attack Score)
? Weighted Values for Results
Weighted Cost
? Weighted Values for Ground Truth
No. Attacks in ResultsNo. Results Sum of
Positions of Attacks in Results Geometric Sum
(No Attacks in Results -1)
Attack Score
No. Attacks in GTNo. Results
Given
100 pts ATTACK 5 pts Background Scan 5 pts
Background Attack -50 pts False Positive
Ground Truth
Proposed Attacks
R0 Background Scan R1 UNASSIGNED R2 Attack R3 Back
ground Scan R4 Background Scan R5 Background
Attack
5 - 50 100 5 5 5 70
GT0 Background Scan GT1 Background
Attack GT2 Background Scan GT3 Attack GT4 Backgrou
nd Scan
5 5 5 100 5 120
NOTE Sorted Based on Score
Weighted Cost 70/120 .5833
Attack Score (1)(6) (2-(1-1))/(1)(6) .6667
06-081
13
The Infrastructure
Skaion Dataset
Processing Results
Results UsingAFRL Schema
Viewing Ground Truth
Cyber Fusion System
AFRL Results Analyzer Tools
AFRL Ground Truth Correlation
Ground Truth
Assignment Matrix
.csv .html
List of Potential Attacks
REPORTS
Alerts correlated to selected Attack Track
Filter by score
Play Buttons
Metric Report
(Confidence, Purity, Cost)
06-081
14
Work has Raised Many Questions Resulting in Few
Answers

• Where do groups, events, activities fit in?
• Can we not track a group, an activity (Why only
  Objects?)
• Is a group or activity only a complex object?
• What is a Situation? Is there more than one? Is
  it Context-based?
• Where does Knowledge Discovery exist? Forensics?
• What is Situation Assessment?
• Is Threat Assessment only of the future what
  about current threat?
• What about forecasting or projecting the future
  state?

No one model answers ALL of these questions
and even addresses them!
07-210
15
so Then What

• Treating Situation as a composite of activities
  and tracking activities as complex objects allows
  for a cleaner distinction between fusion levels
• Situation(s)-gt Activity(s) -gt Group(s)/Entity(s)
  -gt Event(s) These are ALL OBJECTS THAT CAN BE
  TRACKED
• Object Assessment has really been performing
  Tracking Identification LETS TRACK ALL TYPES
  OF OBJECTS
• Knowledge Discovery and a priori knowledge
  necessary and integral to building complex
  objects (e.g., Groups, Activities)
• Updating knowledge/relationships (models) is
  continuous and part of refinement process
• Define Situation Assessment based on Jean Roys
  Definition for Situational Analysis
• Behavior Analysis Activity Level Analysis
• Intent Analysis Salience Analysis
• Capacity/Capability Analysis Impact Analysis
• Threat Analysis

07-210
16
and

• Use Time to distinguish between JDL Level 2 and 3
  as does Endsleys comprehension and projection
• Same analysis is done for both levels only
  difference is time
• Thus JDL Level 2 is assessment of current
  situation and JDL Level 3 is the assessment of
  the current situation projected forward in time.
• Process Refinement involves not only sensor
  movement/collection (sensor management) BUT
  fusion algorithm management (which algorithms and
  which parameters to use) and model management
  from ALL processes. Possible sources to
  refinement include
• L1 Prediction where object is moving/next
  event
• L2 Missing data, increase certainty of current
  assessments
• L3 Forecasted actions/placement to
  pre-position sensors

07-210
17
Revised Situational Awareness Reference Model
(Based on Previous Suggestions)
Level 1 Object Tracking and Identification
Level 2 Assessing the Current Situation(s)
Level 3 Assessing the Forecasted Situation(s)
Based on JDL, Endsleys, and Jean Roys work
07-210
18
Wrap Up

• We proposed a revised Reference Model that
  includes many of the lessons learned to date
• Plans are to continue to apply this revised model
  to Air, Cyber and Space Situation Awareness
  UNIVERSAL SITUATION AWARENESS
• with emphasis on current and forecasted
  situation assessment

07-210