Methods and Metrics for Analysis of Sensemaking

1
Methods and Metrics for Analysis of Sensemaking

• Dr Karen Carr Mr Barry McGuinness
• BAE SYSTEMS
• Advanced Technology Centre

2
Our objectives

• For this meeting
• Contribute what makes sense to us, in our given
  context and with our goals
• In our work
• Develop the ability to supply C3I capabilities
  (in partnership)
• Systems Engineering of socio-technical systems
• Driven by need to deliver usable and demonstrable
  results
• Science as well as engineering and domain
  expertise
• NB We want to ensure that human issues drive the
  developments - but we dont want to forget that
  we have to inform technology (as well as
  organization, process)

3
What we mean by sensemaking

• Why we want to use this concept to try and answer
  the questions we need to answer
• Our question
• How can we develop technology, design and manage
  systems which support/ enhance the human roles in
  defence operations?
• Significant human role is ability to adapt,
  respond to unexpected, creativity, play mind
  games, etc. Need to preserve enhance that -
  not interfere.
• Support is needed for dealing with the
  unexpected, the unknown, as well as recognisable
  situations
• Include broad System of Systems issues,
  developers, rapid change
• Sensemaking ( Situational Awareness) is a
  working concept to enable us to start
  manipulating, analysing, and measuring context,
  goals and human performance

4
Why we want methods and metrics for studying
sensemaking

• Need to attribute effects - predict - in order to
  provide support.
• Move from concepts to metrics to analysis to
  (testable) models.
• Reduce subjective bias (influence of our own
  sensemaking, interpretation)
• No existing clear metrics we can use - no
  absolutes
• 1. Understand how human performs, and what
  conditions facilitate good performance (what
  hinders)
• 2. Identify the properties of organisation,
  process, technology, training, etc which are
  important for success
• 3. Develop design and management methods and
  tools to enable implementation
• NB not necessarily numbers - could be properties

5
Range of methods

• observation (non intrusive)
• subjective investigation (e.g. ethnographic,
  knowledge elicitation)
• storytelling/anecdotes (knowledge building)
• metaphor (pattern matching)
• scientific method (empirical hypothesis testing)
• mathematical analysis (baseline)

6
Methods and Metrics

• Concepts
• Metrics
• Some analyses
• Implications for sensemaking

7
Concepts

• Orientation
• complex, uncertain situations
• SA determines capacity to decide and act
• sensemaking determines SA
• cognitive processes are intrinsically
  goal-directed
• people form nested hierarchy of processes
  outcomes
• Objectives
• Understand SA and sensemaking
• Feed into design development of information
  systems and human systems
• Applied research -- theory into practice

8
What is a Situation?

• A situation is a pattern in state space,
  especially one which appears to deviate from a
  normal intended or expected pattern.
• Example- aircraft fuel x time into flight

Aircraft fuel level
Normal takeoff
Unexpected rate-- we have a situation!
Normal cruise
Time into Flight
9
Unrecognized Patterns

• An unrecognized pattern demands attention.

Attention!
perception
comprehension
???
Unknown pattern?
Known patterns?
Perceived pattern
10
Definitions

• Knowledge
• capacity for action
• Situational Awareness
• dynamic situated knowledge, i.e. capacity to
  act effectively here now in a given specific
  situation
• Sensemaking
• process of creating effective SA in situations
  of uncertainty

• doing
• saying
• thinking

Knowing whats going on so you can figure out
what to do.
11
Situational awareness

• Dynamic mental representation of the current and
  future state of ones domain of action
• includes awareness of
• environment
• entities
• events
• processes
• actions
• others perceptions intentions
• insofar as these are relevant to
• performing an action, or
• choosing a course of action

Through a continuous process of situation
assessment
12
Situational awareness

• SA is based on ...
• prior KNOWLEDGE
• SCHEMAS generalized patterns representing
  typical situations
• based on experience, training, culture
• recent INFORMATION
• direct perception of the environment
• perception of instruments and displays
• received communications

KNOWLEDGE
SA
INFORMATION
instruments
communications
PHYSICAL DOMAIN
13
Central role of SA

• SA both informs and is informed by
• sense-making
• decision-making

Decision- making
Sense- making
SA
COGNITIVE DOMAIN
Action performance
Information acquisition
PHYSICAL DOMAIN
14
Inside SA Cutting up the cake
Observed
Implied
Models situational schemas e.g., Fuel
leak? Faulty sensor?
Projections mental simulations e.g., Risk of
not reaching destination
Abstract (generalized patterns)
Concrete (situation-specific)
Information specific propositions e.g., rate of
fuel loss is high
Intentions selected actions afforded by
situation e.g., Contact ATC and inform
15
Processes involved in SA

• PERCEPTION Acquisition of information
  about the given situation
• COMPREHENSION Diagnostic interpretation of
  the given situation
• PROJECTION Prognostic simulation of
  future situations and their possible outcomes
• RESOLUTION Selection of actions to
  direct the given situation towards the
  desired outcome
• All serving to support dynamically effective
  action

information
models
projections
intentions
16
Sensemaking and SA
PROJECTION
COMPREHENSION


Models
Projections
Decision making
Sense making


Information
Intentions
PERCEPTION
RESOLUTION
Sensing
Acting
Sense-making when comprehension is
uncertain Decision-making when resolution is
uncertain
17
Metacognition

• Defined as
• Thinking about thinking or knowledge about
  knowledge
• i.e. Awareness of your own SA
• noticing uncertainties, gaps, conflicts in your
  mental reps
• identifying information needs
• employing strategies for sensemaking
  decision-making

?
SA
Its like looking over your own shoulder.
Gives a subjective sense of SA
18
SA and metacognition

• Four possible states

Actual awareness
True SA
False SA
Inappropriate Confidence (danger state)
Appropriate Confidence (ideal state)
Confident in SA
Subjective attitude
Inappropriate Sensemaking
Appropriate Sensemaking
Not confident in SA
Need for sensemaking
19
Team SA and shared SA

• Not the same thing
• Team SA sum of current knowledge held across a
  team, irrespective of who has it
• Shared SA those parts of the team SA that are
  common between team members

Team SA
Shared SA
Personal SA
20
What to share, with whom?

• The nature of SA in groups is dictated by goals
• Goals are hierarchic
• Top-level goals are shared by all members
• therefore need shared SA with respect to that
  objective
• Lower-level goals are specific to individuals
• therefore need personal SA with respect to own
  task
• Sharing ones SA is necessary only to the extent
  that the knowledge has bearing on the goals of
  others

21
Team SA and shared SA

• Shared SA elements can be differentially allocated

resolution
comprehension
projection
comprehension projection
perception
perception resolution
perception
perception
perception resolution
perception resolution
22
Distributed SA in the C2 HQ
Metacognition?
u
z
Models (COMPREHENSION)
Intel
m
Projections (PROJECTION)
Ops
m
m
M
m
m
m
Intentions (RESOLUTION)
Information (PERCEPTION)
Commander
Signal
23
So...

• Explicit sensemaking processes are needed when
  comprehension cannot easily occur
• Sensemaking requires metacognitive awareness of
  own knowledge -- uncertainties, gaps
• Metacognitive assessments can be wrong and lead
  to inappropriate subjective attitude -- and
  inappropriate behaviour

24
Measuring SA

• COGNITIVE approach
• queries about the situation
• Reveals mental reps
• Multiple choice (SAGAT)
• True/False (QUASA)
• Sit Reps
• SUBJECTIVE approach
• self-ratings of SA
• Reveals metacognitive state
• Unidimensional (SARS)
• Multidimensional (SART)
• Multidimensional and intelligible! (CARS)

• OBJECTIVE approach
• behavioural physiological correlates
• Reveals changes in metacognitive state
• EEG, fMRI
• Eye pointing

As a rule, take both cognitive subjective
measures together.
25
CARS

• Crew Awareness Rating Scale
• a subjective tool to elicit operators
  subjective sense of SA
• multi-dimensional
• generic, adaptable, easy to use

26
Dimensions
Knowledge Processing
Perception Comprehension Projection Resolution
27
Eight CARS questions
knowledge
1. the most recent information 2. what is really
going on here 3. what could happen 4. what
actions should be taken
Would you say you have a good sense of
processing
1. monitor the flow of information 2. understand
the big picture 3. predict how it is likely to
evolve 4. decide what actions to take
Would you say it is easy for you to
28
Six possible responses
For sure?
Certain Uncertain
YES NO
Do I ?
Think so
Definitely
Definitely not
Think not
Dont know
Dont need it
29
CARS results
Def Prob Prob not Def not DK NA

• CONTENT
• Perception
• Comprehension
• Projection
• Resolution
• PROCESSING
• Perception
• Comprehension
• Projection
• Resolution

30
CARS results
Comprehension knowledge over time
100
80
60
of ratings
40
20
0
Run 1 Run 2 Run 3 Run 4
31
QUASA

• Quantitative Assessment of Situational Awareness
• a probe tool to elicit operators actual SA
• mathematical based on SDT
• still under development, but promising

32
QUASA
Square?

• Signal Detection Theory

YES!
perception
discrimination

• Targets vs non-targets
• Hits, False Alarms, Good Misses, False
  Rejections
• Also applies to internal (mental) representations

33
QUASA

• Is this item true?
• Confidence in perceived truth value of items
  varies

Number of items
Max SENSITIVITY ideal SA
TRUE items
FALSE items
Weak
Strong
Confidence in truth value of items
34
QUASA
No sensitivity, poor SA
Literally cant tell the difference between true
false items They have similar-strength levels
of confidence
Weak
Strong
35
QUASA
Deception
Number of items
Max NEGATIVE sensitivity worst case SA
FALSE items
TRUE items
Weak
Strong
Confidence in truth value of items
36
QUASA
Some positive sensitivity Low positive
bias (acceptance threshold)
Good rejections
Good acceptances
Bad acceptances
Bad rejections
Weak
Strong
37
QUASA
Example probe The tanks adjacent to bridge are
enemy Response YES (accept as true) or NO
(reject as false) Evaluate Sensitivity
(discrimination of true/false situations)
SA Bias (probability of item acceptance/rejection
) IB
38
QUASA
100 0 -100
Maximum negative sensitivity the wrong situation!
Maximum negative bias too cautious
Maximum positive sensitivity ideal SA
Maximum positive bias too rash
TYPICAL
39
QUASA
100 0 -100
Comprehension model of situation
Projection Future developments
Resolution CoA intention
Perception information
40
(No Transcript)
41
QUASA

• Mathematical assessment of SA
• Needs the truth!
• SA, bias, components, temporal
• ? Team shared SA

42
Behavioural correlate of SA
Tracking eye-point-of-gaze (EPOG)
Do EPOG patterns correlate with SA?
43
EPOG research
Entropy known loss of SA
44
SA and flightdeck automation
45
SA and flightdeck automation
46
SA and C2 digitization - ISTAR
Own force positions
Enemy positions
BGHQ crewstation Common Operational Picture
47
SA and C2 digitization - ISTAR
Battlespace digitization demonstrator
Synthetic environment
48
SA and C2 digitization - ISTAR

• 2-hr ISTAR recce operation
• Performed with voice AND digital C2 systems

Measures taken of mental workload situational
awareness
49
SA and C2 digitization - ISTAR
voice
digital
DEF
PROB
rating
PROB NOT DEF NOT
PERCEPTION
PROJECTION
RESOLUTION
COMPREHENSION
aspects of SA (knowledge of enemy)
50
Some implications

• Both actual SA and subjective sense of SA affect
  decision-making performance
• Technology can affect SA for better or worse
• Analyses with metrics provide specific insights

51
Other work

• DS1 trials
• BattleLab trials
• Cognitive modelling
• COGNET in C2 environment
• Ideal Decision Maker
• ? Can be used to predict dips in SA and
  sensemaking needs

Building Industry-MoD partnership
52
Implications for sensemaking

• Thinking about thinking
• Concepts sensemaking as processes supporting SA
• Role of metacognition group context
• Metrics of SA can be used to evaluate
  sensemaking solutions
• Data can feed development of predictive models
• Knowing whats going on so we can figure out what
  to do!