Illustrative Analysis of ELICIT Experiment Data

1
Illustrative Analysis of ELICIT Experiment Data

• Evidence Based Research
• September 26, 2007
• Richard E. Hayes, Ph.D.
• Jimmie G. McEver, Ph.D.
• Danielle M. Martin

2
Agenda

• ELICIT Experimentation
• Task Objectives
• Exemplar Results from Participant-oriented
  Analysis
  
• Exemplar Network Analysis
• Analytic methods and tools
• Way Ahead

3
What is ELICIT?

• ELICIT Experimental Laboratory for
  Investigating Collaboration, Information-sharing,
  and Trust
  
• U.S. DoD (OASD/NII) Command and Control Research
  Program (CCRP) sponsored the design and
  development of platform for experimentation
  focused on information, cognitive, and social
  domain phenomena
• Purpose of ELICIT related Experimentation and
  Analysis
  
• Investigate cognitive and social impacts of C2
  approach and organizational structure within the
  context of an information sharing, shared
  awareness, and knowledge task
• Initial application Comparison of traditional
  hierarchy v. edge organization/approach
  
• ELICIT Platform was designed to allow measurement
  of trust and shared awareness

4
ELICIT Scenario

• Participants receive factoids about a future
  attack
  
• Factoids fall into four task categories who,
  what, when, and where
  
• Factoids are periodically distributed to the
  participants
  
• No one is given sufficient information to solve
  to problem without receiving info from others
  
• The goal of each set of participants is to build
  awareness regarding the attack and identify
  attack attributes
  
• Participants can share factoids with each other
  or post to websites
  
• Participants build awareness by interacting with
  agents, websites
  
• The receiving, sharing, and posting of factoids
  can be constrained

5
Illustrative Scenario

• Who Fact The Lion is known to work only with the
  Azur, Brown, or Violet groups (classification
  key fact)
  
• What Fact Bloggers are discussing the role of
  financial institutions in oppressing the Coral,
  Violet and Chartreuse groups (classification
  supportive fact)
• When Fact The Brown group needs time to regroup
  (classification nonessential fact)
  
• Where Fact The Azur, Brown, Coral, and Violet
  groups have the capacity to operate in Tau,
  Epsilon, Chi, Psi and Omega-lands (classification
  supportive fact)
• Sample Identification Attempt "whoRed Group
  whatTV station whereB-land whenNov 8, 1000
  pm
  
• Illustrative Correct Solution The Green group
  plans to attack a TV station in A-land on 25 Nov
  at 11AM

6
Summary of Experimental Runs

• ELICIT has been used at Boston University, Naval
  Postgraduate School, West Point, Portugal,
  Singapore
  
• Future use planned for Germany, UK, SAS-065
• Currently 26 data sets have been collected
• Boston University 2 runs
• Naval Postgraduate School 16 runs
• West Point Military Academy 2 runs
• Portugal 6 runs
• 13 of each organization type
• 2 civilian groups and 24 military groups
• 7 undergraduate groups and 19 professional
  groups
  
• 20 US and 6 Non-US

7
Agenda

• ELICIT Experimentation
• Task Objectives
• Exemplar Results from Participant-oriented
  Analysis
  
• Exemplar Network Analysis
• Analytic methods and tools
• Way Ahead

8
Analytical Objectives

• A rich set of data has been generated using the
  ELICIT experimentation platform. Given the
  growing interest in ELICIT we expect this
  experimental data to continue to accumulate.
• Therefore, in support of the developing community
  of interest the CCRP was interested in how this
  empirical data could be exploited.
  
• Our task is to conduct a set of illustrative
  analyses to demonstrate what can be gained
  through this research and display tools and
  methods that can be used to such analysis.
• Gain insights into information sharing, shared
  awareness, and emergent leadership.

9
Candidate measures

• shares/minute as f(t)
• pulls/minute
• posts/minute
• identifications/minute
• correct identifications/minute
• shares/individual/minute
• pulls/individual/minute
• posts/individual/minute
• identifications/individual/minute
• correct identifications/individual/minute

10
Sharing Behavior Metrics - Individuals

• Sharing behavior
• (postsshares)/(individual-time)
• shares/(individual-minute)
• web posts/(individual-minute)
• Number of unique sharing partners for each
  individual (out-degree)
  
• of known facts shared via
• postsshares
• shares
• web posts
• shares-with/(individual-minute)
• in-degree
• of known facts received from
• postsshares
• shares
• web posts

Rate
Extent of Sharing
What you are Sharing
Receiving rate
Receiving extent
Facts received
11
Sharing Behavior Metrics - Collective

• Sharing behavior
• (postsshares)/minute
• shares/minute
• web posts/minute
• Density of transaction network
• of all facts shared via
• postsshares
• shares
• web posts

Rate
Extent of Sharing
What is being sharing
12
Information Metrics

• Facts accumulated by individual/minute
• New facts accumulated by individual/minute
• Shared information
• Number of facts known by all
• Number of facts known by at least N participants

13
Network Metrics

• Overarching properties of the transaction network
  that emerged during the course of the experiment
  
• Characteristic path length
• Of individual
• Of all paths to all nodes
• Geodesic (largest path length between two nodes)
• Centrality of nodes nodes of relative
  importance
  
• Connectedness of the network
• Opportunity to influence
• Centralization
• Symmetry of network with regard to relative
  importance of nodes

14
Agenda

• ELICIT Experimentation
• Task Objectives
• Exemplar Results from Participant-oriented
  Analysis
  
• Exemplar Network Analysis
• Analytic methods and tools
• Way Ahead

15
Illustrative Statistical Comparisons

• Military v. non-military
• Military Military Academy
• Non-Military Boston University
• Comparisons between national cultures
• US participant groups v. Portuguese participant
  groups
  
• Undergraduates v. others
• Undergraduates Boston University, Military
  Academy
  
• Professionals NPS Staff, Portuguese
  Participants
  
• Hierarchy v. edge
• Performance with different factoid sets

16
Military vs. Civilian

• Military Military Academy
• Feb. 1, 2007
• Org type Edge
• Factoid set 1
• Nation US
• Undergraduate students
• Run time 59 minutes
• Unrestricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes
  

• Civilian Boston University
• June 22, 2006
• Org type Edge
• Factoid set 1
• Nation US
• Undergraduate students
• Run time 60 minutes
• Unrestricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes
  

17
Military vs. Civilian ComparisonResults Summary
18
Shares by Individuals
Military
Civilian
19
Shares by Individuals (continued)
Note USMA sharesrange up to 200
CIV (BU)
MIL(USMA)
20
Mean Shares by Time Interval (Military vs.
Civilian)
21
Shares Over Time by Participant
Civilian
Military
22
Sharing Rate (Continued)
Civilian
Military
23
Posts by Individuals
Civilian
Military
No significant difference
24
Pulls by Individuals
Civilian
Military
Higher for Military (at 5 level)
25
Shares, Pulls, Posts (Military vs. Civilian)
Civilian
Military
Time Intervals (minutes) 10 0-10 20 10-20 3
20-30
4 30-40 5 40-50 6 50-60
26
Average access to new facts over time by source
Civilian
Military
Fact distribution waves (from the server) occur
at t0, 5, 10 minutes The first two waves occur w
ithin the first time interval and the last wave
occurs within the second time interval.
27
Level of Understanding over Time
28
Comparison Between National CulturesUS vs.
Portuguese participant groups

• Portugal Portugal
• July 4, 2007
• Org type Hierarchy
• Factoid set 2
• Nation US
• Military/Professionals
• Run time 35 minutes
• Unrestricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes
  

• US NPS
• Feb. 2, 2007
• Org type Hierarchy
• Factoid set 2
• Nation US
• Military/Professionals
• Run time 67 minutes
• Restricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes
  
• Postcards were passed as an additional form of
  communication during this run

29
US/Portugal ComparisonResults Summary

Restricted number of identification attempts in
Other group
30
Shares by Individuals
Portugal
United States
31
Shares by Individuals (continued)
United States
Portugal
32
Mean Shares by Time Interval (United States vs.
Portugal)
33
Posts by Individuals
United States
Portugal
No significant difference
34
Pulls
United States
Portugal
No significant difference
35
Shares, Pulls, Posts (Military vs. Civilian)
Civilian
Military
Time Intervals (minutes) 10 0-10 20 10-20 3
20-30
4 30-40 5 40-50 6 50-60
36
Shares, Pulls, Posts (United Stated vs. Portugal)
Portugal
United States
Time Intervals (minutes) 10 0-10 20 10-20 3
20-30
4 30-40 5 40-50 6 50-60
37
Access to new facts
United States
Portugal
Fact distribution waves (from the server) occur
at t0, 5, 10 minutes The first two waves occur w
ithin the first time interval and the last wave
occurs within the second time interval.
38
Level of Understanding
US Run Restricted number of identification
attempts Postcards were passed as an additional f
orm of communication during this run
39
Undergraduate vs. Other

• Other NPS
• Feb. 5, 2007
• Org type Edge
• Factoid set 1
• Nation US
• Non-Undergraduate students
• Run time 48 minutes
• Restricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes
  

• Undergraduate Military Academy
• Feb. 1, 2007
• Org type Edge
• Factoid set 1
• Nation US
• Undergraduate students
• Run time 59 minutes
• Unrestricted number of identification attempts
• Factoid Distribution occurs at 0, 5, and 10
  minutes

40
Undergraduate vs. Other

Restricted number of identification attempts in
Other group
41
Shares by Individuals
Other
Undergraduate Students
42
Shares by Individuals (continued)
Undergraduate
Other
43
Shares by Time Interval (Undergraduate vs. Other)
44
Posts
Undergraduate Students
Other
Undergraduates significantly higher at 5 level
45
Pulls
Undergraduate Students
Other
Undergraduates significantly higher at 5 level
46
Shares, Pulls, Posts (Undergraduate vs. Other)
Other
Undergraduate Students
Time Intervals (minutes) 10 0-10 20 10-20 3
20-30
4 30-40 5 40-50 6 50-60
47
Access to new facts
Undergraduate Students
Other
Fact distribution waves (from the server) occur
at t0, 5, 10 minutes The first two waves occur w
ithin the first time interval and the last wave
occurs within the second time interval.
48
Level of Understanding
Other(NPS) Run Restricted number of
identification attempts
49
Hierarchy vs. Edge ComparisonCorrectness Scores
50
Shares
Hierarchy
Edge
Difference not significant at 5 level
51
Shares by Time Interval (Hierarchy vs. Edge)
52
Posts
Hierarchy
Edge
Difference not significant at 5 level
53
Pulls
Hierarchy
Edge
Difference not significant at 5 level
54
Shares, Pulls, Posts (Hierarchy vs. Edge)
Hierarchy
Edge
Time Intervals (minutes) 10 0-10 20 10-20 3
20-30
4 30-40 5 40-50 6 50-60
55
Level of Understanding
56
Analysis of Effects of Factoid Sets

• During the initial conceptualization of ELICIT,
  Factoid Sets were carefully designed to be
  equivalent, in order to not impact
  experimentation, and to facilitate comparisons
  across experiments using different sets
• While 2007 trials were ongoing, the suggestion
  has been put forward that some factoid sets may
  in fact be more challenging than others
  
• Questions for analysis
• Is this so? If so, which ones, and why?

Evaluate by examining correctness levels achieved
by participants using different factoid sets
57
Comparison of correctness ratings across factoid
sets
Correctness of responses by Factoid Set used

• Empirically, Factoid Set 2 appears to have been
  more difficult for the participants

58
Linear regression model accounting for additional
factors

• Significant factors affecting expected
  correctness
  
• Time
• Whether or not the participant was an
  undergraduate student
  
• Use of Factoid Set 2
• Participant community (military/civilian)
• Use of Factoid Set 3
• Factoid Set 2 is still shows as presenting
  significantly more challenge

59
Illustrative Factoid Set Solution Table
60
Relative difficulty of What solution area
Correctness of What answers

• Difference in correctness of What responses is
  observed among the factoid sets
  
• Accuracy with Set 2 significantly lower than with
  other sets
  

61
Requiring exact answers exacerbates effect
Correctness of What answerswith strict grading

• Evaluation of correct responses when strict
  correctness criteria applied (only exact answers
  receive credit) exacerbates observed effect

62
Correctness with loose grading

• In the loose grading schema, credit was given
  for partially correct answers
  
• school instead of religious school
• embassy instead of zetaland embassy
• Performance difference no longer observed

Correctness of What answerswith loose grading
63
Performance in other areas also affected by
difficulty of set

• Other solution areas (who, where, when) may also
  have been affected by the challenge of the What
  area
  
• Effect also observed in Where and When areas

Example Performance on Who
64
Effects of Language
Portuguese correctness with English- and
Portuguese-language factoid sets

• Portuguese experimenters translated Factoid Set 3
  into Portuguese
  
• Performance improved, but not significant
• Small n only 1 Portuguese experiment used the
  English version of Factoid Set 3

English
Portuguese
65
Agenda

• ELICIT Experimentation
• Task Objectives
• Exemplar Results from Participant-oriented
  Analysis
  
• Exemplar Network Analysis
• Analytic methods and tools
• Way Ahead

66
Civilian Edge Organization Arc Strength
67
Military Edge Organization Arc Strength
68
Civilian EdgeOrganization Reciprocity
69
Military Edge Organization Reciprocity
70
Degree Centrality(Military vs. Civilian)
71
Betweenness Centrality(Military vs. Civilian)
72
United States Hierarchical Organization Arc
Strength
73
Portuguese Hierarchical Organization Arc Strength
74
United States HierarchyOrganization Reciprocity
75
Portuguese Hierarchy Organization Reciprocity
76
Degree Centrality (United States vs. Portugal)
77
Betweenness Centrality(United States vs.
Portugal)
78
Undergraduate Edge Organization Arc Strength
79
Non-Undergrad OrganizationArc Strength
80
Undergraduate Edge Organization Reciprocity
81
Non-Undergrad Organization Reciprocity
82
Degree Centrality(Undergraduate vs. Other)
83
Betweenness Centrality(Undergraduate vs. Other)
84
Agenda

• ELICIT Experimentation
• Task Objectives
• Exemplar Results from Participant-oriented
  Analysis
  
• Exemplar Network Analysis
• Analytic methods and tools
• Way Ahead

85
Library of tools Available for data extraction
and analysis

• Current list of software packages that could
  assist users in analyzing experiment data
  includes
  
• Excel
• NetDraw
• R
• Netstat
• UCINet
• JMP
• SAS
• SPSS

86
Challenges

• Decisions need to be made on how to analyze the
  solution identification submissions and possibly
  revise the instructions to limit
  misinterpretation
• In analysis, decisions are needed for how to
  consistently handle free text responses
  
• Partially correct answers
• Partial identifications
• Future chat capabilities

87
Next Steps

• Incorporate feedback
• Document a data collection and data analysis plan
  for data extraction and analysis
  
• Document the methods and software used to conduct
  this illustrative analysis

88

• Questions ?

89
Hierarchy - Organizational Structure
Sam
Quinn
Morgan
Leslie
Where Website
Robin
Jesse
Kim
Pat
What Website
Alex
When Website
Chris
Francis
Taylor
Sidney
Who Website
Harlan
Whitley
Val
Dale
90
Experimental Runs
91
Level of Situational Understand (old)