Recognizing Teamwork Activity in Observations of Embodied Agents

1
Recognizing Teamwork Activity in Observations of
Embodied Agents

• Linus J. Luotsinen
• School of Electrical Engineering and Computer
  Science
• University of Central Florida
• B. Sc. University of Dalarna, Sweden, 2002
• M. Sc. University of Central Florida, 2004
• Ph.D. Dissertation Defense
• November 06, 2007

2
Committee Members

• School of Electrical Engineering and Computer
  Science
• Dr. Ladislau Bölöni
• Dr. Avelino Gonzalez
• Dr. Kenneth Stanley
• Department of Statistics and Actuarial Science
• Dr. Liqiang Ni

3
Outline
Covered in candidacy

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

4
Introduction

• Recognize teamwork activity in a stream of
  positional agent traces, and annotate them with
  the recognized actions
• Video demonstration

5
Applications

• Surveillance
• Recognize multi-agent activity in surveillance
  video feeds
• Training
• Identify discrepancies and deviations from the
  actions performed by an expert team
• Smarter agents
• Model the opponent team to imitate or
  countermeasure its actions
• Automated annotation
• Automatically index large databases for fast
  content retrieval
• After Action Review
• Digital Video Recorder

6
Challenges

• Observation noise
• Position traces can be distorted by inaccuracies
  in sensors and localization algorithms
• Alignment problems
• Movement performed at different location and
  orientation
• Scaling problems
• Movement performed at different physical scale
• Temporal scaling
• Movement happens slower or faster in time
• Terrain distortion
• Movement distorted because of adaptation to the
  terrain

7
Challenges

• Movement variants
• Movement in alternative ways that map to the same
  label
• Uncertainty regarding the role of the agents in
  the team
• Role count variants
• Movement with different number of agents in the
  team
• Agents changing their roles during the team
  action

8
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Software tools we developed
• Datasets we acquired, segmented and labeled
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

9
Software Tools we Developed

• Teamwork Scenario Editor (TSE)
• Interactive knowledge engineering tool
• Video editor interface
• Visualize large datasets and geographical areas
• All knowledge engineering work in this thesis was
  performed using the TSE
• Acquire the datasets
• Find teamwork activities
• Label the activities

10
Datasets we Acquired, Segmented and Labeled

• Real-world warfare exercise
• Recorded over a three days
• Data collected from hundreds of soldiers and
  tanks equipped with GPS devices, laser range
  finders and laser range detectors

11
Datasets we Acquired, Segmented and Labeled

• Military Operations in Urban Terrain (MOUT)

12
Military Movement

• We extract military movement patterns from the
  warfare databases
• In the military domain the movement techniques
  and formations are selected based on the
  situational awareness of the team
• Traveling
• Enemy contact not expected
• Fast movement speed
• Traveling overwatch
• Enemy contact possible
• Medium movement speed
• Characterized by continuous movement of lead unit
  and alternating advancement of rear units
• Bounding overwatch
• Enemy contact expected
• Slow movement speed
• Alternating or successive bounds

13
Military Movement

• Formations are used in combination with movement
  techniques
• Formation is selected based on visibility needs,
  firepower focus and so on
• Column
• Leader followed by rear units
• Fire power in all directions (flanks, front and
  rear)
• Line
• Fire power in the front
• Wedge
• Fire power in the front and in the flanks
• Echelon
• Enemy contact expected in the front or in the
  echeloned flanks
• Used when one flanks are secured by obstacles

14
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

15
Teamwork Activity Recognition using Hidden Markov
Models

• A spatio-temporal pattern recognition problem
• Recognize teamwork from X and Y coordinates over
  time for multiple agents
• Why Hidden Markov Models?
• Mathematically sound
• Temporal by nature
• Successfully applied in the past (e.g. speech
  recognition)
• Ways of encoding teamwork activity in HMMs
• Knowledge engineering
• Learn from a set of representative examples

16
Workflow Overview
17
The Hidden Markov Model

• The HMM consists of a number hidden states
• Transition probabilities
• Emission probabilities
• Gaussian PDF
• Initial probabilities

HMM with 3 hidden states
18
The Hidden Markov Model

• Learning algorithms
• Baum-Welch
• Optimize by maximizing
• Segmental K-means
• Optimize by maximizing , where
  is the optimum hidden sequence given by the
  Viterbi decoding algorithm
• Classification
• Determine the probability that the input sequence
  was generated by the HMM
• Forward evaluation algorithm

19
Baseline Input Format for the HMM Basic
Preprocessing

• The input is a vector of the agent positions
• VT v1, v2, v3,, vt
• vt x1, y1, x2, y2, x3, y3, , xn, yn
• It is very unlikely that the team action will be
  repeated in the same location and position!
• We perform a pre-processing of the input data
  which allows us to recognize team actions
  happening at arbitrary locations and orientation
• Translation
• Align with team centroid
• Rotation
• Align with x-axis
• This is not sufficient, many other distorting
  factors can happen scaling, terrain distortion,
  different ordering of the agents we are dealing
  with these later in this presentation

20
Experimental Setup

• Test problem
• Real-world military warfare exercise
• Train and test data
• Six activities (extracted using TSE)
• Artificial activities added for testing

21
Results Recognition Accuracy

• Classification accuracy is 82
• Matches the performance of the knowledge
  engineering approach

As presented at the AAMAS-07 conference
22
Results Real-Time Analysis

• HMM with 4 states and 6 classes ? 9.4ms

Hidden States Mean Time (ms)
1 2.543
2 4.743
3 7.02
4 9.394
5 12.06
23
Shortcomings A Lot of Assumptions!

• We assumed that the teamwork activities were
  performed at the same scale
• We assumed that there is no interaction with the
  environment
• Observation input is of fixed arrangement, hence,
  we assumed that recognition is performed on the
  same team the HMM was trained for
• We assumed that all activities can be modeled
  using the same number of hidden states in the
  HMMs
• Dimensionality and state space for larger teams
  will quickly grow out of control
• We do not recognize the roles of the agents in
  the teamwork activity
• Who is the leader and who is the follower?
• The recognizer is not practical!

24
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

25
Team-Oriented Feature Extraction

• Improve recognition accuracy and robustness by
  replacing the vector of positions input with a
  collection of team-oriented features
• The features are extracted through pre-processing
• Some of the features replace the existing input
  vector
• Some of the features are the input of the role
  recognizer (and there is some overlap)
• It requires a more complex recognition workflow
  (which will be shown later, together with the
  role recognition module)
• Extract semantically rich features from the agent
  position traces
• Discretization process
• Intuitive descriptions of teamwork activities
• For classifiers using discrete input
• Three feature function classes
• Agent-oriented features
• Environment-oriented features
• Team-oriented features
• Calculated over a sliding window in time

26
Agent-Oriented Feature Functions

• Focus on individual agents
• Enhance performance of teamwork recognition
• Used to recognize (likely) roles

27
Curvature

• The rate at which a curve changes direction

Finite difference approximation with central
difference
28
Environment-Oriented Feature Functions

• Agent and team interactions with environment
• Environmental objects
• Physical or virtual
• Static or dynamic
• Domain specific examples
• Frontline in war
• Offside line in soccer
• Line-of-scrimmage in football

29
Team-Oriented Feature Functions

• Extract features relative to the team
• Specifically designed for teamwork activity
  recognition

30
Centroid-Relative Position Vector (CRPV)

• Positions are calculated relative the centroid
  position and orientation
• Dimensionality is reduced (compared to previous
  approach)
• Translation, rotation and scale invariant
• An evolution is the Role-Relative Position Vector
  (RRPV)
• Privileged agent

31
Cohesion

• Measures the bonding together of the team
• Derived from the Principal Component Analysis
• PCA Dimensionality is reduced by restricting
  attention to the directions along the scatter
  cloud that are the greatest
• Calculate eigenvalues and eigenvectors from the
  position-based scatter matrix
• In the 2D case there are two eigenvalues and two
  eigenvectors
• Cohesion is the maximum eigenvalue
• CohesionDirection is the direction of the
  eigenvector with maximum eigenvalue
• CohesionGradient is the change in cohesion over a
  sliding window

32
Cohesion

• Position-based scatter matrix

33
Team-Oriented Feature Functions

• Agent-oriented features can be used by replacing
  the team with a virtual agent following the team
  centroid

34
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

35
Role-Based Teamwork Activity Recognition

• The goal is to improve robustness of the teamwork
  recognizer
• What are the role assignments in the team?
• The baseline HMM input we presented previously is
  a fixed arrangement
• 1, 2, 3, 4 not the same as 2, 3, 4, 1
• Brute force solution
• Inefficient One HMM for each permutation
• Role recognition module
• Represent teamwork activity based on roles,
  rather than by agents

36
Extended Workflow Overview
37
Learning Role Models

• Role models are represented by decision trees
• Trained from observations with ID3 algorithm
• Pruned to minimize effects of overfitting
• Intuitive (white-box)
• Visualizes the exact features which were used in
  classification
• Follow leader uses CRPV
• Feature functions for role recognition
• Agent-oriented feature functions
• CRPV feature (assuming that bystander agents are
  filtered out)

38
Role Recognition

• Calculate the role assignment probability Pr( ai
  , rj )
• The probability that agent ai plays role rj
• Extract observation sequences from the movement
  trace of each agent
• Input each observation in the sequence to the
  decision tree classifier
• The output is a sequence of class frequency
  vectors
• Role assignment probability

39
Role Assignment and Mapping

• Identify the best match of role to agent
  assignments by searching the role assignment
  probabilities
• Re-map the team-oriented feature vectors
• Multiple role assignment
• Each agent can play multiple roles

40
Role Assignment and Mapping

• Unique role assignment
• Each agent can play one role

41
Experimental Setup
Activity Sequences Observations
Traveling column 29 319
Traveling line 30 330
Traveling box 33 363
Bounding overwatch 9 189
Wedge 17 187
Team split 15 192
Team merge 15 177

• Warfare exercise dataset
• Same as used previously
• Extended with seven activities
• Four agents and four roles
• Teamwork Activity Recognizer
• Gaussian Mixture Model (GMM)
• Multiple internal HMMs for each activity to
  accommodate for complexity variations in teamwork
  activities
• GPS readings are not always available (e.g.
  positions from opponents)
• Simulate noisy observations
• Offset position with a randomly generated number
  following the Gaussian distribution multiplied
  with a noise magnitude
• Mean accuracy and standard deviation was
  calculated using stratified 10-fold
  cross-validation

42
Results Creating the Idealized Team Actions

• Assume perfect role recognition
• Best parameter configuration
• Four hidden states
• Three mixture components

43
Results Creating the Idealized Team Actions

• Multiple internal HMMs for each teamwork activity
• Four hidden states
• Bar-plot show the distribution of correctly
  classified sequences over all HMMs for each
  teamwork activity

44
Results Performance Evaluation with Unknown
Team-Organization

• Shuffled input

45
Results Performance Evaluation with Unknown
Team-Organization

• Accuracy with noise and unknown team-organization
• 92.62 with standard deviation 5.53
• Previous accuracy was 82 (without noise)
• Role recognition improves robustness
• Team-oriented feature functions improves accuracy

46
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

47
Tuning the Performance of Teamwork Activity
Recognition

• Many components in the recognizers workflow are
  customizable
• HMM Number of hidden states and choice of
  emission probability representation
• Feature extraction Width of sliding window and
  discretization thresholds
• Role recognition module Choice of features,
  parameters in the ID3 learning algorithm and the
  choice of learning algorithm
• The probability density function (PDF) has a
  significant impact on the accuracy and robustness
  of the recognizer

Implementation Choice Multi-modal Continuous Parameters
Simple HMM Yes No States (n), Clusters (k)
HMM with Gaussian No Yes States (n)
HMM with GMM Yes Yes States (n), Components (C)
48
Simple Hidden Markov Model

• Discrete (histogram-based) probability density
  function
• Inputs are sequences of discrete values
• Vector quantization
• Compress observations
• Training of a codebook
• Cluster assignment
• Advantages
• Multi-modal distribution
• Disadvantages
• Extra cluster parameter to train codebook
• Can not estimate unseen observations
• Requires more training data

49
Hidden Markov Model with Gaussian PDF

• Used in the initial recognizer
• Advantages
• Continuous
• Can estimate unseen observations
• Disadvantages
• Unimodal distribution
• Example How to model U-turn to the left and
  right in the same HMM?
• Matrix inversion of covariance matrix can be
  problematic
• Reduce expressiveness by enforcing diagonal
  covariance matrices

50
Hidden Markov Model with GMM PDF

• Used in the extended recognizer
• Advantages
• Continuous
• Can estimate unseen observations
• Multi-modal distribution
• Disadvantages
• Extra parameter to determine number of mixture
  components
• Parameter estimation is computationally expensive

51
Results Simple HMM

• Best recognition accuracy
• 60 clusters with 18 hidden states
• Accuracy is 75.05 14.89
• Difficulties with
• Bounding overwatch
• Team split
• Team merge

52
Results HMM with Gaussian PDF

• Best recognition accuracy
• 3 hidden states
• Accuracy is 91.29 6.69
• Fails when using more than three states (not
  enough training data)
• Bounding overwatch
• Team split
• Team merge
• Reduce expressiveness of the PDF

53
Results HMM with GMM PDF

• Best recognition accuracy
• 3 hidden states
• 2 mixture components
• Accuracy is 93.90 4.68

54
Summary

• Simple HMM
• Performed poorly
• Filters out too much information
• HMM with Gaussian PDF
• Fails when using the general covariance matrix
  when n gt 3
• HMM with Gaussian Mixture Model PDF
• Best choice
• Least sensitive to changes in parameter
  configurations

Implementation Choice Mean Max Min Std. dev.
Simple HMM 62.91 75.05 44.52 5.44
HMM with Gaussian 83.27 91.28 77.76 4.29
HMM with GMM 88.30 93.90 83.19 2.17
55
Outline

• Introduction
• Data Acquisition and Knowledge Engineering
• Teamwork Activity Recognition using Hidden Markov
  Models
• Team-Oriented Feature Extraction
• Role-Based Teamwork Activity Recognition
• Tuning the Performance of Teamwork Activity
  Recognition
• Conclusions

56
Conclusions

• We developed a knowledge engineering tool for
  visualization, identification and extraction of
  teamwork datasets
• We developed a teamwork activity recognizer
  capable of encoding teamwork activity models from
  representative observations
• We improved accuracy of the recognizer using
  team-oriented feature functions
• The robustness of the recognizer was enhanced by
  integrating a role-recognition module in the
  recognition workflow
• We studied the importance of probability density
  function choices in our recognizer

57
Contributions

• Teamwork Scenario Editor
• Corpus of labeled teamwork activity
• Team-oriented feature extraction
• Learning teamwork activity from observation
• Role-based teamwork recognition
• Performance tuning of the teamwork recognizer

58
Future Work

• Team membership assignment
• Larger teams with 100 or more agents
• Preliminary studies
• Kalman filter
• Noise reduction of observation data
• Fusing information from multiple (uncertain)
  sources
• Applied to estimate life-expectancy in a disaster
  response simulation
• Genetic programming approach to learn agent
  strategies
• Applied to a simple foraging game in the single
  agent domain

59
Publications

• Teamwork Recognition of Embodied Agents with
  hidden Markov models. L. J. Luotsinen, H.
  Fernlund and L. Bölöni, IEEE 3rd International
  Conference on Intelligent Computer Communication
  and Processing (ICCP07), September 6-8, 2007.
• Automatic Annotation of Team Actions in
  Observations of Embodied Agents. L. J. Luotsinen
  and H. Fernlund and L. Bölöni, Sixth
  International Conference on Autonomous Agents and
  Multiagent Systems (AAMAS-2007), May, 2007,
  Honolulu, Hawaii.
• A Study of the Robustness of Agent Performance in
  Nine Popular Agent Implementation Paradigms. L.
  J. Luotsinen, M. A. Khan and L. Bölöni, IEEE 3rd
  International Conference on Intelligent Computer
  Communication and Processing (ICCP07), September
  6-8, 2007.
• A Comparison Study of Twelve Paradigms for
  Developing Embodied Agents. Ladislau Bölöni, L.
  J. Luotsinen, Joakim N. Ekblad, T. Ryan
  Fitz-Gibbon, Charles Houchin, Justin Key, Majid
  Ali Khan, Jin Lyu, Johann Nguyen, Rex Oleson,
  Gary Stein, Scott Vander Welde, and Viet Trinh.
  Software Practice and Experience, 2007.
• Comparing Apples with Oranges Evaluating Twelve
  Paradigms of Agency (Book chapter), L. J.
  Luotsinen, J. N. Ekblad, T. R. F. Gibbon, C.
  Houchin, J. Key, M. A. Khan, J. Lyu, J. Nguyen,
  R. Oleson, G. Stein, S. V. Weide, V. Trinh and L.
  Bölöni, ProMAS-2006, Fourth International
  Workshop on Programming Multi-Agent Systems, LNAI
  4411, pp. 95-114, 2007.
• Comparing Apples with Oranges Evaluating Twelve
  Paradigms of Agency. L. J. Luotsinen, Joakim N.
  Ekblad, T. Ryan Fitz Gibbon, Charles Houchin,
  Justin Key, Majid Ali Khan, Jin Lyu, Johann
  Nguyen, Rex Oleson, Gary Stein, Scott Vander
  Weide, Viet Trinh and Ladislau Bölöni,
  ProMAS-2006, Fourth International Workshop on
  Programming Multi-Agent Systems, Hakodate, Japan,
  May, 2006.
• A Two-Stage Genetic Programming Approach for
  Non-Player Characters. L. J. Luotsinen, Joakim N.
  Ekblad, Annie S. Wu, Avelino J. Gonzalez and
  Ladislau Bölöni, FuturePlay, The International
  Academic Conference on the Future of Game Design
  and Technology, East Lansing, Michigan, October,
  2005.
• Collaborative UAV Exploration of Hostile
  Environments. L. J. Luotsinen, Avelino J.
  Gonzalez and Ladislau Bölöni, 24th Army Science
  Conference, Orlando, Florida, December, 2004.

60
Publications (Pending)

• Role-Based Teamwork Activity Recognition in
  Observations of Embodied Agent Actions. L. J.
  Luotsinen and L. Bölöni, Submitted to Seventh
  International Conference on Autonomous Agents and
  Multi-Agent Systems (AAMAS-2008), May, 2008,
  Estoril, Portugal.
• A Robust Method for Estimating Noisy Measurements
  Applied to Disaster Response Operations. L. J.
  Luotsinen, M. A. Khan and L. Bölöni, 2008.
• Building a World Model under Communication
  Constraints for Disaster Response Applications.
  M. A. Khan, L. J. Luotsinen and L. Bölöni, 2008.

61
Questions?
62
Markov Models vs. Hidden Markov Models

• Markov Model vs. Hidden Markov Model
• Regular person can directly determine the state
  of the weather
• If it is rainy, sunny or cloudy
• In this case the states are visible and can be
  modeled using a Markov model
• A prisoner in jail
• Can not directly observe these states
• However the prisoner might be able to observe
  that water is dropping from the ceiling, air is
  dry or the temperature and so on
• In this case the Markov model is hidden
• We cannot directly observe the states of the team
• Actually we don not really know what they are
• Some people refer to the hidden states as the
  mental model of the team and the visible states
  as the physical model of the team
• We can only observe the physical teamwork states
• Velocity and acceleration

63
Markov Process

• HMM assumes that the teamwork activity being
  modeled is a Markov process
• It has the Markov property
• The hidden state is dependent only on the
  previous hidden state
• The visible state is dependent only on the
  current hidden state

64
How to Calculate Pr( ai , rj )

• Input NorthWest, North, Low, North, High,
  Constant
• Acquired from agent ai
• Output f2,4,0,0, f66,113,0,0,
  f34,7,0,0
• Class frequency vectors from the decision tree
• Total sum 2466113327 226
• Sum role 1 26634 102
• Sum role 2 41137 124
• Sum role 3 000 0
• Sum role 4 000 0
• Pr( ai , r1 ) 102/226 0.45
• Pr( ai , r2 ) 124/226 0.55
• Pr( ai , r3 ) 0/226 0.0
• Pr( ai , r4 ) 0/226 0.0

65
Multiple Role Assignment

• An agent can play multiple roles
• E.g. the agent is playing the role of a leader at
  the same time as it is playing the role of a
  flank protector
• Role assignments are selected by maximum
  probability
• r1? a1
• r2? a1
• r3? a4
• r4? a3
• a1 is playing two roles!

r1 r2 r3 r4
a1 0.48 0.52 0.0 0.0
a2 0.43 0.49 0.02 0.06
a3 0.01 0.02 0.49 0.48
a4 0.03 0.02 0.50 0.45
66
Unique Role Assignment

• An agent is limited to one role only
• Find candidate assignments (green)
• r2? a1
• r2? a2
• r4? a3
• r3? a4
• Resolve conflict (red)
• r2? a1
• Next iteration
• r1? a2

Conflict!
r1 r2 r3 r4
a1 0.48 0.52 0.0 0.0
a2 0.43 0.49 0.02 0.06
a3 0.01 0.02 0.48 0.49
a4 0.03 0.02 0.50 0.45