Distributed decisionmaking and reasoning with uncertain image and sensor data

1
Distributed decision-making and reasoning with
uncertain image and sensor data

• Pramod K. Varshney
• Kishan G. Mehrotra
• C. Krishna Mohan
• Electrical Engineering and Computer Science Dept.
• Syracuse University
• Syracuse, NY 13244
• Phone [phone]
• Email varshney_at_syr.edu

2
Feedback from 2002 Review of MURI Project

• Show a stronger connection with military
  scenarios
• Urban warfare is a good scenario to consider
• Relate theoretical work to military applications

3
Our Main Themes

• Develop theory and algorithms oriented towards
  practical military applications
• Decision-making with multiple decentralized
  information streams
• Uncertainty computation
• Reasoning with uncertain data
• Exploitation of models and tools developed by
  partners

4
What is the agents current location ?
Recognize activities of other agents.
What is the likelihood of damage at various
locations ?
What would be the safest paths to a goal/exit
zone ?
5
Main Contributions

• 1. Temporal Bayesian network for target tracking
• 2. Outdoor video tracking using multiple cameras
• 3. Image query-based approach for location
  determination
• 4. Distributed personnel movement planning for
  urban combat zones
• 5. Automated event/scenario recognition from
  video sequences

6
1. Target Tracking

• -An Application of Temporal Bayesian Networks

7
Previous Work

• Defined Temporal Bayesian Networks
  time-dependent updating of beliefs at nodes based
  on information obtained at earlier instants.
• Formulated linear exponential temporal decay
  model.
• Implemented the above model in MATLAB.
• Developed a graphical user interface.

8
Target Tracking Application

• Model The field contains one or more targets,
  and is divided into many rectangular cells.
• Data Each sensor obtains information from a
  distinct subset of cells in its neighborhood,
  updated over time based on the latest data.
• Goal The fusion center must determine if targets
  exist in any cells, based on various sensor
  reports at current and recent time instants.

9
(No Transcript)
10
Causal Network Structure

• Procedure
• Compute beliefs associated with each lower level
  node Oi
• Temporal belief updating at T using timestamps
  associated with information from lower level
  nodes.
• Decision rules to determine target presence.

11
Decision Rules

• Any cell L satisfying the following two
  conditions is considered as a target cell
• Belief (target in cell L) gt Threshold?
• Belief (target in cell L) gt Belief (target in
  each cell neighboring L)?

12
Belief Updating

• Belief updates depend on the length of the time
  interval since the observation.
• If the target is believed to be in cell i at time
  t, then this increases the belief that the target
  is near cell i at time t1.

where
13
Example

• Grid size 30X30 cells
• 36 sensors are uniformly distributed through the
  grid, and each sensor covers 5X5 cells.
• Three targets in the field.
• Results Low error rate in tracking quality upto
  0dB SNR.

14
(No Transcript)
15
Error Rate vs. SNR
16
2. Video Surveillance

• - Outdoor Tracking Using Multiple Cameras

17
Goal and Approach

• Goal Monitoring large outdoor areas
• Problem Difficult to deal with changing
  illumination and weather conditions.
• Our approach
• Heterogeneous sensors (e.g. optical and IR)
• Automatic camera selection
• Measurement fusion techniques

18
Key Steps

• Object tracking within the field of view of the
  connected sensors.
• Automatic camera selection based on appearance
  ratio.
• Data assignment based on gating.
• Measurement fusion using a Kalman filter.

19
Experimental Results

• Tracking two persons with a B/W and a Color sensor

20
Results
B/W Sensor
Data Fusion
Color Sensor
Mean and s.d. of estimation error (in pixels)
21
3. Location Determination

• - Image Query-Based Approach

22
Goal

• Given a city model, find the location based on
  images of the surrounding area
• The city model is composed of street scenes, such
  as the Berkeley model (A. Zakhor et al.)

23
Approach

• Segment the street scene into reference images so
  that one building is in one reference image
• Given a query image, determine its location in
  the city model

24
Matching Process

• Stage I
• Co-occurrence matrix used for initial culling
• Stage II
• Co-occurrence matrix and Gabor filter followed by
  feature-level fusion for matching
• Output
• Locations of sub-images ( , ) in the city
  model

25
Post-Processing

• Case I
• The two sub-images belong to the same building
  (segmentation may have been wrong)
• Case II -1
• The expected result
• , provide the final location
• Case III otherwise
• Weighted summation of the distances of the two
  sub-images to reference images in the feature
  space
• The shortest one is the final location

• Query Images

• Matching Results

26
4. Personnel movement planning in urban combat
zones

• - Path computation algorithms
  for risk minimization

27
Goal and Approach

• Goal To find the safest routes for personnel to
  a target or safe zone
• Model Geographic information (models)
  represented using a graph whose nodes represent
  locations associated with uncertain risk
  estimates
• Approach Maximize probability of traversing a
  path without damage, viz.,
• n
• ?(1- risk(Li)) for the path (L1, , Ln)
• i1

28
Problems

• Personnel on the battlefield need to use the best
  paths computed within a given time.
• Risk estimates change with time, hence best paths
  cannot be precomputed and reused.
• Tradeoff Solution quality vs. Computation time.

29
Iterative Improvement Algorithm

• Shortest paths are computed from a given source
  point to various goal nodes.
• Repeat
• These paths are incrementally modified and
  evaluated
• The modified path replaces the previous path if
  it is of better quality
• Until further small modifications do not improve
  quality.

30
Stochastic Algorithm for Path Planning

• Multiple candidate paths are explored in
  parallel, and successively mutated until
  computational limits are reached.
• A mutation to a candidate path (parent) is
  accepted with a probability that depends on the
  quality of mutated path, its parent, and the
  amount of computation performed so far.

31
Simulations

• Geography A 100X100 grid, with 15 goal points on
  the periphery of the grid.
• The risk estimates were generated from a beta
  distribution with shape parameters (1,5).
• Results reported are averages over the paths
  obtained for 100 randomly chosen source points.

32
Mutation Example
33
Results
34
Solution quality vs. computational effort
35
Observations

• The number of possible paths is too large to
  permit computation of optimal solutions in real
  time.
• Iterative Improvement algorithms can get stuck in
  local optima.
• Best results were obtained using the stochastic
  algorithm.
• Computations can be terminated at any point, and
  fairly good (though suboptimal) solutions are
  obtained very quickly.

36
5. Scenario Recognition

• Feature Extraction and Classification from Video
  Image Sequences

37
Motivation and Goals

• Tasks
• Understand what is happening around agent in
  battlefield or urban combat zone
• Predict behavior of other agents
• Detect anomalies from expected behavior
• Modify behavior models with time
• Current Focus Recognition of the activities
  depicted in a sequence of images

38
A video sample for analysis and feature extraction
39
Main Steps in Recognition

• Detection and tracking of moving objects
• Extraction of features relevant to recognition
  task
• Classification of simple events over short time
  intervals
• Recognition of sub-scenarios consisting of
  multiple events
• Identification of scenarios consisting of
  multiple sub-scenarios

40
Object Detection and Tracking
Background model
Input image sequences
Difference the object from Background
Object detection
Object tracking
Features for sub-scenario recognition
Object template
Sub-Scenario Recognition
41
Identifying the Location of a Person in a
Sequence of Images

Illumination Effects
Frame 112
Frame 141
Shadows
Frame 172
Frame 346
42
Central Theme

• Sequences of values for features extracted from
  successive images are viewed as time series
  (control charts).
• Statistical, A.I. and signal processing
  techniques are used to formulate decision-making
  criteria from these time series.
• Subtasks
• Establishing temporal associations between
  features and (sub)scenarios.
• Detecting where each sub-scenario begins and
  ends.

43
Methods to Obtain Decision Criteria

• Partition ranges of feature values into
  applicable intervals for each category.
• Develop fuzzy membership functions and
  corresponding rules.
• Apply decision tree learning algo. (C5)
• Extract frequency information using short-term
  Fourier Transform.

44
Illustrative example
Slowing down
Walking towards fountain
Features from image sequence
Standing still
Person drinking water
Bending to drink
Classifier
Walking away from fountain
Events
Scenario
Sub -scenarios
45
Detection of Sub-scenarios

• Example Categories for Classification walking,
  bending, running, standing, sitting
• Example Features
• Aspect Ratio (bounding box width/height)
• Area of bounding box
• Relative Upper Density(from 1-D Projection)
• Speed (measured over several frames)

46
1-D Projections

• Distribution of foreground pixels in the
  y-direction varies with activity
• Statistical characterizations of this
  distribution hence provide features useful for
  classification.

47
Control Chart Examples

48
Control Charts for Rule Production

• Approach Rules are obtained using percentile
  ranges of feature values from control charts
• Example rule
• If (0.29 ? AR ? 0.57) and (Height gt Width) and
• (Speed ? 0.5mph) and (Relative Upper Density
  ? 0.16) and (? Distance gt 0.2)
• then walking class
• Results 79 accuracy for 3 class problem

49
Fuzzy Rules

• Approach Fuzzy classification with trapezoidal
  membership functions
• Example Rule
• If velocity is low and aspect_variance is low
  and aspect ratio is high,
• then sitting class.
• Results 82 accuracy (5 classes)

50
Automated Learning

• Approach Apply C5 learning algorithm to obtain
  classification tree and rules.
• Results 92 correct classification (for 3
  sub-scenarios, i.e., standing, bending, walking)

51
Example Decision Tree obtained using C5
true
RUD 0.28
Walking
false
true
true
RUD .28
RUD 0.25
Standing
AR 0.3
Walking
false
true
RUD 0.38
RUD 0.39
RUD 0.39
Bending
false
true
RUD 0.40
AR 0.11
Bending
false
AR 0.13
RUD-Relative Upper Density AR aspect ratio
true
Walking
AR 0.3
false
false
Standing
AR 0.32
52
Frequency-based Analysis

• Motivation Useful information is available in
  the frequency domain, but not the time domain.
• Principle Each category of activity is
  characterized by a distinctive frequency spectrum
  signature.

53
Spectrum-based Classification

• Goal Identify and categorize frequency
  components in the spectrum.
• Example rule
• If max(abs(coeffs(20-40)))gt0.3,
• then walking class
• Results 90 accuracy for a 3-class problem,
  using the spectrum of a single feature (aspect
  ratio)

54
Next Steps

• Recognizing complex scenarios with multiple
  moving objects and overlapping sub-scenarios
• Extract more features useful for classification
• More complex events
• Multi-pass system with feedback

55
Future Work

• Temporal Bayesian reasoning for dynamic route
  planning
• Learning algorithms to determine parameters of
  temporal Bayesian networks
• Generation of high level descriptions from video
  image sequences
• Detection of unusual activities from video image
  sequences
• Extraction of uncertain information from
  temporally contiguous images

56
Appearance Ratio

• This is an effective metric that indicates how
  well the target is detected and how well blobs
  are extracted.

Difference Map
Blob
Normalization with respect to image threshold
57
Measurement Gating
The gating distance (GD) could be chosen
considering the Mahalanobis distance between
each arriving observation and the predicted
target position given by
where is the residual vector and S
is the innovation covariance matrix. GD should be
taken so that
58
Measurement Fusion

• The Kalman filter is used to fuse sensor
  measurements (objects map coordinates).
• Measurements are weighted according to the
  associated AR measure The measurement error
  covariance matrix of the Kalman Filter is defined
  as
• where c is an adjustable constant that can be
  obtained via experiments.

59
Image Segmentation to Construct

Reference Image Database

• Transformation to HSV space
• Metric based on Euclidian distance between
  histogram of either side of a potential boundary

60
Image Segmentation to Construct Reference Image
Database

• Obtain left and right feature of each column as a
  form of vectors,
• where,
• represent thenormalized histogram of the
  region j in the HSV image space
• Compute the weighted Euclidian distance of
  to
• The maximum distances are considered as the
  boundaries of building.

61
Segmentation of Query Images

• Edge Detection of the query image
• Hough transform to detect vertical straight lines
• If the line is long enough to be considered as
  the boundary of the building, segment the query
  image into two sub-images

62
Stochastic Algorithmfor Path Planning

• The mutation is accepted if,
• offspring quality/parent quality gtc
• where c is a number chosen randomly from a
  uniform distribution in the interval (a,1) where
  a is defined by,
• a(c(1-c)t/max epsilon)
• c0.7,epsilon0.001 and maxnumber of
  evaluations after which the algorithm is to be
  terminated.

63
Mutation

• A portion of the candidate path is changed by
  mutation operator.
• Two random points along a path are chosen .
• The path between these points is changed by
  choosing a random neighbor of the starting point
  and choosing successive random neighbors till the
  end point is reached.

64
Uniform Cost Search Algorithm

• Uniform cost search finds an optimal path by
  always expanding the lowest cost node .
• Starting form the source node, the algorithm
  expands by exploring neighbors of each successive
  node in terms of quality .
• The path found is guaranteed to be of optimal
  quality among all paths explored.