Sensor Fusion

1
Sensor Fusion

• Phillip Jones

2
Topics

• Sensor Fusion
• Biological Models
• Cognitive Science Models
• SFX Architecture
• Architecture
• Experiments

3
Taxonomy of Sensing Information

• Proprioception position of the body parts
  relative to an internal frame of reference
• Exproprioception position of the body relative
  to the layout of the environment
• Exteroception supplies information about the
  layout of the environment and the position of
  objects relative to the observer

4
Sensor Fusion

• Historical Aristotle said that the five senses
  converge into a common sense organ the heart
• Current Neurophysiologists and cognitive
  psychologists actively study sensory integration
  and intersensory perception

5
Why Sensor Fusion?

• Single-sensor robots have not been entirely
  successful in demanding tasks
• Multiple sensors are needed to reduce uncertainty
  when features are missing
• Intelligent Sensor Fusion autonomously gathers
  observations from multiple sensors and combines
  them into a single percept

6
Biological Models

• Lawrence Marks theorizes that there is a special
  neural mechanism to integrate multi-sensory
  information
• Studies of the Superior Colliculus in cats
  supports Marks theory
• Input neurons are specific to one sense
• Output neurons are multi-sensory
• Multiple weak stimuli produce greater output than
  single strong stimulus
• Includes inputs from the cerebral cortex which
  modulate behavior

7
Biological Models

• The Neurological model describes several aspects
  of sensor fusion relevant to robotics
• Sensor fusion couples perception with action
• Sensors can be combined in different ways for
  different percepts
• Sensor evidence accrues (not averaged)
• Sensors use sensor-specific representation until
  fusion

8
Cognitive Science Models

• Perceptual Modes
• Intention should cause the fusion process to
  select an appropriate method to the task
• Walking through a room
• Visual stimulus remains the same
• Walk through room avoiding obstacles
• Look for car keys

9
Bowers Taxonomy of Fusion Modes

• Complete Sensor Unity
• Fish swim upright with respect to gravity and the
  horizon line (water surface)
• Altering the horizon with prisms causes the fish
  to swim sideways, a compromise of the two stimuli
• Unity with awareness of discordance and the
  possibility of recalibration
• Reolof Illusion Visual evidence contradicts
  proprioceptive evidence of where the hand is
• Test subjects recalibrate to the new visual
  stimulus

10
Taxonomy of Fusion Modes

• Unity with awareness of discordance and tendency
  toward suppresion
• An Ames room has sloped floors but are painted
  such that they appear flat
• Subjects repress all senses but balance
• No unity at all
• Ventriloquism

11
Ramifications for Robotics

• Sensor integration is a form of closed-loop
  control
• Not all sensors contribute equally
• Control of sensory integration is separate from
  planning for perception

12
SFX Architecture

• Sensor Fusion Effects A biologically and
  Cognitively Plausible Robotic Architecture
• Hybrid Reactive/Deliberative
• Uses Schema Theory to establish
• an object-oriented framework for behaviors
• the formal role of perception within behavior

13
SFX Architecture Deliberative Layer

• Deliberative layer is divided into independent
  situated agents (modular)
• Mission Planner an interface agent
• Task Manager middle agent that instantiates
  reactive behaviors
• Sensing Manager middle agent that is
  responsible for deliberative activities in
  perception
• Cartographer information collecting agent

14
SFX Architecture

• Perceptual process is implemented as a perceptual
  schema
• A perceptual process is tailored for a specific
  percept by the sensing strategy
• Sensing strategy is a set of sensing plans and
  activation conditions
• Each plan is constructed manually and represented
  by directed acyclic graphs (DAGs)
• Dempster-Shafer theory is used as a basis for
  propagation and fusion

15
SFX Architecture Reactive Layer

• Uses motor schemas but not necessarily potential
  field representation
• Tactical behaviors arbitrate conflicts based on
  the immediate situation

16
SFX Architecture Design Principles

1. Software Engineering Principle A robot
   architecture should be composed of autonomous
   specialists which do one thing well
2. General Purpose Behavior principle General
   purpose behaviors should be general purpose
3. Skill Principle Collections of behaviors for
   routine tasks should be encapsulated into
   specialized behaviors
4. Murphys Law principle Sensors will fail, the
   environment will change, expectations will be
   wrong, and behaviors will be triggered at the
   wrong time
5. Real Time principle Robots must operate in
   real-time

17
Security Agent Experiment

• Denning DRV-1 mobile robot
• 4 Sensors
• Color camcorder
• BW video camera
• Infra-red camera
• Ultrasonic transducers
• Potentially discordant sensors served as
  candidates for recalibration

18
Security Agent Experiment

• Use fusion states to monitor three scenes
• Student desk scene
• Sensor Fusion
• VCR and monitor scene
• Sensor suppression
• Drill press scene
• Sensor calibration

19
Security Agent Results

• Outperformed a state-less agent in 9 of 13
  experiments
• Fusion states permit the fusion of observations
  when a state-less implementation could not
• Improves sensing quality the infra-red camera
  compensated for thermal drift using feedback from
  other sensors
• Improves the overall belief in a percept

20
UGV Competition

• 1994
• Vision system uses brightest pixels to find the
  road boundary
• 2 reactive behaviors
• Follow-path
• Move-ahead
• Brittle Judges white shoe caused problems

21
UGV Competition

• 1995
• Image is divided into 6 regions and analyzed to
  find road
• 4 behaviors
• Follow-path avoid-obstacle
• Pan-camera speed-control
• Lack of contrast between dead grass and painted
  lines caused errors
• Pan-camera behavior was implemented as if it used
  exteroception but really used exproprioception.

22
Guidelines for action-oriented Perception

• Decompose the task(s) into functional desired
  activities
• Establish the perceptual context
• Define the sensing needs for each activity in
  terms of proprioception, exteroception, and
  exproprioception
• Determine whether each sensing objective can be
  accomplished via direct or model-based methods

23
?