The Multiple Roles of Anticipation in Developmental Robotics

1
The Multiple Roles of Anticipation
inDevelopmental Robotics

• Douglas S. Blank 1
• Joshua M. Lewis 2
• James B. Marshall 2
• 1 Department of Computer Science, Bryn Mawr
  College, PA
• 2 Computer Science Program, Pomona College, CA

2
Uses of Anticipation

• Bootstrap learning of sensorimotor maps
• Work by Pierce and Kuipers (1997)
• Robot starts with uninterpreted sensory data and
  no knowledge of its own sensor types
• Analyzes correlations observed in sensory stream
• Learns to anticipate the effects of its motor
  commands on sensory features
• Develops a model of its own sensorimotor topology
  and the environment

3
Uses of Anticipation

• Source of internal reinforcement signals
• Work by Marshall, Blank, and Meeden (2004)
• Robot controlled by Simple Recurrent Network
• SRN generates next motor action and predicts next
  visual state
• Prediction error used to generate a reinforcement
  signal for training SRN using Complementary
  Reinforcement Backprop
• Robot learns to visually track a moving decoy

4
Uses of Anticipation

• Learning to distinguish learnable from
  unlearnable patterns with a neural net
• Some input patterns are inherently
  predictableInput 0 0 1 0 Target 0 1
• Some inputs are notInput 1 1 1 0 Target
  random bits
• Network can learn the predictable patterns more
  effectively by learning to anticipate its own
  errors and its own internal representations

5
Anticipation as a Learning Accelerator

• Curious developing robots are vulnerable to
  fixating on random aspects of their environment.
• How can we encourage the robot to discover that
  some parts of its environment are unlearnable?
• Anticipation plays a key role in our attempts at
  this kind of learning enhancement.

6
Problem Space

• Double XOR problem
• 0 75 noisy patterns
• Two flag bits at beginning of input indicate
  whether training sample is predictable

7
Network Architecture

• Standard three-layer backpropagation network with
  the output layer split into several components
• Utilizes two independent learning techniques
  Error Anticipation and Hidden Layer Anticipation

8
Error Anticipation

• Attempts to predict the error vector of another
  component of the output layer.
• Helped most when a large portion of the training
  set was noisy.

9
Hidden Layer Anticipation

• Attempts to predict the activation values of the
  hidden layer for the current training sample.
• Always provides a boost to learning, but most
  significantly in the presence of random data.

10
Experiment Design

• Compared several networks
• Standard backpropagation
• Error Anticipation only
• Hidden Layer Anticipation only
• Combined EAHLA
• 10 trials of 1200 epochs for each network.
• Targeted EA and HLA components to .5 if not in
  use. Network structure was kept the same.

11
Results 0 Noisy Patterns
12
Results 25 Noisy Patterns
13
Results 50 Noisy Patterns
14
Results 75 Noisy Patterns
15
Why Does Error Anticipation Work?

• We believe that error anticipation helps the
  network segregate random and deterministic
  training samples in the network's hidden layer
  representation.
• Os are random, Xs are deterministic

16
Why Does Hidden Layer Anticipation Work?

• In general we believe that HLA reinforces the
  path that learning was initially taking.
• As the desired problem is learned, the error from
  deterministic samples decreases and the random
  samples gain more weight.

17
Follow Up Experiment

• No flag bits
• Same input and output size as XOR problem
• Promising results at 75 noise

18
Conclusions and Future Work

• EA can help learning, but only if noise is
  present.
• HLA assists learning to a greater degree, most
  notably in environments with noise.
• Train networks on temporal XOR problems using
  recurrent networks with EA and HLA.
• Apply techniques to networks in simulated and
  real robotic environments.