CAP6938 Neuroevolution and Artificial Embryogeny Real-time NEAT

1
CAP6938Neuroevolution and Artificial
EmbryogenyReal-time NEAT

• Dr. Kenneth Stanley
• February 22, 2006

2
Generations May Not Always Be Appropriate

• When a population is evaluated simultaneously
• Many are observable at the same time
• Therefore, entire population would change at once
• A sudden change is incongruous, highly noticeable
• When a human interacts with one individual at a
  time
• Want things to improve constantly

3
Steady State GA One Individual Is Replaced at a
Time

• Start by evaluating entire first generation
• Then continually pick one to remove, replace it
  with child of the best

Start Evaluate All
2) Create offpsring from good parents
Repeat
3) Replace removed individual
1) Remove poor individual
4
Steady State During Simultaneous Evaluation
Similar but not Identical

• Several new issues when evolution is real-time
• Evaluation is asynchronous
• When to replace?
• How to assign fitness?
• How to display changes

5
Regular NEAT Introduces Additional Challenges for
Real Time

• Speciation equations based on generations
• No remove worst operation defined in algorithm
• Dynamic compatibility thresholding assumes
  generations

6
Speciation Equations Based on Generations
7
How to Remove the Worst?

• No such operation in generational NEAT
• Worst often may often be a new species
• Removing it would destroy protection of
  innovation
• Loss of regular NEAT dynamics

8
Dynamic Compatibility Thresholding Assumes A Next
Generation
9
Real-time NEAT Addresses Both the Steady State
and Simultaneity Issues

• Real-time speciation
• Simultaneous and asynchronous evaluation
• Steady state replacement
• Fast enough to change while a game is played
• Equivalent dynamics to regular NEAT

10
Main Loop (Non-Generational)
11
Choosing the Parent Species
12
Finally How Many Ticks Between Replacements?

• Intuitions
• The more often replacement occurs, the fewer are
  eligible
• The larger the population, the more are eligible
• The high the age of maturity, the fewer are
  eligible

13
rtNEAT Is Implemented In NERO

• Download at http//nerogame.org
• rtNEAT source soon available (TBA)
• Simulated demos have public appeal
• Over 50,000 downloads
• Appeared on Slashdot
• Best Paper Award in Computational Intelligence
  and Games
• Independent Games Festival Best Student Game
  Award
• rtNEAT licensed
• Worldwide media coverage

14
NERO NeuroEvolving Robotic Operatives

• NPCs improve in real time as game is played
• Player can train AI for goal and style of play
• Each AI Unit Has Unique NN

15
NERO Battle Mode

• After training, evolved behaviors are saved
• Player assembles team of trained agents
• Team is tested in battle against opponents team

16
NERO Training The Factory

• Reduces noise during evaluation
• All evaluations start out similarly
• Robot bodies produced by factory
• Each body sent back to factory to respawn
• Bodies retain their NN unless chosen for
  replacement
• NNs have different ages
• Fitness is diminishing average of spawn trials

17
NERO Inputs and Outputs
18
Enemy/Friend Radars
19
Enemy On-Target Sensor
20
Object Rangefinder Sensors
21
Enemy Line-of-Fire Sensors
22
Further Applications?

• New kinds of games
• New kinds of AI in games
• New kinds of real-time simulations
• Training applications
• Interactive steady-state evolution

23
Next Topic Improving the neural model

• Adaptive neural networks
• Change over a lifetime
• Leaky integrator neurons and CTRNN

Evolutionary Robots with On-line
Self-Organization and Behavioral Fitness by Dario
Floreano and Joseba Urzelai (2000)Evolving
Adaptive Neural Networks with and Without
Adaptive Synapses by Kenneth O. Stanley, Bobby D.
Bryant, and Risto Miikkulainen (2003)
New Homework due 3/8/06 (see next slide)
24
Homework Due 3/8/06

• Genetic operators all working
• Mating two genomes mate_multipoint,
  mate_multipoint_avg, others
• Compatibility measuring return distance of two
  genomes from each other based on coefficients in
  compatibility equation and historical markings
• Structural mutations mutate_add_link,
  mutate_add_node, others
• Weight/parameter mutations mutate_link_weights,
  mutating other parameters
• Special mutations mutate_link_enable_toggle
  (toggle enable flag), etc.
• Special restrictions control probability of
  certain types of mutations such as adding a
  recurrent connection vs. a feedforward connection
  
• Turn in summary, code, and examples demonstrating
  that all functions work. Must include checks that
  phenotypes from genotypes that are new or altered
  are created properly and work.

25
Project Milestones (25 of grade)

• 2/6 Initial proposal and project description
• 2/15 Domain and phenotype code and examples
• 2/27 Genes and Genotype to Phenotype mapping
• 3/8 Genetic operators all working
• 3/27 Population level and main loop working
• 4/10 Final project and presentation due (75 of
  grade)