SEASONs: Spiking, Entropic, Asynchronous, SelfOrganizing Neural Networks

1
SEASONsSpiking, Entropic, Asynchronous,Self-Org
anizing Neural Networks
Neural Networks with Cellular Automata

• On self-modifying machine learning systems
• Ph.D. project Ludovic A. Krundel
• Supervisors Dr. David J. Mulvaney
• Dr. Vassilios A. Chouliaras

2
Theme (talk emphasis)
introduction

• Innovative Emergence
• Connectionist approach
• Bottom-Up engineering (vs. conventional Top-Down
  engineering)
• Self-Modifying systems
• examples of physical implementation of NN on
  hardware device
• Aspects of the work
• Scalability of NN
• No learning algorithm-ready for more than 100s
  neurons
• Interconnects issues
• Incremental Learning trial-and-error processes (
  ? no learning without errors)
• Asynchronous operation of the microelectronic
  system

3
CPU ubiquitous
introduction

• established industrialised solution as general
  purpose processor
• Typical example of full Top-Down design
  engineering
• sequencer
• low software level ? _at_ ? high hardware level

4
CPU insight
introduction
Processing Unit
Memory
Data or Instruction
Instruction Register RI
Ordinal Counter CO
Sequencer (Automaton)
Sequencer (Automaton)
Address Register RAM
Exchange Register RE
Arithmetic and Logic Unit
Accus
Exchange Unit
5
CPU the good(?) old stuff?
introduction

• fatal processing error (e.g. wrong branching)
• not reliable for critical real-time system
  applications
• embedded software for real-time applications
  aircraft autopilot, space devices
• microcontrollers for medical systems
• (exhibits) untrue time-parallelism high
  surface/data-time-processing cost
• Instruction level parallelism
• Pipelining
• Superscalar design
• Thread level parallelism (MIMD)
• Multiprocessing
• Multithreading
• Data parallelism (SIMD) ? multimedia applications
• Vector processor (vectors of data)

subscalar
pipeline
superscalar
6
Manufacture of CPU
introduction

• ASIC design flow

7
Manufacture of CPU
introduction

• Photolithography of ASIC circuit
• CMOS Technology

8
FPGA (Field-Programmable Gate Array)
introduction

• Programmable Chip
• (Re-)Configurable Microelectronic Circuit
• True Time-Parallelism
• Massively parallel, highly distributed processing
  units as in ASIC
• Examples of application ASIC prototyping,
  upgradeable hardware device, labs experiments

9
FPGA insight
introduction
10
FPGA manufacture
introduction
11
SOC (System-On-a-Chip)
introduction

• True Time-Parallelism CPU
• Lower surface/data-time-processing costs

12
SOC examples of application
introduction
13
SOC state-of-the-art
introduction

• Embedded FPGA (eFPGA) www.menta.fr
• Embedded CPU (eCPU) Mitrionics
• Massively parallel eCPU on FPGA
• Supercomputing performance

14
Pure CA (Cellular Automata)
part 1

• Conventional CAM (Cellular Automata
  Machine) CA are a natural way of studying
  the evolution of large physical systems and
  constitute a general paradigm for parallel
  computation Toffoli and Margolus, 1987
• Clocked whole 2D CA landscape updated at each
  clock tick (nth-order systems/rules)
• Two-state cells e.g., alive or dead Conways
  Game of life
• Local (action-at-a-distance forbidden) and
  uniform (single rules recipe) systems rules

15
Examples of self-organizing CAs
part 1

• Innovative Emergence?
• In big CA systems, complex global behaviours can
  emerge! S. Wolfram, 2002

Dendritic CA ?
Neuron-like CAs
16
NNs with CA
part 1

• CA can be configured to create functions
• Non-Linear Dynamics and Chaos
• Organized complexity ? innovative emergence
• Substrate for highly adaptive NNs

17
EHW
part 1

• Intrinsic vs. Extrinsic EHW
• Evolving Neural Networks based on Cellular
  Automata in hardware
• Genetic Algorithms

chromosome
bit-stream
JTAG
18
The CBM
part 1

• The CAM Brain Machine H. de Garis, 1999
• CAM Cellular Automata Machine
• Growing CA-based NN modules for the behaviour of
  a robot cat Robokoneko
• Too many rules
• ? Need Rule Learning

19
CNN Cellular Neural Networks
part 1

• CNN for Machine Vision Pr. Leong Chua
• POEtic embryonic tissue
• Phylogenesis evolution
• Ontogenesis development
• Epigenesis learning

20
CNN POEtic embryonic tissue
part 1
21
CA-based NN
part 1

• Evolved Neural Networks based on Cellular
  Automata for sensory-motor controller Kim Cho,
  2005

Khepera
22
Cell Matrix
part 1

• The PIG Paradigm (Nicholas Macias, 1999)
• disappeared in 2005?
• FPGAs can actually also do it
• ICAP internal configuration access port
• Dynamic-reconfiguration efforts (IPFlex)
• Partial-reconfiguration efforts (JBits)
• Self-reconfiguration efforts (endo-reconfig)

23
Some Interesting Features of NNs
part 2

• Rules Extraction (from non-linear systems)
  Inductive Inference
• Learning and automatic generation of rules
  instead of evolutionary approaches
• Cognitive learning vs. blind learning

24
Entropy Comparison Engine
part 2

• Improvability Evaluation Self-Upgrades
• Similar concept to CPU sequencer systematic core
  engine
• Comparing NN system entropy with environmental
  information

25
Learning Algorithm
part 2

• Robot Controller sensor-activator
• Sensor Signals Fusion
• Unsupervised Learning
• Spiking Neurons
• Self-Adaptive NN
• Entropy-based NN re-organization
• Automatic generation of CA rules

26
Scalability
part 2

• Interconnects issues
• NOC (Network-On-a-Chip)
• But FPGAs are now dominated by interconnects,
  just like NNs
• For more than 100s neurons
• Modular architecture
• multiplicity of learning algorithms?
• Hierarchical architecture
• pseudo-fractal architecture?

27
Ongoing Work Design Approach
part 2

• Top-down design and debug of one cell
• Using conventional EDA tools and design flows
• Implement at hand a minimal set of initial rules
  basic rules
• Bottom-up assembly as multiple instantiations of
  totipotent cells in a modular-hierarchical
  fashion
• Using HDL (Balsa, Verilog) parameterization
  facilities
• recursive instantiation
• instantiation by name
• Real-Time Simulations of the global hardware
  system
• Parametric simulations
• Comprehensive analyses
• Goto 1. Eventual refinements of the cell design
  and initial rules

28
Cell Architecture
part 2
29
Possible Cell Structure
part 2
30
Self-configured CA-based NN
part 2

• Bottom-Up Engineering

31
Selected FPGA design flow
part 3
Xilinx ISE
32
Simulation with Balsa
part 3

• Spiking Entropic Asynchronous Self-Organizing
  Neural networks SEASONs
• Balsa development and simulation environment

33
Simulation with Balsa
part 3

• Balsa simulation environment

34
FPGA Development Board
part 3
35
Other Hardware Machine-Learning Techniques for
Self-Modifying Systems
discussion

• Use SystemC or Handel-C and write
  synthesizable(?) self-modifying code
• Perform software simulations
• Target hardware FPGA
• Perform hardware simulations and compare

36
Neuropeptides
discussion

• nervous system hormonal system (immune system)

37
Neuro-Modulation
discussion

• why hardware machine-learning

38
Main Points
conclusion

• CAM
• SEASON

• SOC
• ASIC / FPGA
• CPU
• Digital Spiking NN based on asynchronous CA that
  dynamically reconfigure themselves to suit the
  changing task at hand
• Aware learning vs. blind adaptation
• Systematic learning mechanisms should not
  dominate a machine-learning technique
• Innovative Emergence of self-adaptation

39
Acronyms

• COTS Commercially Off-The-Shelf
• EHW Evolvable HardWare
• CAM Cellular Automata Machine
• CBM CAM Brain Machine
• JTAG Joint Test Action Group
• ICAP Internal Configuration Access Port
• RTL Register Transfer Level
• EDIF Electronic Design Interchange Format
• HDL Hardware Description Language
• EDA Electronic Design Automation
• VLSI Very Large Scale Integration
• CMOS Complementary Metal Oxide Semiconductor

• FPGA Field-Programmable Gate Array
• CPLD Complex Programmable Logic Device
• ASIC Application Specific Integrated Circuit
• SOC System-On-a-Chip
• FSM Finite State Machine
• CPU Central Processing Unit
• DSP Digital Signal Processor
• NN Neural Network
• CA Cellular Automata