Neural Network Learning and Construction Techniques

1
Neural Network Learning and Construction
Techniques
(A Brief Outline - One Previous and Current
Research of Interest)

Vijanth S. Asirvadam Faculty
of Information Science and Technology
Multimedia University
Melaka
Campus
2
Research Outline gt Variable Memory in Offline
Neural Network Training gt Sliding Window
Technique Progression towards online neural
learning Data Store Management
Techniques gt Recursive Neural Network Training
Algorithms Separable approach (hybrid of
linear and nonlinear optimisation)
Decomposed methods (by Neuron and by Layer)
Insert Direct Link gt Sequential Learning -
dynamic neural network learning method using RBF
network
3
Variable Memory in Offline Neural Network Training

• Feedforward neural network training can be posed
  as an unconstrained optimization problem.
• Objective is to minimize a sum squared error
  performance index

• Algorithms discussed are applicable to general
  feed-forward neural networks
• with linear output layer parameters

4
Variable Memory in Offline Neural Network Training
Weight Update
BFGS Update
5
Variable Memory in Offline Neural Network Training
Memory-Less BFGS
Variable Memory BFGS
6
Variable Memory in Offline Neural Network Training
Optimal Memory BFGS
7
Sliding Window Learning for MLP Network
Weight update
8
Sliding Window Learning for MLP Network
9
Recursive Learning Approach on MLP Network

• Recursive training schemes attempt to minimize
  the performance index by
• accumulating information from successive
  training data which are generated
• online.
• Stochastic back propagation (SBP) weight
  update based on the input available
• at the tth sample instant

• Second order recursive estimation weights
  update is given as

Inverse of Rt is usually estimated as
, which can interpreted as covariance
matrix, using recursive prediction algorithm a)
Recursive Gauss-Newton b) Recursive
Levenberg-Marquardt
10
Recursive Learning Approach on MLP Network
Separable Recursive Training 1. Hybrid recursive
training is archived by separating the linear and
nonlinear weights and using the most
appropriate algorithms to optimize each set
simultaneously. 2. The nonlinear weight are
optimized using recursive nonlinear optimization
e.g. a) Recursive Gauss Newton ( RPE)
b) Recursive Levenberg-Marquardt (RLM) 3.
The linear weights meanwhile are estimated using
linear optimization e.g. a) Recursive Least
Square (RLS) 4. Separable training methods
resulting from combining RPE and RLM, with
linear optimization method, RLS, to form
a) Hybrid Recursive Prediction Error (HRPE)
b) Hybrid Recursive Levenberg-Marquardt (HRLM)
5. To implement hybrid version of recursive
neural network training, the output gradient
vector and the covariance matrix P
must be decomposed.
11
Sequential Learning using Dynamic RBF Network

• RBF network is popularly used as a neural-net
  tool by linear combination of its localised
  Gaussian function.
• Localised basis function learn information at one
  operating point without degrading information
  accumulated at to other operating point (which
  will be the case for globalised basis function
  such as sigmoidal/tangent hyperbolic)
• Main concern of RBF network is when choosing
  centre for Gaussian kernel
• Case of dimensionality if each input vector is
  set as Gaussian centre
• Prior selection of Gaussian centres using
  clustering techniques or other optimisation gives
  no guarantee that centres will be appropriately
  placed if the plant dynamic changes from one
  operating pint regime to another.

• Sequential Learning or Resource Allocation
  Network for RBF network with Gaussian functions
  is to address these problems

12
Chronologies of Sequential Learning (RAN)

• First pioneering work on sequential learning
  Platt J., A Resource-Allocating Network for
  Function Interpolation , Neural Computation,
  vol.. 3, pp. 213-225, 1991.
• The term sequential learning is derived in this
  paper and it is similar with Platts RAN with the
  network weights are updated using second order
  method (using EKF) Kadirkamanathan V. and
  Niranjan M., A Function Estimation Approach to
  Sequential Learning with Neural Networks, vol..
  5, no. 6, pp. 928-934,1993
• The sequential learning method is first applied
  on a control problem Bomberger J.D. and Seborg
  D.E., On-line Updating of Radial Basis Function
  Network Models, IFAC Nonlinear Control Systems
  Design, Tahoe City, California, 1995
• The sequential learning is combined with pruning
  with replacement. The first paper on dynamic RBF
  pruning Molina C. and Niranjan M., Pruning
  with Replacement on Limited Resource Allocating
  Networks by F-Projections., Neural Computation,
  vol.. 8, pp. 855-868, 1996
• Using dynamic pruning method where the size of
  the network varies during training. The method
  known as Minimal RAN (MRAN) has been successful
  applied many in real world application Yingwei
  L., Sundararajan N. and A. Saratchandran P.,
  Identification of Time-Varying Nonlinear Systems
  Using Minimal Radial Basis Function Neural
  Networks., IEE Proceedings Control Theory
  Application, vol.. 144, no. 2, pp. 202-208, March
  1997.

13
Chronologies of Sequential Learning (RAN)

• Direct Link MRAN (DMRAN) using full covariance
  matrix update where the DMRAN show better
  performance compared to MRAN method Yonghong
  S., Saratchandran P., Sundararajan N. A Direct
  Link Minimal Resource Allocation Network for
  Adaptive Noise Cancellation, Neural Processing
  Letters, vol.. 12, no.3, pp. 255-265, 2000
• A method known as Extended MRAN (EMRAN) is
  proposed where is the weight update is limited to
  winner neuron Li Y, Sundararajan N. and
  Saratchandran P., Analysis of Minimal Radial
  Basis Function Network Algorithm for Real-Time
  Identification of Nonlinear Dynamic Systems., IEE
  Proceedings Control Theory Application, vol..
  147, no. 4, pp. 476-484, July 2000.
• Discussion paper on varies decomposed training
  algorithms using full and minimal weight update
  applied on RBF and DRBF Asirvadam V.S.,
  Minimal Update Sequential Learning Algorithms
  for RBF Netowrks, United Kingdom Automatic
  Control Council Conference (Control 2002), pp.
  71-76, September 10-12, 2002.
• This paper shows direct-Link RAN using decomposed
  EKF (DRAN-DEKF) showed better performance
  compared to MRAN with minimal amount of
  computation and memory requirement. Asirvadam
  Vijanth S., Seán F. McLoone, George W. Irwin,
  Sequential Learning using Direct Link
  Decomposed RBF Neural Network Preprints of the
  IFAC International Conference on Intelligent
  Control Systems and Signal Processing, Faro,
  Portugal, pp. 107-112, April 8-12, 2003.

14
Sequential Learning of RBF Network

• Sequential learning combines the new centre
  allocation with weight updating in one routine.
• In sequential learning two main growth criteria
  is given (McLoone 2000)

is the distance between the input vector
and the centre of the nearest neuron,
width of the nearest neuron.
determine the Gaussian locality range.
user defined parameters.
15
Sequential Learning of RBF Network

• If the two growth criteria are NOT satisfied then
  all the the network parameters are adjusted
  (e.g. using recursive prediction error (RPE)
  method or also known as extended Kalman Filter
  (EKF))

• On the other hand if both the growth criteria is
  satisfied then a new Gaussian kernel is assigned
  as follows

determine the Gaussian locality
range.

• The resulting algorithm known is known as RAN-EKF
  

16
Pruning for RBF Network

• Pruning strategy for RBF (YingWei et al. 1997)
  eliminates the Gaussian function which
  contributes the least to the model output.
• The pruning method for RBF is given as follows
  Compute all the output of the Gaussian Kernel, I
  1,2m

• Determine the largest absolute Gaussian basis
  function.

• Calculate the normalised contribution factor.

• If for M consecutive sample
  instances then eliminate jth Gaussian kernel

• Combining RAN-EKF and the pruning method, Minimal
  RAN (MRAN) is derived