How to Make a Computer Think for You

1
How to Make a Computer Think for You

• Jeff Knisley, The Institute for Quantitative
  Biology, East Tennessee State University
• ALABAMA MAA STATE DINNER AND LECTURE, Feb, 2006

2
This is a Neuron
Signals Propagate from Dendrites to Soma
Signals Decay at Soma if below a Certain
threshold
3
Signals May Arrive Close Together
If threshold exceeded, then neuron fires,
sending a signal along its axon.
4
Neurons Form Networks
5
Artificial Neural Network (ANN)

• Made of artificial neurons, each of which
• Sums inputs from other neurons
• Compares sum to threshold
• Sends signal to other neurons if above threshold
• Synapses have weights
• Model relative ion collections
• Model efficacy (strength) of synapse

6
Artificial Neuron
Nonlinear firing function
.
.
.
7
Firing Functions are Sigmoidal
8
Hopfield Network
Imagine Complete Connectivity with weights wij
between ith and jth neurons
Blue 1 White 0
9
Energy
Define the energy to be
Theorem If the weights are symmetric, then the
Energy decreases each time a neuron fires.
10
Applications

• Handwriting Recognition
• http//faculty.etsu.edu/knisleyj/neural/neuralnet3
  .htm
• Universal Classifier
• http//faculty.etsu.edu/knisleyj/neural/neuralnet4
  .htm
• Expert Systems
• Rule-based rather than sequential programs
• Air traffic control, Industrial Controls
• Robotics
• Usually using a 3-layer network

11
3 Layer Neural Network
The output layer may consist of a single neuron
Output
Input
Hidden (is usually much larger)
12
Neural Nets can Think

• A Neural Network can think for itself
• Can be trained (programmed) to make decisions
• Can be trained to classify information

This tiny 3-Dimensional Artificial Neural
Network, modeled after neural networks in the
human brain, is helping machines better
visualize their surroundings.
http//www.nasa.gov/vision/universe/roboticexplore
rs/robots_like_people.html
13
The Mars Rovers

• Must choose where to explore
• Programmed to avoid rough terrain
• Programmed to choose smooth terrain
• ANN decides between rough and smooth
• rough and smooth are ambiguous
• Programmingis by means of many
  examples(lessons)

14
Illustration Colors Terrains

• As a robot moves, it defines 8 squares of size A
  that define directions it can move in
• It should avoid red (rough) terrain
• It should prefer green (smooth) terrain
• It should be indifferent to blue (normal) terrain
• It is impossible to program every possible shade
  and variation
• Instead, a neural network is constructed
• Terrain Block Color class input/output patterns
  are used to train the network

15
Train ANN to Classify Colors
Training Set Input OutputTerrain
ltR,G,Bgt
Red
lt1,0,0gt
Green
lt0,1,0gt
Blue
ColorClass
lt0,0,1gt
TerrainExamples
lt1,0,0gt
Hidden
lt0,0,0gt
16
ANNs Can Also Think for Us

• Mars Rovers do what Humans can do better
• They do not learn on their own
• They are taught by Humans who could make the same
  or better decisions in their place
• They can use their learning independently
• What about problems Humans cant solve at all
• Neural Networks can be used as savants dedicated
  to a single problem too complex for humans to
  decipher
• Examples
• Agent-Based Modeling
• RNA, Protein Structures, DNA analysis
• Data Mining

17
Division of Labor in Wasps Nests

• A Wasp can alternate between laborer, water
  forager, and pulp forager
• Hypothesis Individual wasps choose roles so that
  total water in the nest reaches equilibrium
• Hypothesis Pulp production is maximized when
  total water in the nest is stable
• Limited confirmation of Hypotheses
• (Karsai and Wenzel, 2002) System of 3 odes
• (Karsai, Phillips, and Knisley, 2005) Computer
  Simulation

18
ANN Wasp Societies

• Problems with current models
• Role assigned randomly w.r.t. a Weibull
  distribution
• Parameters selected so model works
• ANN Role is a decision of each wasp
• A wasp is a special type of Artificial Neuron
• Wasp decides its own role
• Wasp learns to make good choices
• System is deterministic yet unpredictable

19
(No Transcript)
20
(No Transcript)
21
Division of Labor by choice?

• Numbers of foragers goes up and down
• But water level in nest becomes stable

22
Data Mining

• We often consider very large data sets
• Microarrays contain about 20,000 data points
• Typical studies use 70 100 microarrays
• Most of the data is not relevant
• ANNs can be trained to find hidden patterns
• Input layer Genes
• Network is trained repeatedly with microarrays
  collected in various physiological states
• ANNs predict which genes are responsible for a
  given state

23
ANNs in Data Mining

• Each neuron acts as a linear classifier
• Competition among neurons via nonlinear firing
  function local linear classifying
• Method for Genes
• Train Network until it can classify between
  control and experimental groups
• Eliminating weights sufficiently close to 0 does
  not change local classification scheme
• First results obtained with a Perceptron ANN

24
Simple Perceptron Model
x1 Gene 1
w1
x2 Gene 2
w2
wn
xn Gene n
The output is the physiological state due to
the relative gene expression levels used as
inputs.
25
Simple Perceptron Model

• Features
• The wi measure gene significance
• Detects genes across n samples references
• Ref Artificial Neural Networks for Reducing the
  Dimensionality of Gene Expression Data, A.
  Narayanan, et al. 2004.
• Drawbacks
• The Perceptron is a globaly linear classifier
  (i.e., only classifies linearly separable data)
• We are now using a more sophisticated model

26
Linearly Separable Data
Separation using Hyperplanes
27
Data that Cannot be separated Linearly
28
How do we select ws

• Define an energy function
• x1,,xn are inputs, y output
• t1,,tm are the outputs associated with the
  patterns to be learned
• y s( S wixi - q) for a perceptron
• Key Neural networks minimize energy

29
Back Propagation

• Minimize the Energy Function
• Choose wi so that
• In practice, this is hard
• Back Propagation s? s(1-s)
• For each pattern ti
• Feed Forward and Calculate E
• Increment weights using a delta rule
• Repeat until E is sufficiently close to 0

30
Perceptron for Microarray DataMining

• Remove of genes with synaptic weights that are
  close to 0
• Create ANN classifier on reduced arrays
• Repeat 1 and 2 until only the genes that most
  influence the classifer problem remain
• Remaining genes are most important in
  classifying experimentals versus controls

31
Functional Viewpoint

• ANN is a mapping f Rn ? R
• Can we train perceptron so that f(x1,,xn) 1 if
  x vector is from a control and f(x1,,xn) 0 if
  x is from an experimental?
• Answer Yes if data can be linearly separated,
  but no otherwise unless we use better ANNs!
• General ANNs also have problems
• Spurious states (sometimes ANNs get the wrong
  answer)
• Hard Margins Training set must be perfect

32
Multilayer Network
.
.
.
.
.
.
33
How do we select ws

• Define an energy function
• t vectors are the information to be learned
• Neural networks minimize energy
• The information in the network is equivalent to
  the minima of the total squared energy function

34
Back Propagation

• Minimize the Energy Function
• Choose wj and aj so that
• In practice, this is hard
• Back Propagation with cont. sigmoidal
• Feed Forward and Calculate E
• Modify weights using a d rule
• Repeat until E is sufficiently close to 0

35
ANN as Classifer

• (Cybenko) For any egt0, the function f(x1,,xn) 1
  if x vector is from a control and f(x1,,xn) 0
  if x is from an experimental can be approximated
  to within e by a multilayer neural network.
• The weights no longer have the one-to-one
  correspondence to genes, so we test significance
  using Monte Carlo techniques.

36
ANN and Monte Carlo Methods

• Monte Carlo methods have been a big success story
  with ANNs
• Error estimates with network predictions
• ANNs are very fast in the forward direction
• Example ANNMC implement and outperform Kalman
  Filters (recursive linear filters used in
  Navigation and elsewhere) (De Freitas J. F. G.,
  et. al., 2000)

37
Recall Multilayer Network
.
.
.
.
.
.
aj correspond to genes, but do not directly
depend on a single gene.
N Genes
N node Hidden Layer
38
Naïve Monte Carlo ANN Method

• Randomly choose subset S of genes
• Train using Back Propagation
• Prune based on values of wj (or aj , or both)
• Repeat 2-3 until a small subset of S remains
• Increase count of genes in small subset
• Repeat 1-5 until each gene has 95 probability of
  appearing at least some minimum number of times
  in a subset
• Most frequent genes are the predicted

39
Additional Considerations

• If a gene is up-regulated or down-regulated for a
  certain condition, then put it into a subset in
  step 1 with probability 1.
• This is a simple-minded Bayesian method.
  Bayesian analysis can make it much better.
• Algorithm distributes naturally across a
  multi-processor cluster or machine
• Choose the subsets first
• Distribute subsets to different machines
• Tabulate the results from all the machines

40
Summary

• ANNs are designed to make decisions in similar
  fashion to how we make decisions
• In this way, they can think for themselves
• Can be considered supplements to existing
  hardware and software tools
• The ability of ANNs to make decisions allows
  them to think for us as well!
• They can find patterns in large data sets that we
  humans would likely never uncover
• They can think for days/months on end

41
Any Questions?
42
References

• Cybenko, G. Approximation by Superpositions of a
  sigmoidal function, Mathematics of Control,
  Signals, and Systems, 2(4),1989, p. 303-314.
• De Freitas J. F. G., et. al. Sequential Monte
  Carlo Methods To Train Neural Network Models.
  Neural Computation, Volume 12, Number 4, 1 April
  2000, pp. 955-993(39)
• L. Glenn and J. Knisley, Solutions for Transients
  in Arbitrarily Branching and Tapering Cables,
  Modeling in the Neurosciences From Biological
  Systems to Neuromimetic Robotics, ed. Lindsay,
  R., R. Poznanski, G.N.Reeke, J.R. Rosenberg, and
  O.Sporns, CRC Press, London, 2004.
• A. Narayan, et. al Artificial Neural Networks
  for Reducing the Dimensionality of Gene
  Expression Data. Neurocomputing, 2004.