Introduction to Independent Component Analysis

1
Introduction toIndependent Component Analysis

• Bart Cramer
• School of Behavioral and Cognitive Neurosciences
• Rijksuniversiteit Groningen

2
Contents

• Introduction in ICA
• Case study ICA in the primary visual cortex
• Hypothesis
• Objections

3
Source separation
4
Source separation

• S Original signalsX Perceived signalsY
  Supposedly original signals
• X ASY WX
• Error function E f(S, Y)
• Try to find a W that minimizes E

5
Which order?

• Second order
• E(S, Y) S (yi - xi)2
• Tries to find a faithful representation
• E.g. Principal Component Analysis (Jolliffe,
  1986)
• Higher order
• E.g Projection pursuit (Friedman, 1987)

6
Second vs higher order
7
Error functions in ICA

• Also called contrast functions, and try to
  capture statistical independence.
• Possibly based on measures of
• non-gaussianity (kurtosis) (Jones Sibson, 1987)
• mutual information (Comon, 1994)
• These are hard to calculate, so approximations
  are needed.

8
Algorithm gradient descent(Jutten Hérault,
1991)

• Start with initial W
• Do small alterations to W so that the error
  function E(S, Y) minimizes
• Repeat this until W converges to a stable solution

9
Visual pathway

• Light reaches us in the photoreceptors.
• The intensity is measured, and sent to the
  primary visual cortex (V1), where basic features
  are extracted (lines, edges).
• These features are localized, oriented and
  bandpassed. (Hubel and Wiesel, 1968)

10
Receptive fields
11
Model of sight

• Assume that these photoreceptors are in a line,
  forming a vector having a length of approximately
  260.000.000.
• So, at each moment in time, all these receptors
  are having a certain activation, according to the
  intensity of the light.

12
Sparse coding

• Sparse coding is a group of methods that tries to
  reduce a big amount of redundant data to a
  compact, meaningful representation.
• Needed for (Olshausen and Field, 2004)
• Increased storage capacity
• Easier understanding
• Energy saving

13
ICA on natural scenes(Olshausen Field, 1996)

• Remember
• X AS
• Y WX
• X is the line of photoreceptors (now 512 x 512
  260.000 pixels)
• But what does S represent?

14
Basis functions

• A basis function is a part of an image in such a
  way that
• Every image can be represented by a linear
  combination of basis functions.

15
Basis functions
16
(No Transcript)
17
Overcompleteness

• Suppose we would find that we need 100
  independent components.
• This number is critical, but we could use just as
  well 200 or 300.
• Having an overcomplete set of basis function
  increases robustness.

18
Overcompleteness

• Bell and Sejnowski (1997) more or less used the
  same method, and found qualitatively the same
  result.
• However Olshausen and Fields method allows for
  overcompleteness, but this is at the cost of loss
  of orthogonality.

19
Hypotheses

• The basis functions that emerge from the
  algorithm resemble strongly the feature detectors
  in the primary visual cortex.
• This means that our brain organizes itself using
  higher-order statistics to find the independent
  component in natural scenes.

20
Objections

• The learning rules are not local.
• The form of the model (XAS) does not allow for
  non-linearities in the real world, as occlusion.
• More generally, the process from retina to V1 is
  more complex than matrix multiplication.