1
A fast method for Underdetermined Sparse
Component Analysis (SCA) based on Iterative
Detection-Estimation (IDE)

• Arash Ali-Amini1
• Massoud BABAIE-ZADEH1,2
• Christian Jutten2
• 1- Sharif University of Technology, Tehran, IRAN
• 2- Laboratory of Images and Signals (LIS), CNRS,
  INPG, UJF, Grenoble, FRANCE

2
Outline

• Introduction to Blind Source Separation
• Geometrical Interpretation
• Sparse Component Analysis (SCA), underdetermined
  case
• Identifying mixing matrix
• Source restoration
• Finding sparse solutions of an Underdetermined
  System of Linear Equations (USLE)
• Minimum L0 norm
• Matching Pursuit
• Minimum L1 norm or Basis Pursuit (?Linear
  Programming)
• Iterative Detection-Estimation (IDE) - our method
  
• Simulation results
• Conclusions and Perspectives

3
Blind Source Separation (BSS)

• Source signals s1, s2, , sM
• Source vector s(s1, s2, , sM)T
• Observation vector x(x1, x2, , xN)T
• Mixing system ? x As
• Goal ? Finding a separating matrix y Bx

4
Blind Source Separation (cont.)

• Assumption
• N gtM (sensors gt sources)
• A is full-rank (invertible)
• prior information Statistical Independence of
  sources
• Main idea Find B to obtain independent
  outputs (? Independent Component AnalysisICA)

5
Blind Source Separation (cont.)

• Separability Theorem Comon 1994,Darmois 1953
  If at most 1 source is Gaussian statistical
  independence of outputs ? source separation (?
  ICA a method for BSS)
• Indeterminacies permutation, scale
• A a1, a2, , aM ,
  xAs ?
• x s1 a1 s2 a2
  sM aM

6
Geometrical Interpretation
xAs
Statistical Independence of s1 and s2, bounded
PDF ? rectangular support for joint PDF of
(s1,s2) ? rectangular scatter plot
for (s1,s2)
7
Sparse Sources
2 sources, 2 sensors
s-plane
x-plane
8
Importance of Sparse Source Separation

• The sources may be not sparse in time, but sparse
  in another domain (frequency, time-frequency,
  time-scale, etc.)
• xAs ? TxA Ts ? T a sparsifying
  transform
• Example. Speech signals are usually sparse in
  time-frequency domain.

9
Sparse sources (cont.)

• 3 sparse sources, 2 sensors

Sparsity ? Source Separation, with more sensors
than sources?
10
Estimating the mixing matrix

• A a1, a2, a3 ?
• x s1 a1 s2 a2 s3 a3
• ? Mixing matrix is easily identified for sparse
  sources
• Scale Permutation indeterminacy
• ai1

11
Restoration of the sources

• How to find the sources, after having found the
  mixing matrix (A)?

2 equations, 3 unknowns ? infinitely many
solutions!
Underdertermined SCA, underdetermined system of
equations
12
Identification vs Separation

• Sources lt Sensors
• Identifying A ? Source Separation
• Sources gt Sensors (underdetermined)
• Identifying A ? Source Separation
• ? Two different problems
• Identifying the mixing matrix (relatively easy)
• Restoring the sources (difficult)

13
Is it possible, at all?

• A is known, at eash instant (n0), we should solve
  un underdetermined linear system of equations
• Infinite number of solutions s(n0) ? Is it
  possible to recover the sources?

14
Sparse solution

• si(n) sparse in time ? The vector s(n0) is most
  likely a sparse vector
• A.s(n0) x(n0) has infinitely many solutions,
  but not all of them are sparse!
• Idea For restoring the sources, take the
  sparsest solution (most likely solution)

15
Example (2 equations, 4 unknowns)

• Some of solutions

Sparsest
16
The idea of solving underdetermined SCA

• A s(n) x(n) , n0,1,,T
• Step 1 (identification) Estimate A (relatively
  easy)
• Step 2 (source restoration) At each instant n0,
  find the sparsest solution of
• A s(n0) x(n0), n00,,T
• Main question HOW to find the sparsest solution
  of an Underdetermined System of Linear Equations
  (USLE)?

17
Another application of USLE Atomic decomposition
over an overcompelete dictionary

• Decomposing a signal x, as a linear combination
  of a set of fixed signals (atoms)
• Terminology
• Atoms ? i , i1,,M
• Dictionary ? 1 , ? 2 ,, ? M

18
Atomic decomposition (cont.)

• MN ? Complete dictionary ? Unique set of
  coefficients
• Examples Dirac dictionary, Fourier Dictionary

Dirac Dictionary
19
Atomic decomposition (cont.)

• MN ? Complete dictionary ? Unique set of
  coefficients
• Examples Dirac dictionary, Fourier Dictionary

Fourier Dictionary
20
Atomic decomposition (cont.)

• Matrix Form
• If just a few number of coefficient are non-zero
  ? The underlying structure is very well revealed
• Example.
• signal has just a few non-zero samples in time ?
  its decomposition over the Dirac dictionary
  reveals it
• Signals composed of a few pure frequencies ? its
  decomposition over the Fourier dictionary reveals
  it
• How about a signals which is the sum of a pure
  frequency and a dirac?

21
Atomic decomposition (cont.)

• Solution consider a larger dictionary,
  containing both Dirac and Fourier atoms
• MgtN ? Overcomplete dictionary.
• Problem Non-uniqueness of ? (Non-unique
  representation) (? USLE)
• However we are looking for sparse solution

22
Sparse solution of USLE
Atomic Decomposition on over-complete dictionaries
Underdetermined SCA
Findind sparsest solution of USLE
23
Uniqueness of sparse solution

• xAs, n equations, m unknowns, mgtn
• Question Is the sparse solution unique?
• Theorem (Donoho 2004) if there is a solution s
  with less than n/2 non-zero components, then it
  is unique with probability 1 (that is, for almost
  all As).

24
How to find the sparsest solution

• A.s x, n equations, m unknowns, mgtn
• Goal Finding the sparsest solution
• Note at least m-n sources are zero.
• Direct method
• Set m-n (arbitrary) sources equal to zero
• Solve the remaining system of n equations and n
  unknowns
• Do above for all possible choices, and take
  sparsest answer.
• Another name Minimum L0 norm method
• L0 norm of s number of non-zero components
  ?si0

25
Example

• s1s20 ? s(0, 0, 1.5, 2.5)T ? L02
• s1s30 ? s(0, 2, 0, 0)T ? L01
• s1s40 ? s(0, 2, 0, 0)T ? L01
• s2s30 ? s(2, 0, 0, 2)T ? L02
• s2s40 ? s(10, 0, -6, 0)T ? L02
• s3s40 ? s(0, 2, 0, 0)T ? L02
• ? Minimum L0 norm solution ? s(0, 2, 0, 0)T

26
Drawbacks of minimal norm L0

• Highly (unacceptably) sensitive to noise
• Need for a combinatorial search
• Example. m50, n30,

On our computer Time for solving a 30 by 30
system of equation2x10-4s
Total time ? (5x1013)(2x10-4) ? 300 years! ?
Non-tractable
27
faster methods?

• Matching Pursuit Mallat Zhang, 1993
• Basis Pursuit (minimal L1 norm ? Linear
  Programming) Chen, Donoho, Saunders, 1995
• Our method (IDE)

28
Matching Pursuit (MP) Mallat Zhang, 1993
29
Properties of MP

• Advantage
• Very Fast
• Drawback
• A very greedy algorithm ? Mistake in a stage
  can never be corrected ? Not necessarily a sparse
  solution

xs1a1s2a2
a1
a3
a2
a4
a5
30
Minimum L1 norm or Basis Pursuit Chen, Donoho,
Saunders, 1995

• Minimum norm L1 solution
• MAP estimator under a Laplacian prior
• Recent theoretical support (Donoho, 2004)
• For most large underdetermined systems of
  linear equations, the minimal L1 norm solution is
  also the sparsest solution

31
Minimal L1 norm (cont.)

• Minimal L1 norm solution may be found by Linear
  Programming (LP)
• Fast algorithms for LP
• Simplex
• Interior Point method

32
Minimal L1 norm (cont.)

• Advantages
• Very good practical results
• Theoretical support
• Drawback
• Tractable, but still very time-consuming

33
Iterative Detection-Estmation (IDE)- Our method

• Main Idea
• Step 1 (Detection) Detect which sources are
  active, and which are non-active
• Step 2 (Estimation) Knowing active sources,
  estimate their values
• Problem Detecting the activity status of a
  source, requires the values of all other sources!
• Our proposition Iterative Detection-Estimation

34
IDE, Detection step

• Assumptions
• Mixture of Gaussian model for sparse sources
• si active ?
  siN(0,?12)
• si in-active ?
  siN(0,?02), ?1gtgt?0
• ?0 probability of
  in-activity ? 1
• Columns of A are normalized
• Hypotheses
• Proposition
• ta1Txs1?
• Activity detection test g1(s)t- ?gt?
• ? depends on other sources ? use their previous
  estimates

35
IDE, Estimation step

• Estimation equation
• Solution using Lagrange multipliers
• Closed-form solution ? See the paper

36
IDE, summary

• Detection Step (resulted from binary hypothesis
  testing, with a Mixture of Gaussian source
  model)
• Estimation Step

37
IDE, simulation results
m1024, n0.4m409
38
IDE, simulation results (cont.)

• m1024, n0.4m409
• IDE is about two order of magnitudes faster than
  LP method (with interior point optimization).

39
Speed/Complexity comparision
40
IDE, simulation results number of possible
active sources

• m500, n0.4m200, averaged on 10 simulations

41
Conclusion and Perspectives

• Two problems of Underdetermined SCA
• Identifying mixing matrix
• Restoring sources
• Two applications of finding sparse solution of
  USLEs
• Source restoration in underdetermined SCA
• Atomic Decomposition on over-complete
  dictionaries
• Methods
• Minimum L0 norm (?Combinatorial search)
• Minimum L1 norm or Basis Pursuit (?Linear
  Programming)
• Matching Pursuit
• Iterative Detection-Estimation (IDE)
• Perspectives
• Better activity detection (removing thresholds?)
• Applications in other domains

42
Thank you very much for your attention