Extensions of Non-negative Matrix Factorization to Higher Order data

1
Extensions of Non-negative Matrix Factorization
to Higher Order data
Morten Mørup Informatics and Mathematical
Modeling Intelligent Signal Processing Technical
University of Denmark
2
Parts of the work done in collaboration with
Sæby, May 22-2006
Mikkel N. Schmidt, Stud. PhDDepartment of Signal
Processing Informatics and Mathematical
Modeling, Technical University of Denmark
Sidse M. Arnfred, Dr. Med. PhD Cognitive Research
Unit Hvidovre Hospital University Hospital of
Copenhagen
Lars Kai Hansen, Professor Department of Signal
Processing Informatics and Mathematical
Modeling, Technical University of Denmark
3
Outline

• Non-negativity Matrix Factorization (NMF)
• Sparse coding (SNMF)
• Convolutive PARAFAC models (cPARAFAC)
• Higher Order Non-negative Matrix
  Factorization(an extension of NMF to the Tucker
  model)

4
NMF is based on Gradient Descent

• NMF V?WH s.t. Wi,d,Hd,j?0
• Let C be a given cost function, then update the
  parameters according to

5
The idea behind multiplicative updates
Positive term
Negative term
6
Non-negative matrix factorization (NMF)
(Lee Seung - 2001)
NMF gives Part based representation
(Lee Seung Nature 1999)
7
The NMF decomposition is not unique
Simplical Cone
NMF only unique when data adequately spans the
positive orthant (Donoho Stodden - 2004)
8
Sparse Coding NMF (SNMF)
(Eggert Körner, 2004)
9
Illustration (the swimmer problem)
Swimmer Articulations
NMF Expressions
SNMF Expressions
True Expressions
10
Why sparseness?

• Ensures uniqueness
• Eases interpretability (sparse representation ?
  factor effects pertain to fewer dimensions)
• Can work as model selection(Sparseness can turn
  off excess factors by letting them become zero)
• Resolves over complete representations (when
  model has many more free variables than data
  points)

11
PART I Convolutive PARAFAC (cPARAFAC)
12
By cPARAFAC means PARAFAC convolutive in at least
one modality
Convolution The process of generating Xby
convolving (sending) the sources S through the
filter A Deconvolution The process of estimating
the filter A from X and S
Convolution can be in any combination of
modalities -Single convolutive, double
convolutive etc.
13
Relation to other models

• PARAFAC2 (Harshman, Kiers, Bro)
• Shifted PARAFAC (Hong and Harshman, 2003)

3
3
cPARAFAC can account for echo effects
cPARAFAC becomes shifted PARAFAC when convolutive
filter is sparse
14
Application example of cPARAFAC

• Transcription and separation of music

15
The ideal Log-frequency Magnitude Spectrogram
of an instrument
Tchaikovsky Violin Concert in D Major

• Different notes played by aninstrument
  corresponds on a logarithmic frequency scale to
  a translation of the same harmonicstructure of
  a fixed temporal pattern

Mozart Sonate no,. 16 in C Major
16
NMF 2D deconvolution (NMF2D1) The Basic Idea

• Model a log-spectrogram of polyphonic music by an
  extended type of non-negative matrix
  factorization
• The frequency signature of a specific note played
  by an instrument has a fixed temporal pattern
  (echo)? model convolutive in time
• Different notes of same instrument has same
  time-log-frequency signature but varying in
  fundamental frequency (shift) ? model
  convolutive in the log-frequency axis.

(1Mørup Scmidt, 2006)
17
NMF2D Model

• NMF2D Model extension of NMFD1

18
Understanding the NMF2D Model
19
The NMF2D has inherent ambiguity between the
structure in W and H
To resolve this ambiguity sparsity is imposed on
H to force ambiguous structure onto W
20
Real music example of how imposing sparseness
resolves the ambiguity between W and H
NMF2D
SNMF2D
21
Extension to multi channel analysis by the
PARAFAC model
Not unique
Unique!!
PARAFAC(Harshman Carrol and Chang 1970)
Factor analysis(Charles Spearman 1900)
22
cPARAFAC Sparse Non-negative Tensor Factor 2D
deconvolution (SNTF2D)
(Extension of Fitzgerald et al. 2005, 2006 to
form a sparse double deconvolution)
23
SNTF2D algorithms
24
Tchaikovsky Violin Concert in D Major
Mozart Sonate no. 16 in C Major
25
Stereo recording of Fog is Lifting by Carl
Nielsen
26
Applications

• Applications
• Source separation.
• Music information retrieval.
• Automatic music transcription (MIDI compression).
• Source localization (beam forming)

27
PART II Higher Order NMF (HONMF)
28
Higher Order Non-negative Matrix Factorization
(HONMF)
Motivation Many of the data sets previously
explored by the Tucker model are non-negative and
could with good reason be decomposed under
constraints of non-negativity on all modalities
including the core.

• Spectroscopy data (Smilde et al. 1999,2004,
  Andersson Bro 1998, Nørgard Ridder 1994)
• Web mining (Sun et al., 2004)
• Image Analysis(Vasilescu and Terzopoulos, 2002,
  Wang and Ahuja, 2003, Jian and Gong, 2005)
• Semantic Differential Data(Murakami and
  Kroonenberg, 2003)
• And many more

29
However, non-negative Tucker decompositions are
not in general unique!
But - Imposing sparseness overcomes this problem!
30
The Tucker Model
31
Algorithms for HONMF
32
Results
HONMF with sparseness, above imposed on the core
canbe used for model selection -here indicating
the PARAFAC model is the appropriate model to the
data. Furthermore, the HONMF gives a more part
based hence easy interpretable solution than the
HOSVD.
33
Evaluation of uniqueness
34
Data of a Flow Injection Analysis (Nørrgaard,
1994)
HONMF with sparse core and mixing captures
unsupervised the true mixing and model order!
35
Conclusion

• HONMF not in general unique, however when
  imposing sparseness uniqueness can be achieved.
• Algorithms devised for LS and KL able to impose
  sparseness on any combination of modalities
• The HONMF decompositions more part based hence
  easier to interpret than other Tucker
  decompositions such as the HOSVD.
• Imposing sparseness can work as model selection
  turning of excess components

36
Coming soon in a MATLAB implementation near You
37
References
Carroll, J. D. and Chang, J. J. Analysis of
individual differences in multidimensional
scaling via an N-way generalization of
"Eckart-Young" decomposition, Psychometrika 35
1970 283--319 Eggert, J. and Korner, E. Sparse
coding and NMF. In Neural Networks volume 4,
pages 2529-2533, 2004 Eggert, J et al
Transformation-invariant representation and nmf.
In Neural Networks, volume 4 , pages 535-2539,
2004 Fiitzgerald, D. et al. Non-negative tensor
factorization for sound source separation. In
proceedings of Irish Signals and Systems
Conference, 2005 FitzGerald, D. and Coyle, E. C
Sound source separation using shifted
non.-negative tensor factorization. In
ICASSP2006, 2006 Fitzgerald, D et al. Shifted
non-negative matrix factorization for sound
source separation. In Proceedings of the IEEE
conference on Statistics in Signal Processing.
2005 Harshman, R. A. Foundations of the PARAFAC
procedure Models and conditions for an
"explanatory" multi-modal factor analysis,UCLA
Working Papers in Phonetics 16 1970
184 Harshman, Richard A.Harshman and Hong,
Sungjin Lundy, Margaret E. Shifted factor
analysisPart I Models and properties J.
Chemometrics (17) pages 379388, 2003 Kiers,
Henk A. L. and Berge, Jos M. F. ten and Bro,
Rasmus PARAFAC2 - Part I. A direct fitting
algorithm for the PARAFAC2 model, Journal of
Chemometrics (13) nr.3-4 pages 275-294,
1999 Lathauwer, Lieven De and Moor, Bart De and
Vandewalle, Joos MULTILINEAR SINGULAR VALUE
DECOMPOSITION.SIAM J. MATRIX ANAL. APPL.2000
(21)12531278 Lee, D.D. and Seung, H.S.
Algorithms for non-negative matrix factorization.
In NIPS, pages 556-462, 2000 Lee, D.D and Seung,
H.S. Learning the parts of objects by
non-negative matrix factorization, NATURE
1999 Murakami, Takashi and Kroonenberg, Pieter M.
Three-Mode Models and Individual Differences
in Semantic Differential Data, Multivariate
Behavioral Research(38) no. 2 pages 247-283, 2003
Mørup, M. and Hansen, L.K.and Arnfred,
S.M.Decomposing the time-frequency representation
of EEG using nonnegative matrix and multi-way
factorization Technical report, Institute for
Mathematical Modeling, Technical University of
Denmark, 2006a Mørup, M. and Schmidt, M.N. Sparse
non-negative matrix factor 2-D deconvolution.
Technical report, Institute for Mathematical
Modeling, Tehcnical University of Denmark,
2006b Mørup, M and Schmidt, M.N. Non-negative
Tensor Factor 2D Deconvolution for multi-channel
time-frequency analysis. Technical report,
Institute for Mathematical Modeling, Technical
University of Denmark, 2006c Schmidt, M.N. and
Mørup, M. Non-negative matrix factor 2D
deconvolution for blind single channel source
separation. In ICA2006, pages 700-707,
2006d Mørup, M. and Hansen, L.K.and Arnfred, S.M.
Algorithms for Sparse Higher Order Non-negative
Matrix Factorization (HONMF), Technical report,
Institute for Mathematical Modeling, Technical
University of Denmark, 2006e Nørgaard, L and
Ridder, C.Rank annihilation factor analysis
applied to flow injection analysis with
photodiode-array detection Chemometrics and
Intelligent Laboratory Systems 1994 (23)
107-114 Schmidt, M.N. and Mørup, M. Sparse
Non-negative Matrix Factor 2-D Deconvolution for
Automatic Transcription of Polyphonic Music,
Technical report, Institute for Mathematical
Modelling, Tehcnical University of Denmark,
2005 Smaragdis, P. Non-negative Matrix Factor
deconvolution Extraction of multiple sound
sources from monophonic inputs. International
Symposium on independent Component Analysis and
Blind Source Separation (ICA)W Smilde, Age K.
Smilde and Tauller, Roma and Saurina, Javier and
Bro, Rasmus, Calibration methods for complex
second-order data Analytica Chimica Acta 1999
237-251 Sun, Jian-Tao and Zeng, Hua-Jun and Liu,
Huanand Lu Yuchang and Chen Zheng CubeSVD a
novel approach to personalized Web search WWW
'05 Proceedings of the 14th international
conference on World Wide Web pages 382390,
2005 Tamara G. Kolda Multilinear operators for
higher-order decompositions technical report
Sandia national laboratory 2006
SAND2006-2081. Tucker, L. R. Some mathematical
notes on three-mode factor analysis Psychometrika
31 1966 279311 Welling, M. and Weber, M.
Positive tensor factorization. Pattern Recogn.
Lett. 2001 Vasilescu , M. A. O. and Terzopoulos ,
Demetri Multilinear Analysis of Image Ensembles
TensorFaces, ECCV '02 Proceedings of the 7th
European Conference on Computer Vision-Part I,
2002