Matrix learning for topographic neural maps

1
Matrix learning for topographic neural maps
Banchar Arnonkijpanich, Khon Kaen University,
Thailand Barbara Hammer, TU Clausthal,
Germany Alexander Hasenfuss, TU Clausthal,
Germany Chidchanonk Lursinsap, Chulalongkorn
University, Thailand
2

• Neural maps
• k-means
• SOM
• NG
• Matrix learning
• Previous work
• Batch matrix learning
• Convergence
• Experiments
• Proof of concept
• UCI
• Image compression

3
Neural maps
4
Neural maps

• Clustering
• too many data
• reduce size, improve accessability

5
Neural maps

• Topographic mapping
• still too many data
• mutual connection
• visualization

6
Neural maps

• Prototype based clustering
• prototypes element of data space
• clustering by means of receptive fields
• Pros
• robust
• interpretable
• incremental
• efficient
• excellent generalization

7
Neural maps

• Prototype based clustering
• quantization error EVQ ?ij?(xj,wi) (xj-wi)2
  where ?(xj,wi) indicates the receptive field
• (xj-wi)2 is the euclidean distance

8
k-means
optimize EVQ ?ij?(xj,wi) (xj-wi)2
kij
k-means repeat optimize
kij given fixed w optimize
w given fixed kij i.e. repeat
kij 1 iff wi winner for xj
wi ?j kij xj / ?j
kij converges, Newton scheme, initialization
sensitive
9
SOM

• Self-organizing map Kohonen
• prototypes equipped with a prior lattice
  structure
• direct visualization in euclidean or hyperbolic
  space

10
SOM
optimize ESOM ?ij?j(i)?kexp(-nd(i,k )/s)
(xj-wk)2
kij
Batch-SOM repeat
optimize kij given fixed w
optimize w given fixed kij i.e.
repeat kij 1 iff wi
winner for xj , I(xi) winner
wi ?j exp(-nd(I(xj),i)/s) xj / ?j
exp(-nd(I(xj),i)/s) converges, Newton scheme,
topological mismatches possible
11
Neural Gas

• Neural Gas Martinetz
• prototypes adapted according to data topology
• optimize ENG ?ijexp(-rk(xj,wi)/s) (xj-wi)2

neighborhood graph induced by Delaunay
triangulation visualization by means of MDS or
similar
3rd
1st
2nd
12
Neural Gas
optimize ENG ?ijexp(-rk(xj,wi)/s) (xj-wi)2
kij
Batch-NG Cottrell, Hammer, Hasenfuss,
Villmann repeat
optimize kij given fixed w
optimize w given fixed kij i.e.
repeat kij rank of
prototype wi given xj wi
?j exp(-kij/s) xj / ?j exp(-kij/s) converges,
Newton scheme
13
Matrix learning
14
Matrix learning

• Matrix learning
• data not properly scaled
• data correlations

15
Matrix learning
16
Previous work

• metric / matrix learning for supervised
  prototype-based schemes
• metric learning for semi-supervised settings
• metric / matrix learning for fuzzy clustering
• metric learning for k-means, SOM, NG by a
  combination of PCA and clustering
• no matrix learning scheme derived from a unique
  cost function for topographic neural maps
• no convergence proofs for these settings

17
Batch matrix learning

• objective (exemplarily for NG)
• optimize ENG (w, ?) ?ijexp(-rk(xj,wi)/s)
  (xj-wi)t ?i (xj-wi)
• where
• det ?i 1
• ?i symmetric and positive semidefinite
• Batch scheme
• optimize ENG (w, ?, k) ?ijexp(-kij)/s) (xj-wi)t
  ?i (xj-wi)
• where
• det ?i 1
• ?i symmetric and positive semidefinite
• kij is permutation of 0 .. k.1 for every j

18
Batch matrix learning

• repeat
• kij rank of prototype wi given xj measured
• according to xi-wj?i
• wi ?j exp(-kij/s) xj / ?j exp(-kij/s)
• ?i Si-1 (det Si)1/n where
• Si ?j exp(-kij/s) (xj-wi)(xj-wi)t

symmetric positive semidefinite
19
Batch matrix learning

• Si corresponds to Mahalanobis distance
• Si aligns with the principal directions of the
  extended receptive field
• scaling is according to inverse eigenvalues

20
Convergence

• batch matrix learning can be seen as interleaved
  local PCA and NG
• derived from a unique cost function
  ENG (w, ?)
  ?ijexp(-rk(xj,wi)/s) (xj-wi)t ?i (xj-wi) resp.
  ENG (w, ?, k)
  ?ijexp(-kij)/s) (xj-wi)t ?i (xj-wi)
• batch optimization finds global optima of
• k given fixed w, ?
• w given fixed k, ?
• ? given fixed w, k
• therefore, ENG (w, ?, k) decreases in every
  step and convergence is guaranteed

21
Experiments
22
Proof of concept
23
Proof of concept
24
UCI
25
Image compression

• local window size 8x8
• matrix NG with 16 clusters
• store only 32 main principle components and
  corresponding directions

26
Image compression

• original
  compressed

27
Thank you for your attention!