Subspace Clustering

1
Subspace Clustering

• Ali Sekmen and Ghan S. Bhatt
• Computer Science and Mathematical Sciences
• College of Engineering
• Tennessee State University

1st Annual Workshop on Data Sciences
2
Part I

• Some Linear Algebra
• Spectral Analysis
• Singular Value Decomposition
• Presenter
• Dr. Ghan S. Bhatt

3
Definitions
4
Range and Null Spaces
5
Range and Null Spaces
6
Definitions
7
Eigenvalues - Eigenvectors
8
Eigenvalues - Eigenvectors
9
Eigenvalues - Eigenvectors
10
Eigenvalues - Eigenvectors
11
Symmetric Matrices
12
Symmetric Matrices
13
Projection on a Vector
14
Projection on a Subspace
15
Singular Value Decomposition
16
Singular Value Decomposition
17
Singular Value Decomposition
18
Singular Value Decomposition
19
Important Lemma
20
Recall Linear Mapping
21
Recall Linear Mapping
22
General Matrix Norms
23
An Intuitive Matrix Norm
This satisfies the general matrix norm properties
Although it is useful, it is not suitable for
large set of problems and we need another
definition of matrix norms
24
Induced Matrix Norms
25
Matrix p-Norm
26
More on Matrix Norms
27
Part II

• Subspace Segmentation Problem
• Motion Segmentation
• Principal Component Analysis
• Dimensionality Reduction
• Spectral Clustering
• Presenter
• Dr. Ali Sekmen

28
Subspace Segmentation

• In many engineering and mathematics applications,
  data lives in a union of low dimensional
  subspaces
• Motion segmentation
• Facial images of a person with the same
  expression under different illumination
  approximately lie on the same subspace

29
Face Recognition
30
Problem Statement
31
Problem Statement
32
Problem Statement
33
What are we trying to solve?
34
Example Motion Segmentation
35
Motion Segmentation
Motion segmentation problem can simply be defined
as identifying independently moving rigid objects
in a video.
36
Motion Segmentation
37
Motion Segmentation
Z
Y
X
38
Motion Segmentation
39
Motion Segmentation
40
Motion Segmentation
41
Motion Segmentation
Y
X
42
Motion Segmentation
Motion Segmentation
43
Motion Segmentation
44
Motion Segmentation
45
Principal Component Analysis

• The goal is to reduce dimension of dataset with
  minimal loss of information
• We project a feature space onto a smaller
  subspace that represent data well
• Search for a subspace which maximizes the
  variance of projected points
• This is equivalent to linear least square fitting
• Minimize the sum of squared distances between
  points and subspace
• We find directions (components) that maximizes
  variance in dataset
• PCA can be done by
• Eigenvalue decomposition of a data covariance
  matrix
• Or SVD of a data matrix

46
Least Square Approximation
47
Principal Component Analysis
48
Principal Component Analysis
49
PCA with SVD
Coordinates w.r.t. new basis
50
Principal Component Analysis
inch
cm
51
Principal Component Analysis
inch cm
10 28
12 19
15 40
20 47
23 56
26 69
52
Solution with SVD
53
PCA Pre-Processing
inch cm
10 28
12 19
15 40
20 47
23 56
26 69
54
PCA Optimization
55
PCA Reduce Dimensionality
56
PCA Reduce Dimensionality
57
General PCA
58
Spectral Clustering

• A very powerful clustering algorithm
• Easy to implement
• Outperforms traditional clustering algorithms
• Example k-means
• It is not easy to understand why it works
• Given a set of data points and some similarity
  measure between all pairs of data points, we
  divide data into groups
• Points in the same group are similar
• Points in different groups are dissimilar

59
Spectral Clustering

• Most of subspace clustering algorithms employ
  spectral clustering as the last step

60
Similarity
61
Spectral Clustering
62
Spectral Clustering
63
Spectral Clustering
64
Spectral Clustering Example
From Lecture Notes of Ulrike von Luxburg
65
Spectral Clustering Example
From Lecture Notes of Ulrike von Luxburg
66
Spectral Clustering Example
From Lecture Notes of Ulrike von Luxburg
67
Spectral Clustering Example
From Lecture Notes of Ulrike von Luxburg
68
Spectral Clustering Example
From Lecture Notes of Ulrike von Luxburg
69
Spectral Clustering Example
70
Spectral Clustering Example
71
Spectral Clustering Example
72
(No Transcript)