Convex transduction with the Normalized Cut

1
Convex transduction with the Normalized Cut

• Tijl De Bie
• Nello Cristianini
• ECS, ISIS, University of Southampton

2
Motivation

• Transduction
• Learn the classes for a set of samples
• Given
• A training set of labeled samples
• A test set of all unlabeled samples
• Find
• the labels of the unlabeled samples
• No classification function is required

3
Motivation

• Image segmentation
• Separate object from background?
  segmentation/clustering of pixels
• User labels a few points ? transduction

SourceYu Shi, Segmentation given partial
grouping constraints
4
Motivation

• Bioinformatics
• E.g. categorize genes into functional classes
  based on a training set ? classification
• All genes, but only a few labels, are known ?
  transduction
• Approach here add label constraints in
  clustering algorithms
• (Note clustering with equivalence /
  inequivalence constraints is a straightforward
  extension in this approach)

5
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A spectral/SDP combined approach
• Experiments Conclusions

6
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Represent data as nodes of a weighted graph
• Weights of the graph similarity measure
• Mincut clustering find bipartitioning minimizing
  the sum of weights of cut edges
• Easy to solve, but
• unbalanced clusterings

7
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solution normalized cut

8
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solution normalized cut

9
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solution normalized cut

10
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solution normalized cut

11
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solution normalized cut

12
The Normalized Cut for Clustering
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Normalized cut optimization problem in algebraic
  form
• Combinatorial problem, very hard!(in contrast to
  mincut)

13
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A spectral/SDP combined approach
• Experiments Conclusions

14
A spectral relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Normalized cut optimization problem
• Rewrite in terms of

15
A spectral relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Then
• Observation ? relax
  combinatorial constraint to this norm constraint

16
A spectral relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Solved by eigenvalue problem
• Eigenvector with second smallest eigenvalue
  approximation for unrelaxed
• (first eigenvector is with eigenvalue 0)
• This result has first been derived, in a
  different way, in Shi Malik

17
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A spectral/SDP combined approach
• Experiments Conclusions

18
An SDP relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Normalized cut optimization problem
• With
  this means

19
An SDP relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Then we get
• Rewrite in terms of andrelax
  to

20
An SDP relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Then
• Extraction of label vector from dominant
  eigenvector, any of its columns, randomized,

21
An SDP relaxation
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Then
• Extraction of label vector from dominant
  eigenvector, any of its columns, randomized,
• Number of dual variables O(n)

22
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A spectral/SDP combined approach
• Experiments Conclusions

23
Transduction based on the Normalized Cut
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Spectral transduction
• Recall, Normalized Cut (before relaxation)
• How to fix training labels efficiently?

24
Transduction based on the Normalized Cut
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Parameterize
• where

Training set
25
Transduction based on the Normalized Cut
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Recall
• Now, the constrained spectral relaxation is
• Again an eigenvalue problem
• Even of a smaller size ? very efficient
  (especially for sparse similarity measures)

26
Transduction based on the Normalized Cut
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• SDP relaxation and transduction
• Now, parameterize the label matrix as
• where
• Then indeed,

27
Transduction based on the Normalized Cut
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• The resulting SDP transduction problem is
• Even easier than the unconstrained problem

28
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A spectral/SDP combined approach
• Experiments Conclusions

29
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Lets go back to clustering for a while
• One can show the spectral relaxation is a
  relaxation of the SDP relaxation
• Can we combine both approaches for speed-up (of
  SDP) / increase in quality (of spectral)?

30
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Similar trick as for transduction
• Assume we know a subspace to which the
  label vector belongs
• Then, relax the label matrix as
• SDP constraint (ideally has
  rank 1)
• much smaller than

31
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Result
• Which to use?

32
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Use spectral relaxation to estimate a good
• Note this subspace will not be perfect
• ?
  is an infeasible constraint
• ? relax to

33
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• So, finally
• Where is obtained using the spectral
  relaxation of the normalized cut

34
A combined approach
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• For transduction same approach, with spectral
  transduction method to find the subspace
• Very efficient reduced number of variables for
  SDP (symmetric matrix ), and smaller SDP
  constraint (of size of )
• For full rank, equal to the unapproximated
  problem

35
Overview

• The Normalized Cut for clustering
• A spectral relaxation
• An SDP relaxation
• Transduction based on the Normalized Cut
• A combined approach
• Experiments Conclusions

36
Experiments conclusions
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Experiments
• Red EnglishFrench versus GermanItalian (780)
• Blue Largest chapter versus other chapters

Test set performance
Training set size
37
Experiments conclusions
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Experiments
• USPS (2007) 0-4 versus 5-9, as a function of
  dimensionality of , for 3 training set sizes

ROC score
Rankof V
38
Experiments conclusions
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• Conclusions
• A fast and tight relaxation of the Normalized Cut
• An extension towards its use for transduction
• A general approximation trick to speed up SDP
  relaxations

39
Experiments conclusions
Normalized cut Spectral relaxation SDP
relaxation Transduction Combined approach -
Experiments

• To do
• Exploiting problem structure for speed up
• Statistical study of Normalized Cut transduction
• Analysis of accuracy of SDP relaxation of
  subspace approximation (cf Goemans Williamson
  for MaxCut)
• Extension to multi-class problems (see Xing and
  Jordan)