Title: Semi-supervised Learning
1Semi-supervised Learning
2Semi-supervised learning
- Label propagation
- Transductive learning
- Co-training
- Active learing
3Label Propagation
Two labeled examples
- A toy problem
- Each node in the graph is an example
- Two examples are labeled
- Most examples are unlabeled
- Compute the similarity between examples wij
- Connect examples to their most similar examples
- How to predicate labels for unlabeled nodes using
this graph?
wij
Unlabeled example
4Label Propagation
5Label Propagation
- Forward propagation
- Forward propagation
6Label Propagation
- Forward propagation
- Forward propagation
- Forward propagation
- How to resolve conflicting cases
What label should be given to this node ?
7Energy Minimization
- Labels Y?0,1n
- wi,j similarity between the i-th example and
j-th example - Energy
- Goal find label assignment Y that is consistent
with labeled examples and meanwhile minimize the
energy function E(Y)
wi,j
8Energy Minimization
Final classification results
9Label Propagation
- How the unlabeled data help classification?
10Label Propagation
- How the unlabeled data help classification?
- Consider a smaller number of unlabeled example
- Classification results can be very different
11Cluster Assumption
- Cluster assumption
- Decision boundary should pass low density area
- Unlabeled data provide more accurate estimation
of local density
12Optical Character Recognition
- Given an image of a digit letter, determine its
value
13Optical Character Recognition
- Labeled_ExamplesUnlabeled_Examples 4000
- CMN label propagation
- 1NN for each unlabeled example, using the label
of its closest neighbor
14Cluster Assumption vs. Maximum Margin
- Maximum margin classifier (e.g. SVM)
w?xb
- Maximum margin
- ? low density around decision boundary
- ? Cluster assumption
- Any thought about utilizing the unlabeled data in
support vector machine?
15Transductive SVM
- Decision boundary given a small number of labeled
examples
16Transductive SVM
- Decision boundary given a small number of labeled
examples - How will the decision boundary change given both
labeled and unlabeled examples?
17Transductive SVM
- Decision boundary given a small number of labeled
examples - Move the decision boundary to place with low
local density
18Transductive SVM
- Decision boundary given a small number of labeled
examples - Move the decision boundary to place with low
local density - Classification results
- How to formulate this idea?
19Transductive SVM Formulation
- Labeled data L
- Unlabeled data D
- Maximum margin principle for mixture of labeled
and unlabeled data - For each label assignment of unlabeled data,
compute its maximum margin - Find the label assignment whose maximum margin is
maximized
20Tranductive SVM
Different label assignment for unlabeled data ?
different maximum margin
21Transductive SVM Formulation
Another Quadratic Programming Problem
22Empirical Study with Transductive SVM
- 10 categories from the Reuter collection
- 3299 test documents
- 1000 informative words selected using MI criterion
23Co-training for Semi-supervised Learning
- Consider the task of classifying web pages into
two categories category for students and
category for professors - Two aspects of web pages should be considered
- Content of web pages
- I am currently the second year Ph.D. student
- Hyperlinks
- My advisor is
- Students
24Co-training for Semi-Supervised Learning
25Co-training for Semi-Supervised Learning
It is more easy to classify this web page using
hyperlinks
It is easy to classify the type of this web page
based on its content
26Co-training
- Two representation for each web page
Content representation (doctoral, student,
computer, university)
Hyperlink representation Inlinks Prof.
Cheng Oulinks Prof. Cheng
27Co-training
- Classifying scheme
- Train a content-based classifier using labeled
web pages - Apply the content-based classifier to classify
unlabeled web pages - Label the web pages that have been confidently
classified - Train a hyperlink based classifier using both the
labeled web pages - Apply the hyperlink-based classifier to classify
the unlabeled web pages - Label the web pages that have been confidently
classified
28Co-training
- Train a content-based classifier
29Co-training
- Train a content-based classifier using labeled
examples - Label the unlabeled examples that are confidently
classified
30Co-training
- Train a content-based classifier using labeled
examples - Label the unlabeled examples that are confidently
classified - Train a hyperlink-based classifier
- Prof. outlinks to students and inlinks from
students
31Co-training
- Train a content-based classifier using labeled
examples - Label the unlabeled examples that are confidently
classified - Train a hyperlink-based classifier
- Prof. outlinks to students and inlinks from
students - Label the unlabeled examples that are confidently
classified
32Co-training
- Train a content-based classifier using labeled
examples - Label the unlabeled examples that are confidently
classified - Train a hyperlink-based classifier
- Prof. outlinks to students and inlinks from
students - Label the unlabeled examples that are confidently
classified