Trading Convexity for Scalability

1
Trading Convexity for Scalability

• Marco A. Alvarez
• CS7680
• Department of Computer Science
• Utah State University

2
Paper

• Collobert, R., Sinz, F., Weston, J., and Bottou,
  L. 2006. Trading convexity for scalability. In
  Proceedings of the 23rd International Conference
  on Machine Learning (Pittsburgh, Pennsylvania,
  June 25 - 29, 2006). ICML '06, vol. 148. ACM
  Press, New York, NY, 201-208.

3
Introduction

• Previously in Machine Learning
• Non-convex cost function in MLP
• Difficult to optimize
• Work efficiently
• SVM are defined by a convex function
• Easier optimization (algorithms)
• Unique solution (we can write theorems)
• Goal of the paper
• Sometimes non-convexity has benefits
• Faster training and testing (less support
  vectors)
• Non-convex SVMs (faster and sparser)
• Fast transductive SVMs

4
From SVM

• Decision function
• Primal formulation
• Minimize w so that margin is maximized
• w is a combination of a small number of data
  (sparsity)
• Decision boundary is determined by the support
  vectors
• Dual formulation

s.t.
5
SVM problem

• Number of support vectors increases linearly with
  L
• Cost attributed to one example (x,y)
• From

6
Ramp Loss Function

• Given

Outliers
Non SV
7
Concave-Convex Procedure (CCCP)

• Given a cost function
• Decompose into a convex part and a concave part
• Is guaranteed to decrease at each iteration

8
Using the Ramp Loss
9
CCCP for Ramp Loss
10
Results
11
Speedup
12
Time and Number of SVs
13
Transductive SVMs
14
Loss Function

• Cost to be minimized

15
Balancing Constraint

• Necessary for TSVMs

16
Results

17
Training Time
18
Quadratic Fit