SVM as an Unconstrained Minimization Problem

1
SVM as an Unconstrained Minimization Problem

• Change (QP) into an unconstrained MP

• Reduce (n1m) variables to (n1) variables

2
Smooth the Plus Function Integrate
Step function
Sigmoid function
p-function
Plus function
3
SSVM Smooth Support Vector Machine

4
Newton-Armijo Algorithm for SSVM
5
Newton Method Quadratic Approximation of SSVM

• The sequence

generated by solving a
quadratic approximation of SSVM, converges to the
of SSVM at a quadratic rate.
unique solution

• Converges in 6 to 8 iterations

• At each iteration we solve a linear system of

• n1 equations in n1 variables

• Complexity depends on dimension of input space

• It might be needed to select a stepsize (Armijo)

6
Comparisons of SSVM with other SVMs
Tenfold test set correctness (best in Red)CPU
time in seconds
SSVM
QP
LP
Linear Eqns.
7
Two-spiral Dataset(94 White Dots 94 Red Dots)
8
The Perceptron Algorithm (Dual Form)
9
Nonlinear SVM Motivation
10
Nonlinear Smooth SVM
Nonlinear Classifier

• Use Newton algorithm to solve the problem

• Nonlinear classifier depends on entire dataset

11
Difficulties with Nonlinear SVM for Large
Problems

• Long CPU time to compute the dense kernel matrix

• Runs out of memory while storing the kernel
  matrix

• Separating surface depends on almost entire
  dataset