Support Vector Regression

1
Support Vector Regression
2
SVR
Drawings and illustrations from Bernhard
Schölkopf, and Alex Smola Learning with Kernels
(MIT Press, Cambridge, MA, 2002)
3
SVR - History
Based on Learning Theory, consisting of few
axioms on learning errors Started in 1960s,
still actively developed SVRs recently
outperformed NNs in recognition tests on US
Postal Services standard set of handwritten
characters libSVM by Chih-Chung Chang and
Chih-Jen Lin provides fast and simple to use
implementation, extended as requests (e.g. from
HEP) come in
4
Formulation of Problem

• Training sample X, observed results Y
• Goal f with yf(x)
• Simplicity
• Linear case,

5
Optimizing the Confidence
Optimal confidence maximal margin Minimize
quadratic problem with Quadratic problem
Unique solution!
6
Non-Linearity
Introduce mapping to higher dimensional space
e.g. Gaussian kernel
7
Calculation
8
L2 b Tagger Parameters
9
L2 b Tagger Parameters
10
L2 b Tagger Output
SVR
NN
11
L2 b Tagger Discussion

• Complex problem increases number of SVs
• Almost non-separable classes still almost
  non-separable in high dimensional space
• High processing time due to large number of SVs
• NNs show better performance for low-information,
  low-separability problems

12
Higgs Parameters
Higgs SVR analysis by Daniel Whiteson, UC Berkley
13
Higgs Parameters
14
Higgs Output
? Background Signal ?
? Background Signal ?
15
Higgs Purity / Efficiency
Purity
16
Kernel Width
Integrated Significance
Kernel Width
17
Summary

• SVR often superior to NN
• Not stuck in local minima unique solution
• Better performance for many problems
• Implementation exists, actively supported by the
  development community
• Further information www.kernel-machines.org

Time for SVR _at_ HEP!
18
L2 b Tagger Correlation
NN
NN
SVR
SVR
b
udcs