Near-Minimax Optimal Learning

1
Near-Minimax Optimal Learning with Decision Trees
Rob Nowak and Clay Scott
University of Wisconsin-Madison and Rice
University
nowak_at_engr.wisc.edu
Supported by the NSF and the ONR
2
Basic Problem
Classification build a decision rule based on
labeled training data
Given n training points, how well can we do ?
3
Smooth Decision Boundaries
Suppose that the Bayes decision boundary behaves
locally like a Lipschitz function
Mammen Tsybakov 99
4
Dyadic Thinking about Classification Trees
recursive dyadic partition
5
Dyadic Thinking about Classification Trees
Pruned dyadic partition
Pruned dyadic tree
Hierarchical structure facilitates optimization
6
The Classification Problem
Problem
7
Classifiers
The Bayes Classifier
Minimum Empirical Risk Classifier
8
Generalization Error Bounds
9
Generalization Error Bounds
10
Generalization Error Bounds
11
Selecting a good h
12
Convergence to Bayes Error
13
Ex. Dyadic Classification Trees
Bayes decision boundary
labeled training data
pruned RDP
complete RDP
Dyadic classification tree
14
Codes for DCTs
code-lengths
ex
code 0001001111 6 bits for leaf labels
15
Error Bounds for DCTs
16
Rate of Convergence
Suppose that the Bayes decision boundary behaves
locally like a Lipschitz function
Mammen Tsybakov 99
C. Scott RN 02
17
Why too slow ?
because Bayes boundary is a (d-1)-dimensional
manifold good trees are
unbalanced
all T leaf trees are equally favored
18
Local Error Bounds in Classification
Spatial Error Decomposition
Mansour McAllester 00
19
Relative Chernoff Bound
20
Relative Chernoff Bound
21
Local Error Bounds in Classification
22
Bounded Densities
23
Global vs. Local
Key local complexity is offset by small volumes!
24
Local Bounds for DCTs
25
Unbalanced Tree
Global bound
J leafs depth J-1
Local bound
26
Convergence to Bayes Error
27
Concluding Remarks

data dependent bound
Neural Information Processing Systems 2002, 2003
nowak_at_engr.wisc.edu