A Brief Introduction to boosting

1
A Brief Introduction to boosting
2
History of Boosting

• "Kearns Valiant (1989) proved that learners
  performing only slightly better than random, can
  be combined to form an arbitrarily good ensemble
  hypothesis."
• Schapire (1990) provided the first polynomial
  time Boosting algorithm.
• Freund (1995) Boosting a weak learning algorithm
  by majority
• Freund Schapire (1995) AdaBoost. Solved many
  practical problems of boosting algorithms. Ada
  stands for adaptive.

3
Boosting Methods

• A method for improving classifier accuracy.
• Many weak classifiers combined to produce a
  strong classifier.
• Weak Classifiers ( error rate slightly better
  than random guess).

4
Schematic of Adaboost
Weighted Sample
hT(x)
..
Weighted Sample
h3(x)
Weighted Sample
h2(x)
Training Sample
h1(x)
(xi, yi)
5
AdaBoost Algorithm
Given m examples (x1, y1), , (xm, ym) where
xiÎX, yiÎY-1, 1
Initialize D1(i) 1/m
For t 1 to T
The weight Adapts. The bigger et becomes the
smaller at becomes.
Boost example if incorrectly predicted.
Zt is a normalization factor.
Linear combination of models.
6
Example of a Good Classifier

-


-
-

-

-
7
Round 1 of 3

-


-
-

-

-
e1 0.30 a10.42
8
Round 2 of 3

-


-
-

-

-
e2 0.21 a20.65
9
Round 3 of 3

-


-
-

-

-
e3 0.14 a20.92
10
Final Hypothesis
Hfinal sign 0.42(h1? 1-1) 0.65(h2? 1-1)
0.92(h3? 1-1)
11
AdaBoost on the Example
12
Shortcomings

• Actual performance of boosting can be
• dependent on the data and the weak learner
• Boosting can fail to perform when
• Insufficient data
• Overly complex weak hypotheses
• Weak hypotheses which are too weak

13
References

• Y.Freund and R.E. Schapire. A short introduction
  to boosting. Journal of Japanese Society for
  Artificial Intelligence, 14(5)771-780, September
  1999.