ID3 and Decision tree

1
ID3 and Decision tree

• by
• Tuan Nguyen

2
ID3 algorithm
ID3 and Decision tree

• Is the algorithm to construct a decision tree
• Using Entropy to generate the information gain
• The best value then be selected

3
Entropy
ID3 and Decision tree

• The complete formula for entropy is
• E(S) -(p)log2(p ) - (p_ )log2(p_ )
• Where p is the positive samples
• Where p_ is the negative samples
• Where S is the sample of attributions

4
Example
ID3 and Decision tree

• The Entropy of A1 is computed as the following

E(A) -29/(2935)log2(29/(2935))
35/(3529)log2(35/(3529))
0.9937 E(TRUE) - 21/(215)log2(21/(215))
5/(521)log2(5/(521))
0.7960 E(FALSE) -8/(830)log2(8/(830))
30/(308)log2(30/(308))
0.7426

• The Entropy of True

• The Entropy of False

5
Information Gain
ID3 and Decision tree

• Gain (Sample, Attributes) or Gain (S,A) is
  expected reduction in entropy due to sorting S on
  attribute A
• So, for the previous example, the Information
  gain is calculated
• G(A1) E(A1) - (215)/(2935) E(TRUE)
• - (830)/(2935)
  E(FALSE)
• E(A1) - 26/64 E(TRUE) - 38/64
  E(FALSE)
• 0.9937 26/64 0.796 38/64
  0.7426
• 0.5465

6
The complete example
ID3 and Decision tree
Consider the following table
7
Decision tree
ID3 and Decision tree

• We want to build a decision tree for the tennis
  matches
• The schedule of matches depend on the weather
  (Outlook, Temperature, Humidity, and Wind)
• So to apply what we know to build a decision tree
  based on this table

8
Example
ID3 and Decision tree

• Calculating the information gains for each of the
  weather attributes
• For the Wind
• For the Humidity
• For the Outlook

9
For the Wind
ID3 and Decision tree
10
For the Humidity
ID3 and Decision tree
11
For the Outlook
ID3 and Decision tree
Gain(S,Outlook) 0.940-(5/14)0.971 -(4/14)0.0
(5/14)0.0971 0.247
12
Complete tree
ID3 and Decision tree

• Then here is the complete tree

13
Reference

• Dr. Lees Slides, San Jose State University,
  Spring 2007
• "Building Decision Trees with the ID3 Algorithm",
  by Andrew Colin, Dr. Dobbs Journal, June 1996
• "Incremental Induction of Decision Trees", by
  Paul E. Utgoff, Kluwer Academic Publishers, 1989
• http//www.cise.ufl.edu/ddd/cap6635/Fall-97/Short
  -papers/2.htm
• http//decisiontrees.net/node/27