Choosing Attributes

1
Choosing Attributes
Yilmaz KILIÇASLAN
2
Choosing attribute tests

• The scheme used in decision tree learning for
  selecting attributes is designed to minimize the
  depth of the final tree.
• The idea is to pick the attribute that goes as
  far as possible toward providing an exact
  classification of the examples.
• A perfect attribute divides the examples into
  sets that are all positive or all negative.
• The Patrons attribute is not perfect, but it is
  fairly good. A really useless attribute, such as
  Type, leaves the example sets with roughly the
  same proportion of positive and negative examples
  as the original set.
• All we need, then, is a formal measure of "fairly
  good" and "really useless.
• The measure should have its maximum value when
  the attribute is perfect and its minimum value
  when the attribute is of no use at all.

3
Amount of Information - I

• One suitable measure is the expected amount of
  information provided by the attribute, where we
  use the term in the mathematical sense first
  defined in Shannon and Weaver (1949).
• Information theory measures information content
  in bits.
• One bit of information is enough to answer a
  yes/no question about which one has no idea, such
  as the flip of a fair coin.

4
Amount of Information - II

• In general, if the possible answers vi have
  probabilities P(vi), then the information content
  I of the actual answer is given by

5
Amount of Information - III

• For decision tree learning, the question that
  needs answering is for a given example, what is
  the correct classification?
• An estimate of the probabilities of the possible
  answers before any of the attributes have been
  tested is given by the proportions of positive
  and negative examples in the training set.
• Suppose the training set contains p positive
  examples and n negative examples. Then an
  estimate of the information contained in a
  correct answer is

6
Amount of Information - IV

• A test on a single attribute A will not usually
  tell us all the information, but it will give us
  some of it.
• We can measure exactly how much by looking at how
  much information we still need after the
  attribute test.
• Any attribute A divides the training set E into
  subsets E1, . . . , Ev, according to their values
  for A, where A can have v distinct values.
• Each subset Ei has pi positive examples and ni
  negative examples, so if we go along that branch,
  we will need an additional I(pi/(pi ni), ni/(pi
  ni)) bits of information to answer the
  question.

7
Amount of Information - V

• A randomly chosen example from the training set
  has the ith value for the attribute with
  probability (pi n i ) / ( p n), so on
  average, after testing attribute A, we will need
  the following amount of information to classify
  the example

8
Amount of Information - VI

• The information gain from the attribute test is
  the difference between the original information
  requirement and the new requirement

9
Amount of Information - VII

• The heuristic used in the CHOOSE-ATTRIBUTE
  function is just to choose the attribute with the
  largest gain. Returning to the attributes
  considered in Figure 3 in the preceding
  presentation, we have
• Gain(Patrons) 0.541 bits.

10
Reference

• Russell, S. and P. Norvig (2003). Artificial
  Intelligence A Modern Approach. Prentice Hall.