Artificial Intelligence: Advanced Search and Machine Learning

1
Artificial Intelligence Advanced Search and
Machine Learning

• Lecture 15

• Decision Tree Induction
• Neural Nets
• Evaluating Machine Learning Systems

2
Decision Tree Induction

• Weve looked so far at how machine learning may
  be applied to classification problems.
• So far just looked at how we can come up with
  very simple rules, based on features that should
  and should not be present, allowing us to do a
  yes/no (e.g., tiger/not tiger) classification.
• But more useful structure for classification is
  the decision tree..

3
Decision Tree
Traverse tree based on values of features. Leaf
node gives classification. (equivalent to
logical formulae that may contain
disjunctions, but more natural representation,
easier for humans to use)
Stripy
yes
no
Long Tail
Big Teeth
no
no
yes
yes
Big Teeth
Lion
Rabbit
zebra
no
yes
Lemur
Tiger
4
Decision Tree Induction

• Aims to come up with simplest decision tree
  covering the example data.
• Simplest tree is more likely to be a general rule
  which works for new cases, rather than a complex
  formula that only applies to the given data.
• Resulting tree is useful representation that can
  easily verified by humans and used by humans or
  programs.

5
A sample data set(with extra tigers)
6
Creating the Decision Tree

• We create a decision tree top down looking for
  features to label nodes with.
• The feature chosen should be a good
  discriminating feature, that helps split the
  examples into different classes (e.g, tiger vs
  not tiger).
• For our tiger data set the best disciminating
  feature is big teeth..
• We label the top node, and look at which examples
  go into yes and no branches for that feature.

7
Tiger Tree
Big Teeth?
yes
no
2, 6
1, 3, 4, 5
Yes branch has 3 tigers, 1 non tiger. No
branch just has non tigers. .. So we need to do
further work on yes branch, but no branch can
have label not tiger
8
Tiger Tree

• We look for feature that best splits examples 1,
  3, 4, 5 into tigers and non-tigers.
• Best feature is stripy

Big Teeth?
yes
no
Stripy
Not a tiger
Not a tiger
Tiger
9
In general..

• To select a discriminating feature given a set
  of examples..
• Pick feature that best splits examples into
  different result categories.
• For each value of feature (e.g., yes/no)
• Create branch in tree labelled with value of
  feature.
• Find subset S of examples such that
  featurevalue.
• If all examples in S are in same category, mark
  relevant node with that category. Otherwise
  repeat selection process using set S.

10
Summary Decision Tree Induction

• Goal is to create decision tree allowing you to
  classify things based on features.
• Do this by taking example data set, and
  repeatedly looking for most discriminating
  feature (that best splits data into different
  classes).
• Works for multiple classes, and data where
  feature can have a number of possible values.

11
Neural Networks

• Back to biologically inspired approaches..
• Neural Networks are inspired by biological
  neurons..

Synapse
Soma
Axon
Axon
Dendrites
12
Biological Neuron

• Human brain consists of thousands of millions of
  neurons.
• Each connected to thousands of other neurons.
• Neuron receives inputs from its neighbours - may
  become excited/activated, giving output in turn
  to other neurons.
• Behaviour can change if connections (synapses)
  between neurons are weakened or strengthened.

13
Perceptron

• The perceptron is a simple computational neuron.
• It takes a number of inputs, with weights on the
  connections (cf strength of synapse)
• if w1x1w2x2w3x3w4x4 gt thresholdthen output
  1else output 0

x4
w4
ouput
x3
w3
inputs
w2
x2
w1
x1
14
Perceptron

• Example All the weights are 0.5
• x1 1, x20, x30, x40, threshold 1.
• Output 0
• x11, x21, x31, x40, threshold 1.
• Output 1
• In general can of course have any number of
  inputs, and different weights on each.
• Also may have many interconnected perceptrons.

15
Learning in Neural Nets

• Machine Learning for Neural Nets means adapting
  the weights, given some examples.
• We keep on adjusting the weights a bit, until the
  neural net gives the output the examples suggest.
• Tiger example
• Let each input in the network correspond to one
  feature (e.g., x1stripy?). Input has value 1 if
  feature has value yes.
• We want an output of 1 if it is a tiger.

16
Learning in NNs

• Basic learning rule is
• See what the output is for an example.
• If it is wrong, adjust the weights a bit.
• Keep going with all the examples, and repeat
  until weights converge and right results
  obtained.
• E.g., tiger 1, intitial weights

ouput
1
0.5
1
0.5
1
OK, dont change .
0.5
1
17
Learning tigers

• Tiger 3 - Not quite right.
• So increase weights on active connections

output
1
0.5
0
0.5
1
0.5
0
output
1
0.6
1
0.6
1
0.5
0
18
Learning tigers

• Example 6 still not quite right.
• Decrease weights on active connections.
• End up with

output
1
0.6
0
0.5
1
0.4
0
19
Perceptron Learning

• Repeat
• For each example
• If actual output is 1 and target is 0, decrease
  weights on active connections by small amount.
• If actual output is 0 and target is 1, increase
  weights on active connections by small amount.
• Until network gives right results for all
  examples.
• (Active connections are those for which the input
  is 1).

20
NN Summary

• Neural Networks provide alternative approach to
  learning, inspired by architecture of brain.
• Based on repeatedly modifying weights in simple
  neurons.
• Result is (often) a network that correctly
  classifies, but which is difficult for human to
  interpret.
• Contrasts with decision trees, which produce
  human-readable output.

21
Evaluating Machine Learning

• How do we test whether the rules/networks
  produced by a machine learning system are any
  good?
• They may correctly classify all the training data
  .. But may lack generality and fail on any new
  unseen data. Consider
• Data small green flying insect large red
  non-flying insect.
• Rule if small and green and flying OR large and
  red and non-flying THEN insect.
• Rule unlikely to work for new insects.

22
Evaluation

• One way is to split the set of examples that a
  human has classified correctly into a training
  set and a test set.
• Train on the training set (say, 50 examples) to
  obtain the rules/networks.
• Then take the test set. Ignore for now the humans
  classification of the data. Test to see what the
  rules/networks predict.
• Compare this with the actual (human)
  classification.
• Score the system according to how many of the
  test case examples are correctly classified using
  the learned rules.

23
Summary

• Inductive Learning systems use example data to
  produce general rules (or networks, or decision
  trees).
• These rules can be used to classify new data.
• Essentially a search/optimisation problem
  search through all possible rules to find set
  that covers the given data best.
• Various approaches GA, NN, Decision trees.