10601 Machine Learning

1
10601 Machine Learning
September 2, 2009

• Recitation 2
• Öznur Tastan

2
Logistics

• Homework 2 is going to be out tomorrow.
• It is due on Sep 16, Wed.
• There is no class on Monday Sep 7th (Labor day)
• Those who have not return Homework 1 yet
• For details of how to submit the homework policy
  please check http//www.cs.cmu.edu/ggordon/10
  601/hws.html

3
Outline

• We will review
• Some probability and statistics
• Some graphical models
• We will not go over Homework 1
• Since the grace period has not ended yet.
• Solutions will be up next week on the web page.

4
Well play a game Catch the goof!

• Ill be the sloppy TA will make intentional
  mistakes
• Youll catch those mistakes and correct me!

Slides with mistakes are marked with
Correct slides are marked with
5
Catch the goof!!

6
Law of total probability

• Given two discrete random variables X and Y
• X takes values in

Y takes values in
7
Law of total probability

• Given two discrete random variables X and Y
• X takes values in

Y takes values in
8
Law of total probability

• Given two discrete random variables X and Y
• X takes values in

Y takes values in
9
Law of total probability

• Given two discrete random variables X and Y
• X takes values in

Y takes values in
10
Law of total probability

• Given two discrete random variables X and Y

Joint probability
Marginal probability
Conditional probability of X conditioned on Y
11
Law of total probability

• Given two discrete random variables X and Y

Formulas are fine. Anything wrong with the names?
Joint probability
Marginal probability
Conditional probability of X conditioned on Y
12
Law of total probability

• Given two discrete random variables X and Y

Joint probability of X,Y
Marginal probability
Conditional probability of X conditioned on Y
Marginal probability
13
In a strange world
Two discrete random variables X and Y take binary
values

Joint probabilities
14
In a strange world
Two discrete random variables X and Y take binary
values

Joint probabilities
Should sum up to 1
15
The world seems fine
Two discrete random variables X and Y take binary
values

Joint probabilities
16
What about the marginals?

Joint probabilities
Marginal probabilities
17
This is a strange world

Joint probabilities
Marginal probabilities
18
In a strange world

Joint probabilities
Marginal probabilities
19
This is a strange world

Joint probabilities
Marginal probabilities
20
Lets have a simple problem

Joint probabilities
Marginal probabilities
21
Conditional probabilities

• What is the complementary event of P(X0Y1) ?
• P(X1Y1) OR P(X0Y0)

22
Conditional probabilities

• What is the complementary event of P(X0Y1) ?
• P(X1Y1) OR P(X0Y0)

23

• The game ends here.

24
Independent number of parameters

• Assume X and Y take Boolean values 0,1
• How many independent parameters do you need to
  fully specify
• marginal probability of X?
• the joint probability of P(X,Y)?
• the conditional probability of P(XY)?

25
Independent number of parameters

• Assume X and Y take Boolean values 0,1
• How many independent parameters do you need to
  fully specify
• marginal probability of X?
• P(X0) 1 parameter only because
  P(X1)P(X0)1
• the joint probability of P(X,Y)?
• P(X0, Y0) 3 parameters
• P(X0, Y1)
• P(X1, Y0)
• the conditional probability of P(XY)?

26
Number of parameters

• Assume X and Y take Boolean values 0,1?
• How many independent parameters do you need to
  fully specify
• marginal probability of X?
• P(X0) 1 parameter only P(X1)
  1-P(X0)
• How many independent parameters do you need to
  fully specify the joint probability of P(X,Y)?
• P(X0, Y0) 3 parameters
• P(X0, Y1)
• P(X1, Y0)
• How many independent parameters do you need to
  fully specify the conditional probability of
  P(XY)?
• P(X0Y0) 2 parameters
• P(X0Y1)

27
Number of parameters

• What about P(X Y,Z) , how many independent
  parameters
• do you need to be able to fully specify the
  probabilities?
• Assume each RV takes
• m values

P(X Y,Z)
q values
n values
28
Number of parameters

• What about P(X Y,Z) , how many independent
  parameters
• do you need to be able to fully specify the
  probabilities?
• Assume each RV takes
• m values

P(X Y,Z)
q values
n values
Number of independent parameters (m-1)nq
29
Graphical models
A graphical model is a way of representing
probabilistic relationships between random
variables Variables are represented by
nodes Edges indicates probabilistic
relationships
You miss the bus
Arrive class late
30
Serial connection
Is X and Z independent?
?
31
Serial connection
Is X and Z independent?
X and Z are not independent
32
Serial connection
Is X conditionally independent of Z given Y?
?
33
Serial connection
Is X conditionally independent of Z given Y?
Yes they are independent
34
How can we show it?
Is X conditionally independent of Z given Y?
35
An example case
36
Common cause
Age
Shoe Size
Gray Hair
X and Y are not marginally independent X and Y
are conditionally independent given Z
37
Explaining away
Flu
Allergy
Z
X
Y
Sneeze
X and Z marginally independent X and Z
conditionally dependent given Y
38
D-separation

• X and Z are conditionally independent given Y if
  Y d-separates X and Z

Neither Y nor its descendants should be observed
Path between X and Z is blocked by Y
39
D-separation example
Is B, C independent given A?
40
D-separation example
Is B, C independent given A?
Yes
41
D-separation example
Observed, A blocks the path
Is B, C independent given A?
Yes
42
Observed, A blocks the path
Is B, C independent given A?
Yes
not observed neither its descendants
43
D-separation example
Is A, F independent given E?
44
Is A, F independent given E?
Yes
45
Is A, F independent given E?
Yes
46
Is C, D independent given F?
47
Is C, D independent given F? No
48
Is A, G independent given B and F?
49
Is A, G independent given B and F? Yes
50
Naïve Bayes Model
J
D
C
R
J The person is a junior D The person knows
calculus C The person leaves in campus R Saw
the Return of the King more than once
51
Naïve Bayes Model
What parameters are stored?
J
D
C
R
J The person is a junior D The person knows
calculus C The person leaves in campus R Saw
the Return of the King more than once
52
Naïve Bayes Model
P(J)
J
D
C
R
P(R/J1) P(R/J0)
P(D/J1) P(D/J0)
P(C/J1) P(C/J0)
J The person is a junior D The person knows
calculus C The person leaves in campus R Saw
the Return of the King more than once
53
Naïve Bayes Model
P(J)
J
D
C
R
P(R/J1) P(R/J0)
P(D/J1) P(D/J0)
P(C/J1) P(C/J0)
J The person is a junior D The person knows
calculus C The person leaves in campus R Saw
the Return of the King more than once
54
We have the structure how do we get the
CPTs?Estimate them from observed data
55
Naïve Bayes Model
J The person is a junior D The person knows
calculus C The person leaves in campus R Saw
the Return of the King more than once
P(J)
J
D
C
R
P(R/J) P(R/J)
P(C/J) P(C/J)
Suppose a new person come and says I
dont know calculus I live in campus
I have seen The return of the king five
times
P(C/J) P(C/J)
What is the probability that he is a Junior?
56
Naïve Bayes Model
Suppose a person says I dont know calculus
D0 I live in campus C1 I have not seen The
return of the king five times R1
J
What is the probability that he is a Junior?
P(J1/D0,C1,R1)
D
C
R
57
What is the probability that he is a Junior?
J
To calculate this marginalize over J
D
C
R
58
Naïve Bayes Model