CS225ECE205A, Spring 2005: Information Theory

1
CS225/ECE205A, Spring 2005Information Theory

• Wim van Dam
• Engineering 1, Room 5109vandam_at_cs
• http//www.cs.ucsb.edu/vandam/teaching/S06_CS225/

2

• Help Wanted
• Note Taker
• You must be sensitive to the needs of students
  with disabilities
• 100
• Talk to me after class

3
Formalities

• We are going to have projects. The emphasis will
  be on a small paper in which you show that you
  understand and can apply the tools of information
  theory. You will also give an ultra-short
  presentation (5 min).
• Grade determination MidtermFinalProject
  1236.

4
Topics for Projects

• Information theory and
• data compression (lossless, lossy, mpeg)
• error correcting codes (802.11, DVDs, satellite)
• gambling and the stock market CT, 6, 15
• Kolmogorov complexity CT, 7
• Quantum information theory Nielsen Chuang
• analog signals, rate distortion theory CT, 13
• information theory and natural languages
• statistical distances and phylogenetics
• anything that uses some nontrivial information
  theory

5
Entropic Inequalities

• Entropic definitions and equalities
• H(X) Sx p(x) log p(x), H(XY) Sy p(y)
  H(XYy), I(XY) H(X)H(XY), D(pq) Sx p(x)
  log p(x)/q(x),
• Entropic inequalities that follow from 0p(x)1
• H(X) 0, H(XY) 0, H(X,Y) H(X),
• Less obvious inequalities
• I(XY) 0, H(X) log X, D(pq) 0,
• How do those inequalities relate?
• How to prove the not-so-obvious inequalities?

6
Information Inequality

• Theorem 2.6.3 For two probabilities distribution
  p and q we have D(pq)0 and D(pq)0 if and
  only if pq.

Q What happened here? A Application of
Jensens inequality to the concavefunction
logR?R.
7
Convex and Concave
8
Jensens Inequality

• Theorem 2.6.2 If f is a convex function on a
  random variable Z, then Ef(Z) f(EZ).
• If f is a concavefunction on a random variable
  Z, then Ef(Z) f(EZ).
• If f is strictly convex or strictly concave,
  thenEf(Z) f(EZ) implies that Z is a
  deterministic variable.
• Proof See Cover and Thomas.
• Note f is convex if and only if f is concave.

9
Jensen for the Log Function

• The function log(z) is concave for z?R, hence
  Elog Z log(EZ)and also Elog 1/Z
  log(1/EZ).
• For our purposes we typically have variables
  X,Yand some additional function gX,Y?Rsuch
  that (with Zg(X,Y)) Jensens inequality tells
  us Elog g(X,Y) log(Eg(X,Y)).
• Important exampleSx p(x) log q(x)/p(x) Elog
  q(x)/p(x)
• log(Eq(x)/p(x)) log(Sx p(x)?q(x)/p(x))
  log(Sxq(x)) 0.

10
Information Inequality, Again

• Theorem 2.6.3 For two probabilities distribution
  p and q we have D(pq)0 and D(pq)0 if and
  only if pq.

Q What happened here? A Application of
Jensens inequality to the concavefunction
logR?R.
11
For Next Tuesday

• Complete the proof that for d(X,Y) H(XY)
  H(YX) it holds that d(X,Y) d(Y,Z) d(X,Z).
• Go through as many entropic inequalities as you
  want and see which ones are the easy ones and
  which one require Jensens inequality.
• Think about a project you want to work on.