Andrea Montanari and Ruediger Urbanke

1
Phase Transitions in Coding, Communications, and
Inference
Andrea Montanari and Ruediger Urbanke TIFR Tuesday
, January 6th, 2008
2
Outline

• 1) Thresholds in coding, the large size limit
•  (definition and density evolution
  characterization)            
• 2) The inversion of limits (length to infty vs
  size to infty)                  
  

                  3) Phase
transitions in measurements                   
 (compressed sensing versus message
passing,   dense versus sparse matrices) 4)
Phase transitions in collaborative filtering
         (the low-rank matrix model)
3
(No Transcript)
4
Channel Coding
code
C000, 010, 101, 111
xMAP(y)argmaxX in C p(x y)
decoding
xiMAP(y)argmaxXi p(xi y)
5
Factor Graph Representation of Linear Codes
every linear code
parity-check matrix
(7, 4) Hamming code
Tanner, Wiberg, Koetter, Loeliger, Frey
6
Low-Density Parity Check Codes
(3, 4)-regular codes
Gallager 60
number of edges is linear in n
7
Ensemble
8
Variations on the Theme
degree distributions as well as structure
irregular LDPC ensemble
regular RA ensemble
irregular MN ensemble
irregular RA ensemble
ARA ensemble
turbo code
protograph
irregular LDGM ensemble
(Luby, Mitzenmacher, Shokrollahi, Spielman, and
Stehman)
Divsalar, Jin, and McEliece
Jin, Khandekar, and McEliece
Abbasfar, Divsalar, Kung
Berrou and Glavieux
Thorpe, Andrews, Dolinar
Davey, MacKay
9
Message-Passing Decoding -- BEC
0
0
0
0
0
0
?
00
0
?
0
?
?
?
?
decoded
0
0
?
?
?
0?
0? ?
?
?
00
00 0
0
?
?
?
?
decoded
10
Message-Passing Decoding -- BSCGallager Algorithm
11
Asymptotic Analysis Computation Graph
probability that computation graph of fixed depth
becomes tree tends to 1 as n tends to infinity
12
(No Transcript)
13
(No Transcript)
14
Asymptotic Analysis Density Evolution -- BSC,
Gallager Algorithm
xt e (1-p(xt-1))(1-e) p-(xt-1)
p(x)((1(1-2x)r-1)/2) l-1
p-(x)((1-(1-2x)r-1)/2) l-1
phase transition eBP so that
xt ? 0 for elt eBP
xt ? x8gt0 for egt eBP
15
Asymptotic Analysis Density Evolution -- BP
16
Inversion of Limits
size versus number of iterations
17
Density Evolution Limit
18
Density Evolution Limit
19
Practical Limit
20
Practical Limit
21
The Two Limits
Easy (Density Evolution Limit) Hard(er)
(Practical Limit)
22
Binary Erasure Channel
DE Limit
implies
Practical Limit
23
What about General Case
expansion
probabilistic methods
Korada and U.
24
Expansion
expansion 1-1/l
Miller and Burshtein Random element of LDPC(l,
r, n) ensemble is expander with expansion close
to 1-1/l with high probability
25
Why is Expansion Useful?
26
Setting Channel
27
Setting Ensemble
28
Setting Algorithm
29
Aim Show for this setting that ...
DE Limit
implies
Practical Limit
30
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

31
Linearized Decoding Algorithm
32
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

33
Combine with Density Evolution
34
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

35
Correlation and Interaction
Expected growth
1
0
0
0
? lt 1
2 e
2 e
(r-1)
0
1
(r-1)
Problem interaction correlation
36
Correlation and Interaction
37
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

38
Witness
39
Witness
40
Witness
41
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

42
Monotonicity
43
Randomizing the Noise Outside
FKG

?
?
/
randomizing noise outside the witness increases
the probability of error
44
Proof Outline

• linearize the algorithm
• combine with density evolution
• correlation and interaction
• witness
• randomizing noise outside the witness
• sub-critical birth and death process

45
Expansion
random graph has expansion close to expansion of
a tree with high probability ? this limits
interaction
1
0
0
0
0
1
46
References
For a list of references see http//ipg.epfl.ch/d
oku.php?idencourses2007-2208mct
47
Results
48
Open Problems
P
b
0.4
0.3
0.2
0.1
0.0
0.2
0.4
0.6
0.8
channel entropy