Performance of LDPC Codes Under Noisy MessagePassing Decoding

1
Performance of LDPC Codes Under Noisy
Message-Passing Decoding

• Lav R. Varshney
• Laboratory for Information and Decision Systems
• Research Laboratory of Electronics
• Massachusetts Institute of Technology
• August 29, 2007

2
(No Transcript)
3
(No Transcript)
4
Rüdiger Urbanke
Emre Telatar
5
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

6
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

7
Problem of Reliable Communication

• Shannon (1948), Gallager (1963), Richardson
  Urbanke (2001), etc.

8
Problem of Fault-Tolerant Computing

• Von Neumann (1956), Moore Shannon (1956), Elias
  (1958), etc.

9
Our Problem
10
Our Problem

• Information theory and coding theory
• unreliable signals / reliable circuits
• Fault-tolerant computing theory
• reliable signals / unreliable circuits
• What is possible and what is impossible with
• unreliable signals / unreliable circuits?

11
Previous Studies

• Basic idea of problem was studied by Michael
  Taylor in the context of constructing a reliable
  information storage device using unreliable
  components (1968) extended in (Kuznetsov, 1973)
• Doesnt seem to have been rediscovered like
  LDPC codes

Taylors doctoral committee
12
Practical Motivations

• Noise is present in physical implementations of
  decoders
• Especially with shrinking physical dimensions and
  desired reduction in power consumption

• If messages are continuous-valued, there must be
  some noise or quantization when representing them

13
Practical Motivations

• Loeliger et al. (2001) have observed that
  decoders are robust to noise in physical
  implementations
• the quantitative analysis of these effects is a
  challenging theoretical problem
• According to circuit and communication system
  designers at MIT RLE, mathematical guidelines for
  power control in decoding circuits would be quite
  useful

14
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Performance of Density Evolution
• Information Storage
• Density Evolution for Another Example

15
Overview of LDPC codes

• Regular LDPC codes are binary linear codes
• An ensemble of codes with parameters (l,r,n) is
  constructed using bipartite graphs. There are n
  variable nodes and nl/r check nodes
• Design rate is 1 - l/r (actual rate might be
  higher since not all checks independent)

?
16
Overview of LDPC codes

• Code is drawn uniformly from the ensemble by
  picking ? randomly from the (nl)! permutations
• (There can be double edges, etc.)
• Can also consider irregular ensembles, defined by
  degree distributions (?(x), ?(x))

17
Decoding LDPC codes

• MAP decoding is rather complicated (Gallager,
  1963 Berlekamp, McEliece, van Tilborg, 1978)
• LP and NLP decoding are relaxations of MAP
  decoding (Feldman et al., 2005 Yang et al.,
  2006)
• The most common class of decoders based on
  iterative message-passing
• Implementations of message-passing on factor
  graphs are used in practice (Blanksby Howland,
  2002 etc.)

18
Message-Passing Decoding

• At time t 0, each variable node has a
  realization of the channel output, ri ? ?
• Each variable node sends a message to a
  neighboring check node through a (noisy) message
  channel
• Check nodes perform a (noisy) computation F(t)
  ?r-1 ? ?
• Each check node sends back a message to each
  neighboring variable node through a (noisy)
  message channel
• Variable nodes perform a (noisy) computation
  ?(t) ? ?l-1 ? ?

19
Message-Passing Decoding

• The message-passing is repeated iteratively
• Each iteration t is one complete variable ?
  check ? variable cycle
• In our analysis, the messages are random
  variables interested in tracking their
  distributions
• The messages are usually log-likelihood ratios or
  some approximation thereof
• Use probability of bit error as the performance
  criterion

20
Channels

• Since binary code, restrict attention to BPSK
  modulation with channel input alphabet 1
• No particular restrictions on channel output
  alphabet or on message-passing channels
• Assume w.l.o.g. that alphabets for all four
  message-passing terminals are identical

21
Merging Noises

• For simplicity, assume noiseless computation
• Only consider noise in message-passing
• Not much loss (theorems can easily be extended)

Noisy Computer
Noiseless Computer
Noisy Channel
Noisy Channel
Noisy Channel
Noiseless Computer
Noisy Channel
Noisy Channel
22
A Folk Theorem
Folk Theorem If everything is unreliable, there
is no way of driving probability of error to
zero Corollary Fault-tolerant computing must use
some sort of trick such as a reliable voting
mechanism, or allow outputs to be in coded form
(thereby assuming a reliable decoder)

• Proceed without any tricks

23
Previous Studies

• Note that our inclusion of stochastic noise in a
  message-passing decoder is quite different than
  quantized message-passing algorithms that work
  deterministically
• Tehrani et al.s Stochastic Decoding of LDPC
  Codes, 2006 is a bit different

24
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

25
Performance Analysis

• Performance analysis of a particular code of
  finite block length n is difficult, see Ihler et
  al., 2005.
• Study average asymptotic performance of an
  ensemble of codes
• Generalize the results of Richardson and Urbanke
  (RU) to the case of noise in the decoder

26
All-One Codeword

• If certain symmetry conditions satisfied by
  system, then probability of error is
  conditionally independent of codeword that is
  sent
• Assume passed messages are in log-likelihood like
  form, i.e. sign indicates the digit estimate
• Require channel symmetry, check node symmetry,
  and variable node symmetry, just like RU
• Also require message channel symmetry p(µ m
  ? m) p(µ -m ? -m),where µ is any
  message received at a node when message sent from
  the opposite node is ?

27
Concentration Ensemble Average

• Performances of almost all LDPC codes closely
  match average performance of ensemble from which
  they are drawn. Average is over instance of the
  code, realization of channel noise, and
  realization of decoder noise
• Assume that number of decoder iterations fixed at
  some finite t. Let Z be number of incorrect
  values held among all variable nodes at the end
  of tth iteration (for particular code, channel
  noise realization, and decoder noise realization)
  and let EZ be expected value of Z

28
Concentration Ensemble Average
Lemma There exists a positive constant ß
ß(l,r,t) such that for any e gt 0, Pr Z - EZ
gt nle/2 2e-ße²n Proof Basically follows
RU. Construct Doobs Martingale by sequentially
revealing instance of code, channel noise
realization, decoder noise realization. Compute
bounded difference constants and apply
Hoeffding-Azuma inequality.
29
Sequentially Expose Random Things

• Sequentially expose ln edges of bipartite graph

30
Sequentially Expose Random Things

• In these steps, difference bounded by 8(lr)t

31
Sequentially Expose Random Things

• Sequentially expose n received values

r
32
Sequentially Expose Random Things

• In these steps, difference bounded by 2(lr)t

r
r
r
r
r
r
r
r
r
r
33
Sequentially Expose Random Things

• Sequentially expose 2tln decoder noises

r
r
r
r
r
r
r
r
r
r
34
Sequentially Expose Random Things

• In these steps, difference bounded by 2(lr)t

r
r
r
r
r
r
r
r
r
r
35
Hoeffding-Azuma Inequality

• Let Z0,Z1,,Zm be a Martingale such that Zi -
  Zi-1 ?i lt 8 for each i 1,,m,Then, for all
  n 1 and any a gt 0, Pr Z0 - Zn avn 2
  exp-a2n / 2(?1² ?n²)
• Apply inequality to the Martingale constructed by
  sequentially exposing things to yield Pr Z -
  EZ gt nle/2 2e-ße²n

36
Convergence to the Cycle-Free Case

• Take n ? 8 before increasing t
• Average performance of LDPC ensemble converges to
  performance of tree ensemble
• Use computation tree with independent messages

37
Convergence to the Cycle-Free Case

• Let p be expected number of incorrect messages in
  a tree-like neighborhood

Lemma There exists a positive constant ?
?(l,r,t) such that for any e gt 0 and n gt
2?/e, EZ - nlp lt nle/2 Proof Identical to
RU. Basically, probability of repeats in
computation tree goes to zero.
38
Concentration around Cycle-Free Case
Lemma There exist positive constants ß ß(l,r,t)
and ? ?(l,r,t) such that for any e gt 0 and n gt
2?/e, Pr Z - nlp gt nle 2e-ße²n Proof
Follows directly from previous two
39
Simplifications Made

• With the three simplifications, removed all
  randomness from explicit consideration and made
  all messages independent
• Reduced to problem of analyzing a discrete-time
  dynamical system
• Density evolution

40
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

41
A Simple Example

• Consider decoding the output of a BSC(e) using
  the Gallager A algorithm
• At a check node, the outgoing message along edge
  e is the product of all incoming messages
  excluding the one incoming on e
• At a variable node, the outgoing message is the
  original received message unless all incoming
  messages give the opposite conclusion
• All messages in decoding are passed through
  identical BSC(a) channels

42
Deriving the Density Evolution Equation

• Let ? (2a-1)(2x-1) and define

Theorem The density evolution equationwith
initial condition x0 e almost surely gives the
performance of this system for large n. Proof
Follows from computation and lemmas
43
The Folk Theorem Revisited

• Let e 0, a gt 0
• The density evolution recursion starts at zero
  but does not have a fixed point there
• Hence as t ? 8, the probability of error
  increases to some non-zero value
• Similarly, no fixed-point at zero for e gt 0, a gt
  0
• Reconfirms the Folk Theorem
• Analyze performance anyway maybe can drive
  probability of error to something small, if not
  zero

44
(No Transcript)
45
(No Transcript)
46
Example

• If a 110-10 and e 0.0394636560, then final
  error probability is 7.822810-11
• Rather than considering the (3,6) regular code,
  one can consider Bazzi et al.s optimized
  irregular rate 1/2 code

47
(No Transcript)
48
Optimizing code for noisy decoder

• Optimizing the code when taking decoder noise
  into account is open question
• Unclear whether one can do better than Bazzi et
  al. for a gt 0
• (Superficially) similar question to the
  Chen-Tong-P.K. Varshney channel-aware distributed
  detection problem, presented on Monday

49
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

50
Reliable Memories of Unreliable Parts

• Taylor developed reliable information storage in
  memories designed from unreliable components
• Each type of memory has an information storage
  capacity C such that for memory redundancies
  greater than 1/C arbitrarily reliable information
  storage can be achieved
• The complexity of a stable memory is required to
  be bounded by ?k, where k is information storage
  capability and ? does not depend on k
• LDPC message-passing decoders have this linear
  complexity property

51
Reliable Memories of Unreliable Parts

• Allow coded outputs to be read out of the memory
  (assume that eventually there is a noiseless
  decoder)
• Construct a memory with noisy registers and an
  LDPC noisy Gallager A correcting network. Taylor
  shows that this has C gt 0
• Using our density evolution results and the ML
  decoding threshold (Montanari, 2005), the
  capacity can be computed

52
Outline

• Motivation and Background
• Basic Problem Formulation
• Tools for Performance Analysis
• Concentration around Ensemble Average
• Convergence to Cycle-free Performance
• Density Evolution for a Simple Example
• Information Storage
• Density Evolution for Another Example

53
Another Example

• Consider
• binary input AWGN channel with variance s²
• Gaussian approximation to belief propagation as
  the decoder (Chung et al., 2001), which yields a
  one-dimensional density evolution recursion
• Bounded additive noise from oracle adversary in
  message-passing, -d/2 w d/2 (worse than
  stochastic noise)

54
Adversarial Noise

• Assume that all-one codeword was sent
• Then all log-likelihood messages should be as
  positive as possible
• Worst noise is to subtract d/2 from all messages
  that are passed since oracle adversary knows the
  codeword sent, this can be done
• Adversarial noise becomes deterministic, so easy
  to derive density evolution recursion
• Probability of error going to zero is equivalent
  to message value going to 8

55
Density evolution

• State variable mV is mean of Gaussian
  distribution at the variable node
• Noiseless density evolution (Chung, et al.)
• Density evolution with adversarial noise

56
Analysis of Density Evolution

• Are there cases with s gt 0 and d gt 0 such that
  the state variable mV goes to 8?
• Yes and there is a threshold phenomenon
• With possibly unbounded messages and bounded
  noise on them, an instance where the Folk Theorem
  is false

57
Numerical Computation of Thresholds

• For (3,6) regular LDPC code

58
Numerical Computation of Thresholds
59
Summary

• Extended asymptotic analysis tools to the case of
  noisy decoders
• Derived density evolution equations for two
  simple examples
• In one case, the probability of error can be
  driven to small values, displaying a nonlinear
  estimation threshold
• In the other case, the probability of error can
  be driven to arbitrarily small values, achieving
  Shannon reliability even when there is noise in
  the decoder, thereby falsifying a folk theorem