Shannons Theory

1
Shannons Theory

• A Eugene M. Taranta II Presentation

2
Things to Come

• Claude Shannon
• Results of Research
• Hierarchy
• Error Detection
• Error Correction
• Shannons Theory

3
Claude E. Shannon

• Research Mathematician for Bell Laboratories
  during the 1940s

• Intuitively, concerned with telephone
  communications
• Large amounts of data
• Maintain data integrity

• A Mathematical Theory of Communication 1948

4
Results of Research

• Gave life to Information Theory

• Introduced concept of the Communication Channel

• Applied general mathematical models to
  communication

5
A Communication Study Hierarchy
6
Communication Channel
7
Error Detecting

• Communication process is stochastic in nature
• Involves an amount of chance and probability

• The channel experiences interference
• Referred to as noise

• Receiver cannot detect error if information is
  sent without redundant data

8
Error Detection Parity

• The even or odd quality of the number of 1's or
  0's in a binary code

• Parity as related to data communication
• Typically an additional bit that drives a words
  parity
• odd or even.

Add 1 if using odd parity
Consider 01010011
Add 0 if using even parity
9
Error Detection Using Parity

• Original Message 01010011

Encoder counts bits in message 01010011 (4)
adds 1 bit, set high
010100111 (odd parity)
Messaged corrupted on Channel 011100111
Decoder detects error 011100111 sees even parity
10
Error Correction

• Fixing corrupted data is possible by providing
  addition information

• Trade off speed for accuracy
• Extra time spent decoding

• Trade off channel capacity for accuracy
• Less information sent in same time

• Usually less costly than packet retransmission

11
Error Correction (continued)

• Transmit less information than Channel Capacity
  allows

• Combine redundancy with your information to fill
  the Channel Capacity.

12
Correction with Duplication

• Transmit data several times
• Take a majority vote

message 10110
encoded 10110 10110 10110
noise 10111 10100 11110
decoded
1
0
1
1
1
13
A Closer Look at Duplication

• There is a probability,p, that a bit will be
  changed

Look at a 1 replication, encoded as a set of
three bits (111)
Recognizable Patterns 111, 110, 101, 011
Irrecoverable Patterns 000, 001, 010, 100
14
A Closer Look at Duplication (continued)
What are the chances that an irrecoverable
pattern will occur ?
(p)(p)(p)
000 001 010 100
p3
(1-p)(1-p)(p)
(1-p)2p
(1-p)(p)(1-p)
(1-p)2p
(1)(1-p)(1-p)
(1-p)2p
(1-p)2p (1-p)2p (1-p)2p p3
3(1-p)2p p3
3p2-3p3 p3
3p2-2p3
15
A Closer Look at Duplication (continued)
What are the chances that an irrecoverable
pattern will occur ?
3p2-2p3
Suppose p .1
3(.1)2-2(.1)3 .028
An Improvement 10 versus 2.8
16
Duplication Analyzed

• Not Practical
• - 66 of bandwidth is spent on redundancy

• Not terribly clever, therefore probably not the
  best method.

17
Another method

• Encode two message bits with two check bits

- First check bit duplicates a message bit
- Second check bit forces even parity
These leads to four combinations, codewords
data check bits 0 0
0 0 0 1 1 1
1 0 0 1 1 1
1 0
18
Another method (continued)

• Note 24 (32) combinations are possible
• We only only have 4 valid combos

Here we apply the principle of maximum-likelihood
Hamming Distance Number of bits that needed to
transform a string S in to string T.
We want to minimize hamming distance of a
received string by looking for a closest valid
strings.
19
Another method (continued)

• Assume that the forth check bit doesnt change

If parity is not even If replica bit 3 is not
equal to data bit 2 invert data bit 2 else
invert data bit 1 End
20
Another method (continued)
What are the chances that an irrecoverable
pattern will occur ?
Irrecoverable 0001 0011 0110 0111
1001 1010 1011 1100
1101 1110 1111
List of acceptable patterns 0000
(expected/example pattern) 0010 (error is
ignored) 0100 (parity mismatch, corrected by
replica) 0101 (parity mismatch, corrected by
replica) 1000 (parity mismatch, corrected by
replica)
21
Another method (continued)
Let q represent the probably that a bit will be
received correctly (1-p).
Irrecoverable 0001 0011 0110 0111
1001 1010 1011 1100
1101 1110 1111
Probably of irrecoverable code qqqp qqpp qppq
qppp pqqp pqpq pqpp ppqq ppqp pppq
pppp
22
Another method (continued)
What are the chances that an irrecoverable
pattern will occur ?
Suppose p .1
irrecoverable probability .109
An Improvement 18 versus 10.9
23
The Bottom Line Preface

• An encoding method will have a discrete word set
  size, typically labeled M.

• Each word is encoded using n bits.

• The information rate, r, is a ratio of the
  message data divided by n.
• Written as Log2(M) / n

24
The Bottom Line Preface (continued)

• Let Z the average of all word probabilities to
  be incorrectly decoded.

• We want to minimize Z.

25
The Bottom Line Shannons Theorem

• Z ? 0 as n ? infinity

Under the following two conditions

• Information rate is within the range
• 0 lt r lt 1 p log2 p q log2q

• M and n satisfy the equation
• M 2nr

26
Summary

• Shannon pushed Information Theory into Light
• Information Theory
• Message Encoding
• Message Redundancy
• A Communication Channel is Stochastic in Nature
• Shannon gave us a theorem by which we can
  minimize the effects of interference.

27
References

• Dewdney, A.K. The New Turing Omnibus. New York
  Henry Holt and Company, 2001.
• Pless, Vera. Introduction to the Theory of
  Error-Correcting Codes 3rd ed. 1998
• Roman, Steven. Introduction to Coding and
  Information Theory. Springer. 1996
• Shannon, C.E. A Mathematical Theory of
  Communication. The Bell System Technical
  Journal, Vol. 27 pp.379-423, July October, 1948.
• Van Lint, J.H. Introduction to Coding Theory 3rd
  ed. Springer. 1998.