Source Coding

1
Source Coding
Binit Mohanty Ketan Rajawat
2
Information

• What is information?
• Can it be measured?
• Fundamental Unit of Info Bits
• Represents an ON/OFF state
• Complex Info Array of Bits

3
Quantizing Information

• Example I
• Class of 32 people
• Need log2(32) 5 bits
• Not enough!
• Need Info definition and a Dictionary
• What about a class of 25 people? 5 people?

4
Information and Uncertainty

• Information Extra knowledge you gain
• Uncertainty Knowledge that you do not have
• Before the knowledge of the outcome you have
  uncertainty
• After it you have Information!
• They are the same thing!

5
Quantizing Information

• Example II
• Class of 32 people 6 females, 26 males
• Choose one at random
• Give the knowledge of gender
• By what amount is uncertainty reduced?

6
Quantizing Info

• Uncertainty log(1/Probability)
• Average this
• Entropy!

7
Entropy for a single event
8
Source Coding

• To compress information
• Lossless Perfect reconstruction possible
• Eg GIF (LZ), WINRAR
• Lossy Tolerable degradation allowed
• Eg JPEG, MP3
• Exploit recurrence of sequences

9
Huffman Code

• David A. Huffman of MIT
• Short codes for high probability
• Longer codes for low probability
• An Example
• P(A)0.5, P(B)0.25, P(C)0.25
• A0, B10,C11

10
LZ Code

• By Abraham Lempel Jacob Ziv
• Two versions LZ-77 LZ-78
• LZ 78
• Parse
• Insert a comma after every new sequence
• Transmit
• New bit and previous occurrence index

11
Some Other Codes

• LZ 77
• LZW - Text based coding
• Elias Coding
• Lynch-Davisson Codes