Lossy Image Compression

1
Lossy Image Compression

• From lossless to lossy
• Subjective and Objective Image Quality
• Quantization basics
• Uniform Quantization and 6dB/bit Rule
• Lossy predictive coding
• Differential Pulse Coded Modulation (DPCM)
• Lossy transform coding
• Image Transforms (Discrete Cosine Transform)
• Joint Photographic Expert Group (JPEG)

2
Why Lossy?

• In most applications related to consumer
  electronics, lossless compression is not
  necessary
• What we care is the subjective quality of the
  decoded image, not those intensity values
• With the relaxation, it is possible to achieve a
  higher compression ratio (CR)
• For photographic images, CR is usually below 2
  for lossless, but can reach over 10 for lossy

3
A Simple Experiment
Bit-plane representation
Aa0a12a222 a727
Least Significant Bit (LSB)
Most Significant Bit (MSB)
Example
A129 ? a0a1a2 a710000001
a0a1a2 a700110011 ? A4864128204
4
A Simple Experiment (Cont)

• How will the reduction of gray-level resolution
  affect the image quality?
• Test 1 make all pixels even numbers (i.e., knock
  down a0 to be zero)
• Test 2 make all pixels multiples of 4 (i.e.,
  knock down a0,a1 to be zeros)
• Test 3 make all pixels multiples of 4 (i.e.,
  knock down a0,a1,a2 to be zeros)

5
Experiment Results
Test 1
original
Test 3
Test 2
6
How to Measure Image Quality?

• Subjective
• Evaluated by human observers
• Do not require the original copy as a reference
• Reliable, accurate yet impractical
• Objective
• Easy to operate (automatic)
• Often requires the original copy as the reference
  (measures fidelity rather than quality)
• Works better if taking HVS model into account

7
Objective Quality Measures

• Mean Square Error (MSE)
• Peak Signal-to-Noise-Ratio (PSNR)

original
decoded
Question Can you think of a counter-example to
prove objective measure is not consistent with
subjective evaluation?
8
Answer
Shifted (MSE337.8)
Original cameraman image
By shifting the last row of the image to become
the first row, we affect little on subjective
quality but the measured MSE is large
9
Lossy Image Compression

• From lossless to lossy
• Subjective and Objective Image Quality
• Quantization basics
• Uniform Quantization and 6dB/bit Rule
• Lossy predictive coding
• Differential Pulse Coded Modulation (DPCM)
• Lossy transform coding
• Image Transforms (Discrete Cosine Transform)
• Joint Photographic Expert Group (JPEG)

10
What is Quantization?

• In Physics
• To limit the possible values of a magnitude or
  quantity to a discrete set of values by quantum
  mechanical rules
• In image compression
• To limit the possible values of a pixel value or
  a transform coefficient to a discrete set of
  values by information theoretic rules

11
Examples

• Unlike entropy, we encounter it everyday (so it
  is not a monster)
• Continuous to discrete
• a quarter of milk, two gallons of gas, normal
  temperature is 98.6F, my height is 5 foot 9
  inches
• Discrete to discrete
• Round your tax return to integers
• The mileage of my car is about 55K.

12
A Game Played with Bits

• Precision is finite the more precise, the more
  bits you need (to resolve the uncertainty)
• Keep a card in secret and ask your partner to
  guess. He/she can only ask Yes/No questions is
  it bigger than 7? Is it less than 4? ...
• However, not every bit has the same impact
• How much did you pay for your car? (two thousands
  vs. 2016.78)

13
Scalar vs. Vector Quantization

• We only consider the scalar quantization (SQ) in
  this course
• Even for a sequence of values, we will process
  (quantize) each sample independently
• Vector quantization (VQ) is the extension of SQ
  into high-dimensional space
• Widely used in speech compression, but not for
  image compression

14
Definition of (Scalar) Quantization
original value
quantization index
quantized value
f
f -1
s?S
x
Quantizer
Q
x
f finds the closest (in terms of Euclidean
distance) approximation of x from a codebook C
(a collection of codewords) and assign its index
to s f-1 operates like a table look-up to
return the corresponding codeword
15
Numerical Example (I)
C-1,0,1
S1,2,3
x?R
1
2
3
1
-1
0
red codewords green index in C
Three codewords
1
-0.5
x
0.5
-1
16
Numerical Example (II)
C8,24,40,56,72,,248
S1,2,3,,16
x?0,1,,255
16
1
2
3
...
248
40
8
...
24
red codewords green index in C
sixteen codewords
(note that the scales are different)
40
24
8
x
floor operation
16
32
17
Numerical Example (III)
232 216 232 232 216 200 216
40 56 56 72 40 136 104
88 24
225 222 235 228 220 206 209
44 49 56 64 42 128 106
94 27
Q
x
Notes
? For scalar quantization, each sample is
quantized independently
? Quantization is irreversible
18
Lossy Data Compression
discrete source X
S
entropy coding (e.g., Huffman)
binary bit stream
f
super-channel
S
entropy decoding

f-1
X
19
Quantization Noise (Distortion)
Q
x
Quantization noise
For a continuous random variable, distortion is
defined by
probability distribution function
For a discrete random variable, distortion is
defined by
probabilities
20
Revisit the Game with Bits

• Recall the goal of communication is to transmit
  bits for the purpose of resolving uncertainty
• Not all bits are equal in the sense of resolving
  uncertainty (e.g., MSB vs. LSB)
• The game rule now changes to
• Given a bit budget, how to most effectively
  resolve the uncertainty (reduce the distortion)?
• Given a distortion target, how to use the fewest
  bits to achieve it?

21
Shannons Rate-Distortion Theory
Distortion
Q1
Q2
Rate (bps)
Question which quantization scheme wins, 1 or 2?
22
The Game of Quantizer Design

• Assume P(X) or Pi is given, design the mapping
  scheme such that it achieves the smallest
  distortion for a given rate or vice versa
• Usually this game is complicated however, we do
  know how to play this game for certain class of
  source (e.g., uniform distribution)

23
Uniform Quantization for Uniform Distribution
Uniform Quantization
A scalar quantization is called uniform
quantization (UQ) if all its codewords are
uniformly distributed (equally-distanced)
Example (quantization stepsize ?16)
?
?
?
?
248
40
8
...
24
Uniform Distribution
denoted by U-A,A
f(x)
1/2A
x
-A
A
24
Quantization Noise of UQ
?
?
?
?
A
-A
f(e)
1/ ?
e
- ?/2
?/2
Quantization noise of UQ on uniform distribution
is also uniformly distributed
Recall
Variance of U- ?/2, ?/2 is
25
6dB/bit Rule
Signal X U-A,A
Noise e U- ?/2, ?/2
Choose N2n (n-bit) codewords for X
(quantization stepsize)
N2n
26
How Much is 6dB?
6dB increases in SNR implies the reduction of MSE
by 75
To cut MSE by half, we approximately need 0.5bps
27
Lossy Image Compression

• From lossless to lossy
• Subjective and Objective Image Quality
• Quantization basics
• Uniform Quantization and 6dB/bit Rule
• Lossy predictive coding
• Differential Pulse Coded Modulation (DPCM)
• Lossy transform coding
• Image Transforms (Discrete Cosine Transform)
• Joint Photographic Expert Group (JPEG)

28
Predictive Coding Revisited
- Encoding
x1 x2 xN
y1 y2 yN
y1x1
ynxn-xn-1, n2,,N
- Decoding
y1 y2 yN
x1 x2 xN
x1y1
xnynxn-1, n2,,N
yn
yn
xn
xn
_

xn-1
xn-1
xn
D

D
xn-1
Decoder
DPCM Loop
Encoder
29
Open-loop DPCM


yn
yn
xn
xn
_
Q


xn-1
xn-1
xn
D

D
xn-1
Decoder
Encoder
Notes
? Prediction is based on the past unquantized
sample
? Quantization is located outside the DPCM loop
30
Catastrophic Error Propagation with Open-loop DPCM
0
-1
error caused by quantization

90
original samples
92
91
93
93
95
a
b
a-b
prediction residues
90
2
0
2
0
2
a
b
decoded samples

90
92
92
94
94
96
ab
31
Closed-loop DPCM



yn
yn
yn
xn
xn
_
Q



xn-1
xn
D

D

xn-1
Decoder
Encoder
Xn,yn unquantized samples and prediction residues


Xn,yn decoded samples and quantized prediction
residues
Notes
? Prediction is based on the past decoded sample
? Quantization is located inside the DPCM loop
32
Numerical Example

xn
90
92
91
93
93
95
a
yn
90
2
-2
3
0
2
a-b
b
Q
90
3
-3
3
0
3

yn
a
b
ab
xn


90
93
90
93
93
96
33
Closed-loop DPCM Analysis

yn
yn
xn
_
Q
A
B


xn-1
xn

D

xn-1
A
B
The distortion introduced to prediction residue
yn is identical to that introduced to the
original sample xn
34
DPCM Summary

• Open-loop DPCM
• Prediction is based on original unquantized
  samples
• Catastrophic error propagation problems (decoder
  does not have access to the original but
  quantized samples to do prediction)
• Closed-loop DPCM
• Both encoder and decoder employ decoded samples
  to do prediction
• Quantization noise affects the accuracy of
  prediction
• DPCM is only suitable for lossy image coding at
  high bit rate (small quantization noise)

35
Lossy Image Compression

• From lossless to lossy
• Subjective and Objective Image Quality
• Quantization basics
• Uniform Quantization and 6dB/bit Rule
• Lossy predictive coding
• Differential Pulse Coded Modulation (DPCM)
• Lossy transform coding
• Image Transforms (Discrete Cosine Transform)
• Joint Photographic Expert Group (JPEG)

36
An Example of 1D Transform with Two Variables
x2
y2
y1
(1,1)
(1.414,0)
x1
Transform matrix
37
Generalization into N Variables
forward transform
basis vectors (column vectors of transform matrix)
38
Decorrelating Property of Transform
x2
y1
y2
x1
x1 and x2 are highly correlated
y1 and y2 are less correlated
p(x1x2) ? p(x1)p(x2)
p(y1y2) ? p(y1)p(y2)
39
TransformChange of Coordinates

• Intuitively speaking, transform plays the role of
  facilitating the source modeling
• Due to the decorrelating property of transform,
  it is easier to model transform coefficients (Y)
  instead of pixel values (X)
• An appropriate choice of transform (transform
  matrix A) depends on the source statistics P(X)
• We will only consider the class of transforms
  corresponding to unitary matrices

40
Unitary Matrix and 1D Unitary Transform
Definition
conjugate
transpose
A matrix A is called unitary if A-1AT
When the transform matrix A is unitary, the
defined transform is called
unitary transform
Example
For a real matrix A, it is unitary if A-1AT
41
Inverse of Unitary Transform
For a unitary transform, its inverse is defined
by
Inverse Transform
basis vectors corresponding to inverse transform
42
Properties of Unitary Transform

• Energy compaction only few transform
  coefficients have large magnitude
• Such property is related to the decorrelating
  role of unitary transform
• Energy conservation unitary transform preserves
  the 2-norm of input vectors
• Such property essentially comes from the fact
  that rotating coordinates does not affect
  Euclidean distance

43
Energy Compaction Example
Hadamard matrix
significant
insignificant
44
Energy Conservation
A is unitary
Proof
45
Numerical Example
Check
46
Implication of Energy Conservation
Q
T
T-1
A is unitary
47
Summary of 1D Unitary Transform

• Unitary matrix A-1AT
• Unitary transform A unitary
• Properties of 1D unitary transform
• Energy compaction most of transform coefficients
  yi are small
• Energy conservation quantization can be directly
  performed to transform coefficients

48
From 1D to 2D
Do N 1D transforms in parallel
49
Definition of 2D Transform
2D forward transform
1D column transform
1D row transform
50
2D TransformTwo Sequential 1D Transforms
(left matrix multiplication first)
column transform
row transform
row transform
(right matrix multiplication first)
column transform
Conclusion
? 2D separable transform can be decomposed into
two sequential ? The ordering of 1D transforms
does not matter
51
From Basis Vectors to Basis Images

• 1D transform matrix A consists of basis vectors
  (column vectors)
• 2D transform corresponds to a collection of
  N-by-N basis images

basis image
52
Example of Basis Images
Hadamard matrix
j
i
Bij
53
2D Unitary Transform
Suppose A is a unitary matrix,
forward transform
inverse transform
Proof
Since A is a unitary matrix, we have
54
Energy Compaction Property of 2D Unitary
Transform
? Example
A coefficient is called significant if its
magnitude is above a pre-selected threshold th
insignificant coefficients (th64)
55
Energy Conservation Property of 2D Unitary
Transform
2-norm of a matrix X
A unitary
Example
You are asked to prove such property in your
homework
56
Implication of Energy Conservation
Q
T
T-1
Similar to 1D case, quantization noise in the
transform domain has the same energy as that in
the spatial domain
57
Important 2D Unitary Transforms

• Discrete Fourier Transform
• Widely used in non-coding applications
  (frequency-domain approaches)
• Discrete Cosine Transform
• Used in JPEG standard
• Hadamard Transform
• All entries are ?1
• N2 Haar Transform (simplest wavelet transform
  for multi-resolution analysis)

58
Discrete Cosine Transform
59
1D DCT
forward transform
inverse transform
Properties
? Real and orthogonal

• excellent energy compaction property

? DCT is NOT the real part of DFT
Fact
The real and imaginary parts of DFT are
generally not orthogonal matrices
? fast implementation available O(Nlog2N)
60
2D DCT
Its DCT coefficients Y (2451 significant
coefficients, th64)
Original cameraman image X
Notice the excellent energy compaction property
of DCT
61
Summary of 2D Unitary Transform

• Unitary matrix A-1AT
• Unitary transform A unitary
• Properties of 2D unitary transform
• Energy compaction most of transform coefficients
  yi are small
• Energy conservation quantization can be directly
  performed to transform coefficients

62
A Tour of JPEG Coding Algorithm
C
T
Q
Flow-chart diagram of DCT-based coding algorithm
specified by Joint Photographic Expert Group
(JPEG)
63
Transform Coding of Images

• Why not transform the whole image together?
• Require a large memory to store transform matrix
• It is not a good idea for compression due to
  spatially varying statistics within an image
• Idea of partitioning an image into blocks
• Each block is viewed as a smaller-image and
  processed independently
• It is not a magic, but a compromise

64
8-by-8 DCT Basis Images
65
Block Processing under MATLAB

• Type help blkproc to learn the usage of this
  function
• B BLKPROC(A,M N,FUN) processes the image A by
  applying the function FUN to each distinct M-by-N
  block of A, padding A with zeros if necessary.
• Example

I imread('cameraman.tif') fun
_at_dct2 J blkproc(I,8 8,fun)
66
Block-based DCT Example
I
J
note that white lines are artificially added to
the border of each 8-by-8 block to denote that
each block is processed independently
67
Boundary Padding
Example
12 13 14 15 16 17 18 19
12 13 14 15 16 17 18 19
12 13 14 15 16 17 18 19
12 13 14 15 16 17 18 19
padded regions
When the width/height of an image is not the
multiple of 8, the boundary is artificially
padded with repeated columns/rows to make them
multiple of 8
68
Work with a Toy Example
Any 8-by-8 block in an image is processed in a
similar fashion
69
Encoding Stage I Transform
Step 1 DC level shifting
128 (DC level)
_
70
Encoding Step 1 Transform (Cont)
Step 2 8-by-8 DCT
8?8 DCT
71
Encoding Stage II Quantization
Q-table
specifies quantization stepsize (see slide 28)
Notes

• Q-table can be specified by customer
• Q-table is scaled up/down by a chosen quality
  factor
• Quantization stepsize Qij is dependent on the
  coordinates
• (i,j) within the 8-by-8 block

• Quantization stepsize Qij increases from
  top-left to bottom-right

72
Encoding Stage II Quantization (Cont)
Example
sij
xij
f
73
Encoding Stage III Entropy Coding
zigzag scan
(20,5,-3,-1,-2,-3,1,1,-1,-1, 0,0,1,2,3,-2,1,1,0,0,
0,0,0, 0,1,1,0,1,EOB)
End Of the Block All following coefficients are
zero
Zigzag Scan
74
Run-length Coding
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0
,0,0,0,1,1,0,1,EOB)
DC coefficient
AC coefficient
- DC coefficient DPCM coding
encoded bit stream
- AC coefficient run-length coding (run, level)
(5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0,0,
0,0,1,1,0,1,EOB)
(0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-
1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0
,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB
Huffman coding
encoded bit stream
75
JPEG Decoding Stage I Entropy Decoding
encoded bit stream
Huffman decoding
(0,5),(0,-3),(0,-1),(0,-2),(0,-3),(0,1),(0,1),(0,-
1),(0,-1),(2,0),(0,1), (0,2),(0,3),(0,-2),(0,1),(0
,1),(6,0),(0,1),(0,1),(1,0),(0,1),EOB
encoded bit stream
DPCM decoding
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0
,0,0,0,1,1,0,1,EOB)
DC coefficient
AC coefficients
76
JPEG Decoding Stage II Inverse Quantization
(20,5,-3,-1,-2,-3,1,1,-1,-1,0,0,1,2,3,-2,1,1,0,0,0
,0,0,0,1,1,0,1,EOB)
zigzag
f-1
77
JPEG Decoding Stage III Inverse Transform
8?8 IDCT
128 (DC level)

78
Quantization Noise

X
X

Distortion calculation
MSEX-X2
Rate calculation
Ratelength of encoded bit stream/number of
pixels (bps)
79
JPEG Examples
90 (58k bytes)
50 (21k bytes)
10 (8k bytes)
best quality, lowest compression
100
0
worst quality, highest compression
80
JPEG Coding Algorithm Summary
81
Lossy Compression of Color Images
Color-space transformation
luminance
chrominance
forward transform
inverse transform
Luminance channel can be treated just like a
gray-scale image
82
Compression of Chrominance Channels
Q-Table
Down-sampling
2
Cr/Cb
Human eye is relatively insensitive to the
high-frequency content of the chrominance
channels.
83
Image Example
raw image data
JPEG compressed data
768512 241.18M bytes
26K bytes
Compression ratio44.6
84
A Glimpse into Wavelet Transform

• The idea of multi-resolution analysis (MRA)

Low-resolution (LR)
High-resolution (HR)
85
MRA by Haar Transform
LP
HP
Example
86
Reordering Coefficients into Subbands
approximation
horizontal details
Apply Haar transform to each 2-by-2 block
diagonal details
vertical details
ù
é
0
0
200
200
ú
ê
0
0
100
100
ú
ê
regrouping

Y
0
0
0
0
ú
ê
ú
ê
0
0
0
0
û
ë
87
Multi-level Wavelet Transform
Apply one-level Haar transform
Apply Haar transform to the LL band only
88
Image Example
89
Wavelet vs. DCT
discrete cosine transform based
wavelet transform based
JPEG (CR64)
JPEG2000 (CR64)
90
Lossy Image Compression Summary

• Quantization introduces irreversible information
  loss
• Lossy predictive coding open-loop DPCM vs.
  closed-loop DPCM
• Lossy transform coding energy compaction an
  preservation properties of unitary transforms
• Objective measure for image distortion
• MSE or PSNR are widely used for their convenience
• Does not necessarily match subjective quality