Encoder-Based Rate Smoothing and Quality

1
SPIE VCIP, Jan 17, 2002
Encoder-Based Rate Smoothing and Quality Control
for Low Delay Video Coding
Zhihai He and Chang Wen Chen
Interactive Media Sarnoff Corporation
2
Introduction - Problem

• Video coding has two basic modes
• constant bit rate (CBR) coding
• variable bit rate (VBR) coding

CBR video coding

• Low-delay video coding.
• Video quality fluctuation due to the
  time-varying
• scene activities in the video sequence.

3
Introduction - Problem
MPEG-4 Table-tennis QCIF 96 kbps 15 fps
PSNR

• Very large PSNR
• fluctuation

Frame
4
Introduction - Problem
VBR Coding

• Smooth video presentation
• quality
• Large rate fluctuation

Bits
Controlled PSNR 35 dB
From the standpoint of transmission efficiency,
network traffic management and resource
allocation, bursty VBR video stream is much
harder to handle than the CBR video.
5
Introduction - Problem
Rate smoothing
Video Encoder

• Bits stream buffers
• on the transmission path
• encoder buffer / router buffer /
• decoder buffer
• Function as low pass filters Examples

Channel
Transmission buffer based rate smoothing

• Introduce large transmission delay
• Low-pass filtering mechanism is not very
  effective,
• especially for medium/long-term rate
  fluctuation.

6
Introduction - Problem
3 frames
7 frames
15 frames
30 frames
7
Encoder-Based Rate Smoothing
In this work

• Encoder-based rate smoothing
• The encoder smoothes out the rate shape by
  itself while
• maintaining a controlled video quality.
• No additional transmission delay is introduced.

How

• The encoder estimates the rate-distortion (R-D)
  function
• of the current frame.
• Makes a good trade-off between rate and quality
  (distortion)
• based on the estimated R-D functions.

8
Estimation of R-D Functions

• R-D Estimation requirements
• Fast (small computation delay) and accurate
• a prior estimation before quantization and
  coding

In our previous work Zhihai He and Sanjit K.
Mitra, August CSVT 2001, we have developed a
fast and accurate R-D estimation algorithm,
which can estimate the R-D functions before
quantization and coding.
9
Estimation of R-D Functions Brief Review
H.263 / MPEG-4
Coding system
Coding performance depends on
Characteristics of the input source data
Capability of the coding algorithm (MPEG-2,
H263, MPEG-4, etc) to explore these
characteristics
?
?
10
Estimation of R-D Functions
Definition
Distribution of DCT coefficients
q quantization step size ?
percentage of zeros among the quantized
transform coefficients.
q
Rate R(q) R(?)
?
Distortion D(q) D(?)
Observations
? monotonically increases with q. There is a
one-to-one mapping between them
q-domain
? -domain
11
Ideas 1 Characteristic Rate Curves
Describe the source data

• Classical R-D analysis variance

insufficient
Proposed Characteristic rate curves

• Introducing two rate curves in the ? -domain

Qz(?) and Qnz(?)
to characterize the input source data
12
Ideas 2 Rate Curve Decomposition
Model the coding algorithm
Fourier analysis
f(t) ? Ancos(nt) Bnsin(nt)
R(?) actual rate curve
Rate curve decomposition
R(?) A(?) Qz(?) B(? )Qnz(?) C(?)
Qz(?) and Qnz(?) modeling the source
data A(?), B(? ) and C(?) modeling the coding
algorithm
13
Rate Curve Decomposition
A Set of Small and Simple Problems
A Large Problem
R(?) A(?) Qz(?) B(? )Qnz(?) C(?)
Zeros Qz(?)
Zeros Qz(?)
Input Picture
Input Picture
Non-zeros Qnz(?)
R-D functions
Zeros Qz(?)
R-D functions
Coding Algorithm
Input Picture
14
Estimation of R-D Functions
How does it work?

• We first show that the two characteristic rate
  curves
• Qz(?) and Qnz(?) have unique properties.
• Based on these properties, they can be estimated
  with
• a fast algorithm.

15
Characteristic Rate Curves
Sample images
Wide range of image characteristics
16
Characteristic Rate Curves
Plots of
Qnz(?) Top Qz(?) Bottom

• UTQ
• Wavelet

17
Characteristic Rate Curves
Motion compensated pictures from Foreman MPEG-4

• Very small picture-
• dependent variation
• Straight line

18
Characteristic Rate Curves
The same curves in the q-domain
Image-dependent variation (most difficult
part.) Highly nonlinear
19
Estimation of R-D Functions
R(?) A(?) Qz(?) B(? )Qnz(?) C(?)

• For a given encoder, its decomposition
  coefficients
• A(?), B(? ) and C(?) is fixed.
• Using summation R(?) A(?) Qz(?) B(?
  )Qnz(?) C(?)
• we obtain the rate curve R(?), which is then
  mapped into
• q-domain R(q).

20
Estimation of R-D Functions
Six test images
Coding algorithms SPIHT Stack-run
Estimate their R-D curves before coding.
21
Estimation of R-D Functions
For SIPHT
Relative estimate error less than 4.
22
Estimation of R-D Functions
For JPEG The first 4 images
Relative estimate error less than 5. Digital
camera applications
23
Estimation of R-D Functions
Frames 30, 60,90 and 120
MPEG-4 Carphone 15 fps Accurate
frame-level rate control
PSNR
Bits
24
Encoder-Based Rate Smoothing
Trade-off between rate and quality for
low-delay video coding.
Objective

• The output bit rate shape of the video encoder
  is smoothed,
• not bursty. (Just like the transmission buffer
  rate smoothing)
• The output picture quality should be well
  controlled
• or maintained if possible.

• Introduce no addition transmission delay.

25
Encoder-Based Rate Smoothing
Observations

• Good video presentation quality in practice
• Dramatic change of quality from picture to
  picture is
• not acceptable.
• However,
• Smooth change (temporal variation) of picture
  quality
• within a controlled range is acceptable. Demo
  later

26
Encoder-Based Rate Smoothing
Idea
How to smooth out the rate shape while
maintain good video quality?
Find the shortest path
Rate as smooth as possible. Quality
very smooth change with /- 1dB
35 dB
Rate
33 dB
Frame time
27
Encoder-Based Rate Smoothing
Algorithm
Frame n-1 Just coded Bit Rate Rn-1
If R-- lt Rn-1 lt R if Rn-1 gt R if Rn-1 lt
R-
Rn-1 R R
Current frame Target 34 dB 341 dB R 34-1 dB
R--
Rn
28
Encoder-Based Rate Smoothing
Properties

• Rate as smooth as possible.
• Quality very smooth change with /- 1dB of the
  
• target quality.
• Very effective in smoothing, both short term and
  long term.
• No addition delay

35 33
29
Experimental Results
MPEG-4 Foreman Target 34 dB Controlled variati
on 1dB
30
Experimental Results
PSNR
31
Experimental Results
Demo1 Foreman CIF 15 fps Target 34 dB
Demo2 NBA CIF 15 fps Target 33 dB Demo3
TexasWild CIF 15 fps Target 32 dB
32
Conclusion

• An encoder-based rate smoothing algorithm
• has been developed, smooth out the rate shape
• while maintaining well-controlled visual
  quality
• For very low delay coding application.