Video Coding For Compression . . . and Beyond

1
Video Coding For Compression. . . and Beyond

Compression

• Bernd Girod
• Information Systems LaboratoryDepartment of
  Electrical Engineering
• Stanford University

2
Bit Consumption of US Households
Bit equivalent, assuming state-of-the-art
compression, year 2000
Total for 70M households 230 Exabyte/year
Television 94
Radio 1.7
Recorded Music 0.4
Newspaper 0.0003
Books 0.0002
Magazines 0.0002
Home video 3.3
Video games 0.6
Internet 0.0003
Source UC Berkeley How much Information
3
Desirable Compression Ratios
SDTV broadcasting 2 Mbps
ITU-R 601 166 Mbps
100 1
DSL 200 kbps
1,000 1
Dial-up modem, wireless link 20 kbps
10,000 1
4
Outline

• Video compression state-of-the-art
• Beyond compression
• Rate-scalable video
• Wavelet video coding
• Error-resilient video transmission
• Unequal error protection
• Optimal scheduling for packet networks
• Distributed video coding

5
Outline

• Video compression state-of-the-art
• Beyond compression
• Rate-scalable video
• Wavelet video coding
• Error-resilient video transmission
• Unequal error protection
• Optimal scheduling for packet networks
• Distributed video coding

6
It has been customary in the past to transmit
successive complete images of the transmitted
picture. ... In accordance with this
invention, this difficulty is avoided by
transmitting only the difference between
successive images of the object.
7
Motion-Compensated Hybrid Coding
Video in
Standards H.261, MPEG-1, MPEG-2, H.263, MPEG-4,
H.264/AVC
8
Motion-Compensated Hybrid Coding
Video in
¼-pixel accuracy
Standards H.261, MPEG-1, MPEG-2, H.263, MPEG-4,
H.264/AVC
9
Motion-Compensated Hybrid Coding
Video in
Standards H.261, MPEG-1, MPEG-2, H.263, MPEG-4,
H.264/AVC
10
Motion-Compensated Hybrid Coding
Video in
Standards H.261, MPEG-1, MPEG-2, H.263, MPEG-4,
H.264/AVC
11
Motion-Compensated Hybrid Coding
Video in
Generalized B-Frames
Standards H.261, MPEG-1, MPEG-2, H.263, MPEG-4,
H.264/AVC
12
Rate-Distortion Optimized Coder Control

• Minimize Lagrangian cost function
• Strategy minimize Ji for each block i
  separately, using a common Lagrange multiplier l
  

13
Multiple Reference Frames in H.264/AVC
Mobile Calendar (CIF, 30 fps)
38
37
36
35
34
33
32
PSNR Y dB
31
30
29
PBB... with generalized B pictures
28
PBB... with classic B pictures
PPP... with 5 previous references
27
PPP... with 1 previous reference
26
0
1
2
3
4
R Mbit/s
14
Multiple Reference Frames in H.264/AVC
Mobile Calendar (CIF, 30 fps)
38
37
36
35
34
33
32
PSNR Y dB
31
30
29
PBB... with generalized B pictures
28
PBB... with classic B pictures
PPP... with 5 previous references
27
PPP... with 1 previous reference
26
0
1
2
3
4
R Mbit/s
15
Multiple Reference Frames in H.264/AVC
Mobile Calendar (CIF, 30 fps)
38
37
36
35
34
33
32
PSNR Y dB
31
30
29
PBB... with generalized B pictures
28
PBB... with classic B pictures
PPP... with 5 previous references
27
PPP... with 1 previous reference
26
0
1
2
3
4
R Mbit/s
16
Outline

• Video compression state-of-the-art
• Beyond compression
• Rate-scalable video
• Wavelet video coding
• Error-resilient video transmission
• Unequal error protection
• Optimal scheduling for packet networks
• Distributed video coding

17
Surprising Success of ITU-T Rec. H.263
What H.263 was developed for . . .
. . . and what is was used for.
??
Analog videophone
18
Internet Video Streaming
Streaming client
DSL
Media Server
Internet
dial-up modem
wireless

• How to accommodate heterogeneous bit-rates?
• How to react to network congestion?
• How to mitigate late or lost packets?

19
Fine Granular Scalability (FGS)
2dB gap

• H.264 with/without FGS option
• Foreman sequence (5fps)

20
Wavelet Video Coder
Originalvideoframes
LH
LH
LLH
LLL
TemporalWavelet Transform
Spatial WaveletTransform
Embedded Quantization Entropy Coding
Taubman Zakhor, 1994 Ohm, 1994 Choi
Woods, 1999 Hsiang Woods, VCIP 99 . . .
and others
21
Lifting
22
MC Wavelet Coding vs. H.264/AVC
38
36
Non-scalable H.264/AVC
34
32
30
Luminance PSNR (dB)
28
26
Scalable MC 5/3 Wavelet

• Sequence Mobile CIF
• H.264/AVC
• high complexity RD control
• CABAC
• PBBPBBP . . .
• 5 prev/3 future reference frames
• data courtesy of M. Flierl

24
22
20
2.0
1.8
1.6
1.4
1.2
1.0
0.6
0.4
0.2
0.8
Taubman Secker, VCIP 2003courtesy D. Taubman
bit-rate (Mbps)
23
Wavelet Synthesis with Lossy Motion Vector
Videoin
Videoout
Inverse Wavelet Transform
MC Wavelet Transform
Embedded Encoding
Decoder
Minimize JDlR
Embedded Encoding
Decoder
Motion Estimator
Minimize JDlR
Taubman Secker, ICIP03
24
R-D Performance with Lossy Motion Vector
Taubman Secker, VCIP 2003courtesy D. Taubman
25
Outline

• Video compression state-of-the-art
• Beyond compression
• Rate-scalable video
• Wavelet video coding
• Error-resilient video transmission
• Unequal error protection
• Optimal scheduling for packet networks
• Distributed video coding

26
Priority Encoding Transmission (PET)
information symbols

block of packets

• Albanese, Blömer, Edmonds, Luby, Sudan,
  1996 Davis Danskin, 1996
• Horn, Stuhlmuller, Link, Girod, 1999 Puri,
  Ramchandran, 1999
• Mohr, Riskin, Ladner, 2000 Stankovic,
  Hamzaoui, Xiong, 2002
• Chou, Wang, Padmanabhan, 2003 . . . and many
  more . . .

27
Packet Delay Jitter and Loss
pdf
e
(1-e)
loss
k
?
delay
28
Smart Prefetching
Idea Send more important packets earlier to
allow for more retransmissions
Server
Client
Internet
Podolsky, McCanne, Vetterli 2000 Miao, Ortega
2000 Chou, Miao 2001
29
Rate-Distortion Preamble
I
I
P
P
I
B
B
B
P
P
P
I
B
B
B
P

• Each media packet n is labeled by
• Bn size in bits of data unit n
• Ddn distortion reduction if n is decoded
• tn decoding deadline for n

30
Rate-Distortion Preamble
I
I
P
B
P
P
I
B
B
P
P
I
B
B
B
P

• Each media packet n is labeled by
• Bn size in bits of data unit n
• Ddn distortion reduction if n is decoded
• tn decoding deadline for n

31
Markov Decision Tree for One Packet
... N transmission opportunities before
deadline
32
R-D Optimized Streaming Performance
PSNR dB

• Foreman
• 120 frames
• 10 fps, I-P-P-
• H.263 2 Layer SNR scalable
• 20 frame GOP
• Copy Concealment
• 20 loss forward and back
• G-distributed delay
• ? 10 ms
• µ 50 ms
• s 23 ms
• Pre-roll 400ms

Bit-Rate kbps
33
Naive Coding Questions

1. To achieve graceful degradation in case of
   channel error for a digitally encoded signal, is
   an embedded signal representation (aka layers,
   aka data partitioning) always needed?
2. Can one, in general, send refinement information
   for an analog (i.e. uncoded) signal transmission
   over a noisy channel?

34
Digitally Enhanced Analog Transmission
Analog Channel (uncoded)

• Forward error protection of the signal waveform
• Information-theoretic bounds Shamai, Verdu,
  Zamir,1998
• Systematic lossy source-channel coding

35
Forward Error Protection of Compressed Video
Analog channel (uncoded)
Any OldVideo Encoder
Video Decoder with Error Concealment
S
S
Error-Prone channel
Aaron, Rane, Girod, ICIP 2003

• Graceful degradation without a layered signal
  representation

36
Wyner-Ziv MPEG Codec
Rane, Aaron, Girod, VCIP 2004
37
Graceful Degradation with Forward Error Protection
38
Visual Comparison of Degradation at Same PSNR
Foreman 50 CIF frames _at_ symbol error rate 4 x
10-4
With FEC 1 Mbps 120 kbps (38.32 db)
With FEP 1 Mbps 120 kbps (38.78 db)
39
Superior Robustness of FEP
Foreman 50 CIF frames _at_ symbol error rate 10-3
With FEC 1 Mbps 120 kbps (33.03 db)
With FEP 1 Mbps 120 kbps (38.40 db)
40
Lossy Compression with Side Information
Source
Encoder
Decoder
Wyner, Ziv, 1976 For mse distortion and
Gaussian statistics, rate-distortion functions
of the two systems are the same.
Source
Encoder
Decoder
41
Ultra-Low-Complexity Video Coding
42
R-D Performance Ultra-Low-Complexity Video Coder

• Sequence Foreman
• WZ frames - even frames
• Key frames - odd frames
• Side information - motion compensated
  interpolation of key frames

43
Ultra-Low-Complexity Video Coder
Wyner-Ziv Codec 274 kbps, 39.0 dB
H263 Intraframe Coding 330 kbps, 32.9 dB
44
Ultra-Low-Complexity Video Coder
Wyner-Ziv Codec 274 kbps, 39.0 dB
H263 I-B-I-B 276 kbps, 41.8 dB
45
Stanford Camera Array
Courtesy Marc Levoy, Stanford Computer Graphics
Lab
46
Stanford Camera Array
Courtesy Marc Levoy, Stanford Computer Graphics
Lab
47
Light Field Compression
Wyner-Ziv, Pixel-Domain
JPEG-2000
Rate 0.11 bpp PSNR 39.9 dB
Rate 0.11 bpp PSNR 37.4 dB
48
Conclusions

• Video compression is very important. . . but
  there is more to video coding than compression
• Rate-scalable video representations mc lifting
  break-through
• Robust video transmission
• Virtual priority mechanisms by packet scheduling
• RD gains easily larger than from super-clever
  compression
• Distributed video coding radically different
  approach
• Graceful degradation w/o layers
• Ultra-low-complexity coders
• Ubiquitous JDlR

49

• Acknowledgments
• Anne M. Aaron
• Jacob Chakareski
• Philip A. Chou
• JDlR
• Markus Flierl
• Sang-eun Han
• Mark Kalman
• Marc Levoy
• Yi Liang
• Shantanu Rane
• David Rebollo-Monedero
• Andrew Secker
• David Taubman
• Thomas Wiegand
• Xiaoqing Zhu
• Rui Zhang

50
Progress is a wonderful thing,if only it would
stop . . .

• Robert Musil

51
(No Transcript)