Advances in Network-adaptive Video Streaming

1
Advances in Network-adaptive Video Streaming

• Bernd Girod
• J. Chakareski, M. Kalman, Y. J. Liang, E. Setton,
  R. Zhang
• Information Systems LaboratoryDepartment of
  Electrical Engineering
• Stanford University

2
Streaming media a huge success

• Hundreds of thousands of streaming media servers
  deployed
• gt 1 million hours of streaming media content
  produced per month
• Hundreds or millions streaming media players
• RealPlayer
• Most popular Internet application second only to
  Internet Explorer Media Metrix
• More than 400 million unique registered users
• More than 200,000 new users per day
• Open source code

3
Internet Media Streaming
Streaming client
DSL
Media Server
Internet
56K modem
wireless

• Challenges
• compression
• rate scalability
• error resiliency
• low latency

• Best-effort packet network
• low bit-rate
• variable throughput
• variable loss
• variable delay

4
Outline

• What is network-adaptive video streaming?
• Better delivery of video packets by considering
  source coding, signal processing, and packet
  transport jointly
• Application-layer joint source-channel coding
  techniques for the Internet
• This talk review recent advances in
• Adaptive media playout
• Rate-distortion optimal packet scheduling
• Network-adaptive packet dependency management

5
Adaptive Media Playout
Idea reduce latency and packet loss
simultaneously by continuously adapting playout
deadline to network conditions
Fixed deadline
Flexible deadline
5 packet loss 2 sec average receiver buffer
Steinbach, Färber, Girod, ICIP 2001
6
Modification of Playout Speed

• Video adaptation of display rate
• Audio and speech Stretching based on time-domain
  interpolation algorithm WSOLA Verhelst et al.,
  1993, Liang 2001

Template
Pitch- period
0
2
1
3
4
Original packet
7
Audio Speed Adjustment

• Waveform Similarity Overlap Add (WSOLA) method
  allows speed adjustment without changing pitch
• Speech demo
• Music demo

slower 30
original
faster 30
original
slower 30
faster 30
8
Reduced Pre-roll Time for Stored Streams
Probability of buffer underflow lt 1
Kalman, Steinbach, Girod, ISCAS 2002
lG1.092, lB0.42, TG20 sec, TB2 sec, TRTT220
ms
9
Rate Scalability by Playout Speed Adjustment
Server
Channel (mean throughput)
(29.2 dB, 53.3 kbps)
25 kbps
20 kbps
55 kbps
50 kbps
85 kbps
95 kbps
100 kbps
(33.1 dB, 93.60.984.2 kbps)
10
First Things First Smart Prefetching
Idea Send more important packets earlier to
allow for more retransmissions
Server
Internet
Client
11
Streaming as a Packet Scheduling Problem
server
pre-encoded media units
transmission opportunities
time

• Which media units should be selected for
  transmission, and when?
• Requirements
• Meet rate constraint
• Meet latency constraint
• Maximize reconstruction quality
• Rate-distortion framework proposed, e.g., in
  Podolsky, McCanne, Vetterli 2000 Miao, Ortega
  2000 Chou, Miao 2001

12
Markov Decision Tree for One Packet
... N transmission opportunities before
deadline
13
Packet Delay Jitter and Loss
pdf
e
(1-e)
loss
k
?
delay
14
Source Description
I
I
P
P
I
B
B
B
P
P
P
I
B
B
B
P



A
A

• Each media packet n is labeled by
• Bn size in bits of data unit n
• Ddn distortion reduction if n is decoded
• tn decoding deadline for n

15
Source Description
I
I
P
B
P
P
I
B
B
P
P
I
B
B
B
P



A
A

• Each media packet n is labeled by
• Bn size in bits of data unit n
• Ddn distortion reduction if n is decoded
• tn decoding deadline for n

16
R-D Optimized Streaming Performance
PSNR dB

• Foreman
• 120 frames
• 10 fps, I-P-P-
• H.263 2 Layer SNR scalable
• 20 frame GOP
• Copy Concealment
• 20 loss forward and back
• G-distributed delay
• ? 10 ms
• µ 50 ms
• s 23 ms
• Pre-roll 400ms

Bit-Rate kbps
17
R-D Optimized Streaming with a Proxy Server
Last hop
Chakareski, Chou, Girod, Asilomar 2002, MMSP
2002
18
RaDiO Edge Experiment

• Video Foreman, QCIF, 130 frames
• Compression H.264
• 3-layer temporal scalability 72144 kbps Liang,
  2002
• Backbone
• Packet loss rate 10
• Delay shifted G-distribution
• Last hop
• Packet loss rate 1
• Delay shifted G-distribution

19
Streaming with Diversity
Server Diversity
Packet Path Diversity
Channel 1
Channel 2
Client
Channel N
20
R-D Optimized Streaming over 2 Channels

• Video Foreman, QCIF, 130 frames
• Compression H.263
• 2-layer SNR scalability 32/64 kbps
• 2 identical, independent 2-state Markov channels
• Good/bad packet loss rates 3/15
• G-distributed delays short/long

Chakareski, Girod, DCC 2003
21
R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)
PSNR, in dB
Latency 100 ms RTT 100 ms
Bit-Rate, in kbps
22
R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)Latency 100ms
R-D Optimized Streaming with Accelerated
Retroactive Decoding (ARD)
Multiple Deadlines Rate 68.8 kbps Mean PSNR
27.0 dB
Single Deadline Rate 89.0 kbps Mean PSNR 23.9 dB
23
Network-adaptive Packet Dependency Management

• Very low latency no retransmissions
• Highly robust compressed representation by
  network-adaptive management of packet
  dependencies
• Utilize ACK/NACK in source coder
• H.263 RPS, MPEG-4 NEWPRED, H.264 multiframe
  prediction

Transmission error
NACK
Time
24
Error Resilience vs. Coding Efficiency
P5
I
230 frames of Foreman coded using H.26L TML8.5.
Average PSNR33.4dB
25
Rate-Distortion Optimal Reference Picture
Selection
Wiegand, Färber, Girod, 2000 Liang, Girod,
2002
26
RD Performance of Optimal Reference Picture
Selection

• LTM buffer V5 frames
• Feedback round-trip time dfb7 frames
• Packet loss rate p10
• Comparison with P-I scheme, where each NACK
  triggers insertion of I-frame

Liang, Girod, 2002
27
ORPS Performance over Time Axis
28
MaD Sequence at 10 packet lossNo Retransmissions
Optimal Reference Picture Selection Rate 320
kbps Mean PSNR 39.2 dB
Adaptive I-Frame Insertion Rate 320 kbps Mean
PSNR 38.2 dB
29
Conclusions

• Network-adaptive video streaming jointly
  optimize compression, error control, packet
  transport, and decoding
• Adaptive media playout real-time more
  flexible than we thought
• RaDiO streaming can provide virtual priority
  mechanisms
• Proxy servers with RaDiO can improve streaming
  performance
• Path Diversity can improve streaming performance
• RaDiO streaming with multiple deadlines for
  accelerated retroactive decodingof late
  retransmissions (catch-up decoding)
• Packet dependency management allows robust
  representations for streaming w/o retransmissions