Scalable Joint SourceNetwork Coding of Video

1
Scalable Joint Source-Network Coding of Video

• Yufeng Shan
• Supervisors Prof. Shivkumar Kalyanaraman
• Prof. John W. Woods
• April 2007

2
Outline

• Problem statement and motivations
• Our proposed techniques
• Fine grain adaptive FEC (FGA-FEC)
• Generalized FGA-FEC for wireless networks
• Overly multi-hop FEC scheme
• Distributed FGA-FEC over multihop network
• Conclusion and future work

3
Problem statement and challenges

• Problem
• Simultaneously streaming video to diverse users,
  such as powerful PCs, laptops and handset
  devices, over heterogeneous networks.
• Challenges
• Different users may have different video frame
  rate /resolution /quality preferences.
• May be in different networks, such as high speed
  wired network, multi-hop ad hoc wireless network
  or cellular network.
• Neither the network nor the video application can
  provide quality assurances working independent of
  each other.

4
Conventional Approaches

• Windows media, Real player
• Send separate copies of the same bitstream to
  different users, bandwidth utilization is
  inefficient (unicast to each user).
• Layered multicast (by Steven McCanne)
• Send different video layer to different
  multicast groups, limited by number of video
  layers, no efficient error correction approach.
• Proxy-based streaming
• Caches video content at local proxy disk and
  transcodes it for different users, delay and
  computation inefficiency.

5
Our Techniques
Scalable joint source-network coding ? Scalable
video Scalable FEC in network adaptation
Video Performance

• FGA-FEC
• Generalized FGA-FEC
• OM-FEC
• Distributed FGA-FEC
• Header CRC/FEC
• Cross-layer FEC

Encoding
G-FGA-FEC
Video
server

FGA-FEC De/re-code
Adaptation

G-FGA-FEC
DSN

DSN

Router

DSN

Adaptation
DSN

Router

G-FGA-FEC
Router

DSN

DSN

Router

Video Performance
Router

Router

FGA-FEC De/re-code
6
Outline

• Problem statement and motivations
• Our proposed techniques
• Fine grain adaptive FEC (FGA-FEC)
• Generalized FGA-FEC for wireless networks
• Overly multi-hop FEC scheme
• Distributed FGA-FEC over multihop network
• Conclusion and future work

7
Background -- scalable video

• We treat three types of scalability
• SNR (aka quality, bitrate) scalability
• Spatial (resolution) scalability
• Temporal (frame-rate) scalability

8
SNR (quality) scalability
Embedded compressed file
Decoding more data corresponds to better quality
9
Resolution scalability
Embedded compressed file
Decoding more data corresponds to larger picture
size
10
Frame-rate scalability
Original video
One GOP - 16 frames
Half frame rate 8 frames
¼ frame rate 4 frames
1/8 frame rate 2 frames
Begin
Embedded compressed file
End
Decoding more data corresponds to displaying
frames at higher rate
11
3-D representation of bitstream
Digital items in view of three forms of
scalability (quality / resolution / frame-rate).
Adaptation of the bitstream is achieved by
choosing a subset of these items along users
preferred adaptation order For example SNR -
Temporal - Spatial
12
Next, we need scalable FEC

• To protect the scalable video bitstream, we have
  these basic requirements for channel coding
• High adaptivity If part of the video bitstream
  is actively dropped (scaled), FEC protecting that
  part of data should also be removed
• Accuracy near that of robust non-scalable video
• Efficiency Low computational cost

13
Fine Grain Adaptive FEC

• Goal
• To encode a video to facilitate efficient and
  precise adaptation of the encoded bitstream at
  intermediate overlay nodes for diverse users.
• Main idea
• Extend existing approaches (PET, MD-FEC) to
  work with scalable video and perform the
  adaptation in the network.

14
Multiple Description FEC (MD-FEC)
by Puri and Ramachantran
MSB
LSB
If any i out of N packets are received, the
decoder can decode up to Ri.
Our work extends this concept to provide
fine-grained scalability of FEC, applied to
scalable video
15
FGA-FEC overview
FGA-FEC uses overlay infrastructure to assist
video streaming to heterogeneous users
simultaneously by providing light weight support
at intermediate overlay nodes.
Adaptation
Adaptation
30/C/1M
30/C/1M
30/C/2M
30/C/3M
15/Q/384k
DSN
Encoding
15/C/384k
15/Q/1M
30/C/3M
Adaptation
A-G users and their requirements (frame
rate/resolution/bitrate)
DSN Data service node, which aggregates video
requirements to server and performs data
adaptation for users, without transcoding.
16
FGA-FEC Encoding

• Bitstream is divided into N sections from MSB
  to LSB
• Each section is further split into small blocks
• RS(N,i) codes are applied at block level to
  section Si, vertically
• Each block column is independently accessible
• one description one network packet

17
FGA-FEC optimal rate allocation
FEC assignment is optimized by minimizing the
expected distortion over the channel with packet
loss probability p, available bandwidth B.
Subject to
The result of the optimization is the video
source-rate breakpoints Ri
where qi is the probability of receiving i out of
N packets, Rtotal is the total bandwidth occupied
by both FEC and video data. Also ri is the
source rate of section Si. N is the number of
descriptions encoded for each GOP
18
FGA-FEC adaptation at intermediate nodes
Shorten drop
Two adaptation methods are used in FGA-FEC

• Direct truncation only shorten each packet in a
  GOP by removing unwanted blocks.
• FGA-FEC adaptation adapt the encoded GOP by a
  combined shortening and/or dropping packets.
• An algorithm is proposed to near optimally adapt
  the encoded GOP for available bandwidth and user
  preference
• No video or FEC transcoding, only a packet
  shorting/dropping

19
Simulation-- FGA-FEC adaptation vs. optimal
decoding/recoding

• Comparison of FGA-FEC, optimal decoding/recoding
  solution, and direct truncation at different
  available bandwidths
• In FGA-FEC, MC-EZBC encoded GOP 7, Foreman CIF
  is adapted from 1100 Kbps to satisfy different
  users
• FGA-FEC optimization is based on 15 loss rate
  at source, N64.

20
ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding
MD-FEC transcoding FEC decode/recode at each
intermediate node
FGA-FEC adaptation Encode at server, adaptation
in the intermediate nodes
Want to see the received video quality at User 4
and User 12, Link loss probability is 0.01, N64,
Foreman, CIF
21
ns-2 Simulation-- FGA-FEC vs. MD-FEC transcoding
Receiver 12 FGA-FEC is 0.4 dB lower
Receiver 4 FGA-FEC is 0.01dB lower
The quality loss is due to the adaptation
precision.
22
ns-2 Simulation-- FGA-FEC vs. Layered multicast
Layered multicast Base layer bit rate 0.6, and
enhancement layers 0.32, 0.42, 0.32 Mbps,
respectively
FGA-FEC adaptation Encode at server, adaptation
in the intermediate nodes
Want to see the received video quality at User 5
and User 7, Link loss probability is 0.01, N64,
Foreman, CIF
23
ns-2 Simulation-- FGA-FEC vs. Layered multicast
Receiver 7 Layered multicast can subscribe up to
three layers
Receiver 5 Layered multicast can subscribe up to
two layers
The quality loss of layered multicast is due to
coarser available layers.
24
Outline

• Problem statement and motivations
• Our proposed techniques
• Fine grain adaptive FEC (FGA-FEC)
• Generalized FGA-FEC for wireless networks
• Overly multi-hop FEC scheme
• Distributed FGA-FEC over multihop network
• Conclusion and future work

25
Generalized FGA-FEC for wireless
In addition to congestion related packet losses,
in wireless must deal with

• channel bit errors due to channel fading and
  noise
• large bandwidth fluctuations, and
• intermediate node computational capability
  constraint
• limited maximum transmission unit (MTU) size.

26
Generalized FGA-FEC for wireless

Bit level protection
FGA-FEC
Generalized FGA-FEC

• A kind of Product codes (extend Sherwood and
  Zegers codes)
• Column codes Reed Solomon for packet loss
• Row codes CRC BCH for bit errors

A BCH code is represented as BCH(n,k,t)
n2m-1, n-k
27
Optimal product code assignment

• Find a concatenated column RS code assignment cc
  and row BCH code assignment cr from a set of RS
  codes CRS and BCH codes CBCH BCH(n,k,t), such
  that the end-to-end video distortion over a lossy
  channel is minimized.

Subject to channel rate constraint
where Rs , RRS , RCRC and RBCH are rates
allocated to the video source, RS parity bits,
CRC and BCH check bits, respectively. Here B
denotes the maximum available channel bandwidth
28
Fast optimization method
n8191, m13
Given a BCH code BCH(n,k,t)
Probability of correctly decoding BCH code
Near optimal points, search starts here, it can
find the optimal solution within a few iterations
Exhaustive search results
GOP 7 of Foreman, CIF, N 64, pdrop0.05, BER
pb varies
29
Multi-cluster descriptions coding

• Single-cluster description coding one GOP ? N
  descriptions

• Multi-cluster descriptions coding one GOP ?
  Multiple of N descriptions due to
• Limited network packet size
• Easier adaptation, such as encode each temporal
  layer as one cluster as below.

One GOP of scalable bitstream
Low frame rate
High frame rate
Mid frame rate
Example of encoding one GOP into 3 clusters, each
cluster is coded with product codes
30
Simulation FGA-FEC vs. MD-FEC in wireless
Channel changes over time between node 2 and 3
Consider two adaptation orders SNR-Temporal,
SNR- Spatial
pdrop0.05 at node2
Resolution scaling
Frame rate scaling
31
Outline

• Problem statement and motivations
• Our proposed techniques
• Fine grain adaptive FEC (FGA-FEC)
• Generalized FGA-FEC for wireless networks
• Overly multi-hop FEC scheme
• Distributed FGA-FEC over multihop network
• Conclusion and future work

32
Motivations -- an example
Given an overlay path
(B6 , P6 , RTT6 )
(B1, P1, RTT1 )
Receiver
Sender
OM-FEC vs. Traditional FEC
The parameter of each hop
33
OM-FEC algorithm
OM-FEC partitions a given overlay path into
segments
By solving the following minimize(Nsegement)
Subject to
The i-th node is boundary node, if the two
inequalities hold
minBstart , , Bi ? (Bdata
BFEC(start?i)) minBstart, , Bi1BFEC(start?(i1))
FEC that should be added to the Segment is
BFEC(start -i),
34
Video Simulations
Topology loss rate on each hop (L1L10) random
within 0.4-1.2

• H.263 encoded Foreman CIF, at bitrate 1530
  Kbps
• Available bandwidth for each hop 1656 Kbps

Average number of segments partitioned is only 2
vs. 10 in hop-by-hop FEC
35
PlanetLab Experiments

• Video Foreman QCIF, H.263, 512 Kbps, 30 fps, I
  frame every second
• Packet-loss rate from Utah to CMU is set to 5,
  other links are set to 1.
• The available bandwidth from Utah to CMU is also
  upper bounded to
• 550 Kbps.
• Two paths 1) with nodes shown at map
• 2) Add Node 4 planet1.ecse.rpi.edu
  at last hop with 1 loss rate

36
OM-FEC experimental results
Video PSNR of OM-FEC scheme vs. e2e scheme (5
virtual links)
Video PSNR of OM-FEC scheme vs. e2e scheme
(4 virtual links)

• Results show
• OM-FEC outperforms e2e FEC scheme
• As more congested links are encountered, gain
  due to OM-FEC increases
• Large PSNR loss of the e2e scheme is due to the
  dependency of bitstream,
• one packet drop could cause later packets of the
  same GOP useless .

37
Outline

• Problem statement and motivations
• Our proposed techniques
• Fine grain adaptive FEC (FGA-FEC)
• Generalized FGA-FEC for wireless networks
• Overly multi-hop FEC scheme
• Distributed FGA-FEC over multihop network
• Conclusion and future work

38
Problem statement and our solution

• In FGA-FEC
• We assume no congestion between DSNs
• Remaining problem
• What if there is congestion between DSNs? For
  example, in a rate constrained multihop ad hoc
  wireless network.
• Our solution distributed FGA-FEC
• - FEC coding and optimization algorithm will be
  run at DSNs in a distributed way to serve the
  diverse users.

39
The optimization algorithm
Problem
Solution

• Minimize the e2e distortion for each user

Use Lagrange multiplier method
Subject to
where
Given one value of ?,
corresponds to the point on the D(R) curve with
slope equal to
Solution can be found by searching over ?
40
The Distributed FGA-FEC Algorithm

• To reduce the overall computation while maintain
  the best possible video quality, we propose two
  approaches
• A coordination method between optimization
  processes running at adjacent nodes to reduce the
  optimization computation.
• Full search, search with previous GOP, search
  with neighbor
• Applied the idea of OM-FEC to reduce the number
  of FGA-FEC decode/recode nodes, do FEC
  computation at selected DSNs

41
Simulation - reduce optimization computation

• Test number of ? iterations need to reach
  optimization stop points, Foreman, CIF.
• The proposed coordination between adjacent nodes
  can reduce the optimization computation
  requirement.

42
Simulation distributed FGA-FEC vs. Hop-by-hop
FEC decode/recode
Hop-by-hop FGA-FEC decode/recode FGA-FEC
decode/recode at each intermediate node
Distributed FGA-FEC Identify the congested links
and select appropriate nodes to do FGA-FEC
decode/recode.
Want to see the received video quality at User 5
and User 12 Link loss probability is 0.01, N64,
Foreman, CIF
43
Simulation distributed FGA-FEC vs. Hop-by-hop
FGA-FEC decode/recode
Receiver 12
Receiver 5
The two schemes deliver similar video quality,
but distributed FGA-FEC uses fewer FEC
computation nodes 3 vs. 6 at FEC decode/recode
44
Measured CPU time

• We measure the CPU time using FGA-FEC adaptation
  and FGA-FEC decode/recode in a Dell PC with 1.6
  GHZ CPU, 256 M memory, running Red Hat linux 8.2,
  N64.

Distributed FGA-FEC can greatly reduce the
computation burden, while can deliver near
optimal video quality
45
Conclusions

• Fine grain adaptive FEC (FGA-FEC)
• to encode and adapt a scalable video
• Generalized FGA-FEC for wireless networks
• to do encoding and adaptation over wireless
  networks
• Overly multi-hop FEC scheme
• to efficiently utilize one congested path
• Distributed FGA-FEC over multihop network
• to stream video over a congested heterogeneous
  networks
• Header error correction
• To improve the effective throughput of wireless
  networks
• Cross-layer FEC
• To joint optimize protocol with application layer
  FEC

46
Future work

• Evaluate FGA-FEC over DCT-based standard coders
  such as H.264/AVC and SVC
• Extend the distributed FGA-FEC to work with
  Spatial and Temporal scalability
• Extend FGA-FEC idea to multi-point video
  conferencing

47
Publication and submissions
1 Yufeng Shan, Ivan Bajic, John W. Woods and
Shivkumar Kalyanaranman Scalable video
streaming with fine grain adaptive
forward error correction submitted to IEEE
trans. CSVT, 2006 2 Su Yi, Yufeng Shan,
Shivkumar Kalyanaraman and Babak Azimi-Sadjadi,
"Video streaming over 802.11 Ad Hoc
wireless ntworks with header error protection",
submitted to Ad Hoc Networks, 2006 3 Yufeng
Shan, Ivan V. Bajic, Shivkumar Kalyanaraman and
John W. Woods, Overlay multi-hop FEC
scheme for video streaming, Signal
Processing Image Communications Vol. 20/8,
2005 4 Yufeng Shan, John. W. Woods and
Shivkumar Kalyanaraman Fine grain adaptive FEC
over wireless networks, submitted to ICIP
2007 5 Su Yi, Yufeng Shan, Shivkumar
Kalyanaraman and Babak Azimi-Sadjadi, "Header
error protection for multimedia data
transmission in wireless AdHoc networks", ICIP,
2006 6 Yufeng Shan, Ivan Bajic, Shivkumar
Kalyanaraman, and John W. Woods, "Joint
source-network error control coding for
ccalable overlay streaming," ICIP, 2005 7
Yufeng Shan, Su Yi, Shivkumar Kalyanaraman and
John.W. Woods, "Two-Stage FEC scheme for
scalable video transmission over wireless
networks" SPIE Communications/ITCom, Multimedia
Systems and Applications, Oct. 2005 8
Yufeng Shan, Ivan Bajic, Shivkumar Kalyanaraman,
and John W. Woods, "Overlay multi-hop FEC
scheme for video streaming over peer-to-peer
networks," ICIP, 2004 9 Yufeng Shan and
Shivkumar Kalyanaraman Hybrid video
downloading/streaming over peer-to-peer
networks, ICME, 2003 
48
Thanks