Robust Video Transmission with H'264AVC

1
Robust Video Transmissionwith H.264/AVC

• Pierpaolo Baccichet
• IEIIT - CNR

2
Presentation Outline

• Overview of H.264/AVC over the network
• Error concealment improvements
• Whole frame loss error concealment
• Multi-Hypothesis error concealment
• Redundant Slices FMO
• Evaluation
• A preliminary solution

3
Application Scenario

• Transmission of low bitrate sequences
• No assumptions for the transmission channel
• No channel feedback
• No PLR a priori knowledge
• Strict H.264/AVC Baseline Profile Compliance
• Suitability for real time implementations (now!)

4
H.264/AVC over the network
Network Abstraction Layer
5
H.264/AVC over the network
Network Abstraction Layer

• Unchanging parameters are sent offline by means
  of the Parameter Sets (both for Sequence and
  for Picture)
• VCL NALUs correspond to coded slices or slice
  partitions
• Each NAL unit has one byte header
• forbidden_zero_bit to signal errors
• nal_ref_idc to signal the importance
• nal_unit_type to signal the payload type
• Annex B specifies Byte Stream format
• RTP payload format for H.264 video
• RFC 3984
• Specifies aggregation, segmentation,
  interleaving, etc

6
H.264/AVC over the network
Access Units

• An access unit specifies the decoding order of
  NAL units delivered to a compliant decoder
• The standard specifies some constraints about the
  order of NAL units belonging to the same primary
  coded picture
• In any case, redundant representations (even
  more than one) of a coded picture must
  immediately follow the corresponding primary
  coded picture

7
H.264/AVC over the network
Error Resilience Tools

• Intra Refresh
• Slicing
• Data partitioning
• Spare Pictures
• Switching (SI/SP) slices
• Flexible Macroblock Ordering
• Redundant Slices

Done Rose, Stockhammer, Done
(probably) Done Stockhammer,Wenger, Done
Hannuksela Done Karczewicz,Setton Not done
(not completely) Not done at all
8
Presentation Outline

• Overview of H.264/AVC over the network
• Error concealment improvements
• Whole frame loss error concealment
• Multi-Hypothesis error concealment
• Redundant Slices FMO
• Evaluation
• A preliminary solution

9
Whole-Frame Concealment (1/4)
Motivation
A picture can often be lost if no one particular
slicing policy is in use (especially at low
bitrates)
Introducing slicing involves a great overhead
10
Whole-Frame Concealment (2/4)
CNR Algorithm

• Motion Vector Projection
• Statistics collection
• Mean Mi,j Variance Vi,j number of contributes
  Ni,j, (either for 4x4 and for 16x16 blocks)
• Motion Field Estimation (16x16 block level)
• If ( Ni,j gt PIXEL_THR and Vi,j lt VARIANCE_THR )
  ? MVi,j Mi,j
• Interpolation of MVs for MBs having at least 3
  stable neighbours
• Motion Field Estimation (4x4 block level)
• For each missing 4x4 block
• if ( Ni,j gt PIXEL_THR and Vi,j lt VARIANCE_THR )
  ? MVi,j Mi,j
• Interpolation of motion vectors for still empty
  blocks
• Picture Reconstruction

P. Baccichet et al. A low complexity
concealment algorithm for the whole-frame loss in
H.264/AVC Proc. IEEE MMSP 2004
11
Whole-Frame Concealment (3/4)
Simulation results
CIF sequences 1 ref frameComparison over
reconstructed pictures
Encoded 150 frames at 15fps Fixed quantizer 28
No intra refresh Simulated the loss of a frame
for all pictures between the 4th and the 150th of
each sequence
12
Whole-Frame Concealment (4/4)
Some examples
Concealed image
Concealed image
Correctly received stream
Copy previous frame
Polito
CNR
13
Slice loss concealment (1/3)
Rationale

• Typically temporal error concealment algorithm
  try to estimate the MV for the lost macroblock
• H.264/AVC produces really chaotic motion vector
  fields, especially when using the Full Search
  motion estimator
• Using a unique motion vector for a whole
  macroblock may produce severe artifacts
• Multi-hypothesis interpolation is an alternative
• Basic idea Efficiency analysis of
  multihypothesis Girod_2000
• Al-Mualla proposed to interpolate the motion
  vector for each pixel on the basis of the MV of
  neighboring MBs
• We follow the same basic approach, working on the
  pixel domain
• Very simple to implement, no computational
  overhead
• (if compared with Boundary Matching solutions)

14
Slice loss concealment (2/3)
Simulation Results
Intra picture every 15
Intra MB refresh every 15
15
Slice loss concealment (3/3)
Example - Foreman
16
Presentation Outline

• Overview of H.264/AVC over the network
• Error concealment improvements
• Whole frame loss error concealment
• Multi-Hypothesis error concealment
• Redundant Slices FMO
• Evaluation
• A preliminary solution

17
Flexible Macroblock Ordering
Mapping functions
Interleaved (FMO0)
Dispersed (FMO1)
Foreground Left Over (FMO2)
18
Flexible Macroblock Ordering
FMO Type 1 (checkerboard) performances
Simulations performed using the WMHI ERCAvg of
50 traces, random losses, 400 bytes x slice
19
Flexible Macroblock Ordering
Considerations

• FMO performances vary as a function of the
  concealment capabilities
• If we use the Time Replacement concealment, we
  do not gain anything using FMO
• FMO protects the whole signal, performing a kind
  of structural protection, even when it is not
  necessary
• Why to protect parts of the signal that do not
  need to be protected?
• Redundant Slices support the retransmission of
  important portions of the signal, allowing a
  selective protection of the signal

20
Redundant Slices
Syntax Elements

• redundant_pic_cnt_present_flag (in the PPS)
  enables the possibility to introduce redundant
  slices
• redundant_pic_cnt (in the slice header) specifies
  if the current slice is part of the primary
  coded picture - PCP ( if equal to 0 ) or part of
  a redundant representation ( if gt 0 )
• Through this field a particular slice is
  associated to the redundant picture it belongs
  to. In practice, it is possible to specify many
  different redundant representations for the same
  PCP.

21
Redundant Slices
Some constraints

• Slice header fields of redundant slices must
  follow some constraints
• field_pic_flag, bottom_field_flag, idr_pic_id
  must be equal to the values in the PCP
• If nal_ref_idc 0 in the PCP, then it has to be
  0 also in the redundant pictures
• frame_num, TopFieldOrderCnt and
  BottomFieldOrderCnt and the reference picture
  marking process must produce the same results
• The pic_parameter_set_id shall be such that the
  value of pic_order_present_flag in the PPS in use
  in a redundant picture is equal to the
  pic_order_present_flag in the PPS in use in the
  corresponding PCP
• NOTE There should be no noticeble difference
  between any area of the decoded primary coded
  picture and a corresponding area that would
  results from application of the decoding process
  for any redundant picture in the same access unit

22
Exploiting Redundant Slices
1 - Blind protection

• The easiest approach is to retransmit the whole
  picture
• Exploit the native rate control available in the
  encoder to control the inserted redundancy
• Simple Multiple Description Coding approach

23
Exploiting Redundant Slices
1 - Blind protection
Simulations performed using the WMHI ERCAvg of
50 traces, random losses, 400 bytes x slice
24
Exploiting Redundant Slices
2 ROI based approach

• Identify Regions Of Interest (ROIs) and send them
  multiple times
• We synthetize the concealed image into the
  encoder
• We construct a Significance Map for the MBs,
  computing the energy of the error (difference
  between the concealed image and the original one)
• We cover the most significant macroblocks with
  some (up to 7) rectangles
• Exploit FMO type 2 to signal the portion of the
  image that will be retransmit
• Recode motion vectors and residual according to
  the defined mapping function

25
Exploiting Redundant Slices
2 ROI based approach
Concealed
Residual Energy
Error Free
-

26
Exploiting Redundant Slices
2 ROI based approach
Fixed Redundancy bitrate 10 of total
27
Exploiting Redundant Slices
3 Content Adaptive approach

• Using the ROI approach does not provide stable
  results
• Good results for slow motion sequences, or with a
  fixed background
• Poor results for fast motion sequences
• FMO type 1 can be exploited to retransmit the
  whole picture when the sequence is rapidly
  changing
• A very simple R/D model is used to choose the
  best solution on a picture by picture basis

28
Exploiting Redundant Slices
3 Content Adaptive approach
Fixed Redundancy bitrate 10 of total
29
Exploiting Redundant Slices
Comparison with FMO
30
Conclusions

• A very simple approach allows to outperform FMO,
  while keeping very efficient the PCP layer
• Further improvements are possible
• A better model to choose among different
  representations of the signal
• Adapt the percentage of the redundancy to insert
  as a function of the PLR
• Change the rate control to allocate different
  amount of redundancy to different pictures,
  depending on the importance
• Integrate with the SLEP scheme