Structural AlignmentBased Temporal Concealment of PacketLoss in Video

1
Structural Alignment-Based Temporal Concealment
of Packet-Loss in Video

• Ajit S. Bopardikar, Odd Inge Hillestad and Andrew
  Perkis
• Centre for Quantifiable Quality of Service in
  Communication Systems
• Trondheim, Norway.

2
Publications Based On This Work

• A. S. Bopardikar, O. I. Hillestad and A. Perkis,
  Temporal error concealment algorithm based on
  structural alignment for packet video,
  International Conference on Signal Processing and
  Communications (SPCOM), Bangalore, India,
  December 2004.
• A. S. Bopardikar, O. I. Hillestad and A. Perkis,
  Structural alignment-based temporal concealment
  of packet-loss in video in review, International
  Conference on Acoustics Speech and Signal
  Processing, Philadelphia, PA, USA, May 2005.

3
This Presentation Is About.

• A novel temporal algorithm to conceal packet-loss
  related distortion in video.
• Algorithm based on the insight that the spatial
  region around the area in the previous frame that
  yields the best concealment will in general be
  highly correlated with the corresponding area
  around the region to be concealed in the present
  frame.

4
Overview

• Packetization overview
• Packet-loss related artifacts in block-based
  video compression schemes.
• Spatial and temporal concealment methods.
• BMA-type temporal concealment methods.
• Proposed method.
• Results and discussions.

5
Flowgraph for Video Stream Packetization
Video stream
File format
Smart mapping
Flows
RTP/UDP
IP packets
Internet/ Simulation Testnet
6
Example MPEG-2 Bitstream
7
MPEG-2 Bitstream

• Each packet can be decoded independently
  irrespective of the errors in decoding previous
  packets.
• Loss of packet means loss of spatial and motion
  vector information for that part of the frame.

8
Packet-loss Related Artifacts in Block-Based
Coding Schemes
9
Concealment Methods for Packet-Loss Related
Errors in Video

• Spatial
• Temporal

10
Spatial Concealment Methods

• Bilinear interpolation based methods.
• Projection onto convex sets (POCS).
• Requires no knowledge of the previous frame(s).
• Simple and can cause blurring.
• Better results can be obtained using temporal
  methods.

11
Temporal Concealment Methods

• Makes use of the high degree of correlation that
  generally exists between consecutive frames.
• The idea is conceal artifacts using
  best-fitting parts from the previous frame
  which is assumed to be error-free.
• Examples
• Temporal Replacement (TR)
• Boundary Matching Algorithm (BMA).

12
Temporal Replacement

• Replaces lost and damaged macroblocks (MBs) in
  the present frame with MBs from the same spatial
  location in the previous, error-free frame.
• In presence of significant motion, TR can give
  rise to visible discontinuities at boundaries of
  replaced MBs.

13
Boundary Matching Algorithm

• Motion vector information typically lost with the
  packet.
• These motion vectors need to be estimated for
  effective concealment.
• BMA assembles a set of candidate motion vectors
  and then chooses a best fit from among the MBs in
  the previous frame associated with them.
• Best fit MB minimizes the sum of mean squared
  error (MSE) or the sum of absolute difference
  (SAD) between its boundaries and the boundaries
  adjacent to them from the top, bottom and left
  MBs around the area to be concealed.

Candidate MB
14
Proposed Algorithm

• Based on the high degree of correlation that can
  exist between spatial region around the best-fit
  candidate MB in the previous frame and the
  corresponding region around the MB-area to be
  concealed in the present frame.

15
Proposed Algorithm

• Insert candidate MB (M-1C) in the area to be
  concealed.
• Compute x and y Sobel gradient maps for the two
  composite blocks.
• Compute the overall magnitude gradient map.
• In the gradient map, compute the SAD between the
  corresponding red and green boundaries of the
  surrounding MBs.
• Choose the MB from the previous frame that
  minimizes the sum of the three SADs for
  concealment.

16
Simulations Particulars

• Full search adopted to search for the best
  fitting MB.
• The area in the previous frame for the search is
  52x52 pixels centered on the spatial position of
  the MB to be concealed.
• All video clips in SIF (352x240 pixels, 30 frames
  per second) MPEG-2 coded at 1.5Mbps.
• Packet-loss simulated using NTT DoCOMo software.
  PLR 0.001.
• Performance compared to a BMA-like algorithm.

17
Simulations Results for Frame 346 of Table
Tennis sequence.
Original
PL (13.84 dB)
BMA (28.91 dB)
Proposed (36.34 dB)
18
Simulation Results PSNR
19
Conclusions

• Proposed algorithm better at aligning edges.
• Relies on matching the structure around the MB to
  be concealed and the candidate MB.
• Low computational complexity.