Error Resilience

1
Error Resilience Error Concealment Based on
Motion Vector Correction Missing Frame
Prediction

• Chen-Yu Tseng (???)
• Department of Electronic Engineering,
• National Chiao Tung University,
• Taiwan, R.O.C.

2
Introduction
Transmitted Video
Wireless Network
Error Resilience
Received Video
Packet loss
Error Concealment
3
Outline

• Compressed Video Over Wireless Network
• Prior Arts of Error Concealment
• Proposed Error Concealment
• Motion Vector Correction
• Motion Recovery
• Proposed Error Resilience Based on Motion Vector
  Correction
• Experiments
• Conclusion

4
Compressed Video Over Wireless Network
Packet-Based Video Transmission
Encoded Sequence
Decoded Sequence
P
P
I
I
Packet Loss
P
P
P
P
P
P
P
P
Error Propagation
time
time
5
Prior Arts of Error Concealment

• Block-Level Concealment
• Spatial Based
• Spatial-Temporal Based
• Motion Vector Field Based
• Frame-Level Concealment

Missing Block
Correctly Received Parts
6
Block-Level Concealment

• Spatial Interpolation Based 9-11

Original Sequence
Edge Preserving
Bilinear Interpolation
7
Block-Level Concealment

• Spatial-Temporal Based 12-15

Missing MB
Substitute MB
Reference Frame
Current Frame
Boundary Matching Algorithm (BMA) W. M. Lam et
al. 12
8
Block-Level Concealment

• Motion Vector Field Based 16-17

Recovered MV
Missing MB
Substitute MB
Reference Frame
Current Frame
Interpolation in MV Field
9
Whole-Frame Loss

• Example of 3G Applications
• QCIF Video Transmission at 64 kbps.
• Frame Rate 10 fps.
• Average 800 bytes per frame
• General Packet Size 1kB
• Packet Loss May Cause Whole-Frame Loss!!

10
Frame-Level Concealment

• Motion Vector Recovery Based on Optical Flow

Recovered MV
Ref MV
Substitute MB
Reference Frame
Current Missing Frame
11
Frame-Level Concealment

• S. Belfiore, M. Grangetto, E. Magli, G. Olmo, An
  error concealment algorithm for streaming video,
  Proc. ICIP 2003, 2003 22
• S. Belfiore, M. Grangetto, E. Magli, G. Olmo,
  Concealment of whole-frame loss for wireless low
  bit-rate video based on multiframe optical flow
  estimation, IEEE Trans. Multimedia, vol. 7, no.
  2, pp. 316-329, Apr. 2005. 23
• P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni,
  and F. Rovati, Frame concealment for H.264/AVC
  decoders, IEEE Trans. Consumer Electronics, vlo.
  51, no. 1, pp. 227-233, Feb. 2005. 24
• Zhenyu Wu and J. M. Boyce, An error concealment
  scheme for entire frame losses based on
  H.264/AVC, Proc. ISCAS 2006, May 2006.25

12
Optical Flow (OF)

• Motion Consistency

t-2
t-2
t-1
t-1
t
time
Optical Flow
t
13
Motion Recovery Based on Optical Flow
Packet lost
t-2
time
time
t-1
Corrupted Optical Flow
Original Optical Flow
t
time
Optical Flow
14
Optical Flow Estimation

• General Model of Optical Flow Equation27
• Motion Vector substitutes for OF

(1)
(2)
(i, j)
(iF, jF)
Fn-1
Fn
Motion Vector Projection
15
MV Versus OF

• MV ? Block Matching
• OF ? Motion Trajectory

Block Matching
Reference Frame Foreman 4
Current Frame Foreman 5
16
Temporal Aliasing Problem
Wrong MV
Correct MV
Projected MV
Projected MV
t-2
t-2
t-1
t-1
t
Missing Frame
t
Correct MV Projection
Wrong MV Projection
17
Pixel-based Concealment

• Schematic of Belfiores Algorithm 22

Reference Frame Buffer
Estimation of optical flow with temporal
regularization
Spatial regularization of FMV field
Projection of prev. frame onto missing frame
Filtering and downsampling
Interpolation of missing pixels
18
Pixel-based Concealment

• Schematic of Belfiores Algorithm 22

Reference Frame Buffer
Estimation of optical flow with temporal
regularization
Spatial regularization of FMV field
Projection of prev. frame onto missing frame
Filtering and downsampling
Interpolation of missing pixels
19
Pixel-based Concealment

• Estimation of optical flow with temporal
  regularization

n
n-1
n-2
MV
MV
MV
MV
TERM
MVH
MVH
TERM
MVH
MVH
There are other temporal interpolators for the
MVH, including median filter and weighted
averages decaying overtime. Belfiore et al. 22
find that mean value consistently yields the best
result, thus validating the linear velocity
assumption.
20
Pixel-based Concealment

• Schematic of Belfiores Algorithm 22

Reference Frame Buffer
Estimation of optical flow with temporal
regularization
Spatial regularization of FMV field
Projection of prev. frame onto missing frame
Filtering and downsampling
Interpolation of missing pixels
t
t-1
21
Pixel-based Concealment

• Schematic of Belfiores Algorithm 22

Reference Frame Buffer
Estimation of optical flow with temporal
regularization
Spatial regularization of FMV field
Projection of prev. frame onto missing frame
Filtering and downsampling
Interpolation of missing pixels
t
t-1
22
Pixel-based vs. Block-Based
t
t
t-1
t-1
Pixel-based
Block-based
23
Block-based Concealment

• Schematic of P. Baccichts Algorithm 24

24
Problems of MV Projection

• Conflict State
• Non-covered State

Non-covered State
Conflict State
t-1
25
Problems of MV Projection

• Conflict State
• Non-covered State
• Possible Reasons
• Wrong Motion Vector
• Object Warping, Occlusion, or Non-translation
  Motion.
• Frame Boundary

26
Outline

• Compressed Video Over Wireless Network
• Prior Arts of Error Concealment
• Proposed Error Concealment
• Motion Vector Correction
• Motion Recovery
• Proposed Error Resilience Based on Motion Vector
  Correction
• Experiments
• Conclusion

27
Motion Vector Correction

• Motion Vector ? True Motion Trajectory
• Property of True Motion Trajectory
• Temporal Consistency
• Spatial Consistency

Temporal Consistency
t-2
t-1
t
Spatial Consistency
28
Motion Vector CorrectionBased on Fuzzy Logic
Motion Vector Field
Original MV Corrected MV Reliability of
Spatial Reliability Temporal
Reliability
Spatial Difference
Temporal Difference
29
Temporal Reliability

• Temporal Difference

t-2
t-2
t-1
t-1
t
t
time
time

• High temporal
• difference
• Low temporal
• reliability

• Low temporal
• difference
• High temporal
• reliability

1
0
30
Spatial Reliability

• Spatial Difference

Motion Vector Field
Motion Vector Field
31
Motion Vector CorrectionBased on Fuzzy Logic
y
x
Motion Vector Field
Original Frame
Original MV Field
Corrected MV Field
32
MV Projection with MV Correction
y
x
Original MV Field
Corrected MV Field
t
t
t-1
t-1
33
Proposed MV Projection

• Why Concealment?
• Release Error Propagation

Can We Skip from Error Frame?
P
P
I
I
P
P
P
P
P
P
P
P
Error Propagation
Error Frame
Error Frame
time
time
34
MV Projection Forward vs. Backward
Forward MV Projection
Packet lost
Packet lost
Backward MV Projection
time
time
time
Corrupted Optical Flow
Original Optical Flow
Corrupted Optical Flow
35
MV Projection Forward vs. Backward
OF
OF
Reference Frame
Reference Frame
Missing Frame
Missing Frame
Backward projected OF
Forward projected OF
OF
(a) Forward motion projection
(b)Backward motion projection
36
MV Projection Forward vs. Backward
Backward Projected MV
Missing frame
Missing frame
(i, j)
(iF, jF)
(iB, jB)
Fn-1
Fn
(i, j)
Fn-1
Fn
Fn1
37
Advantage ofBackward MV Projection

• Easy to implement.
• Avoid from conflict or non-cover states.
• Invariance of I-frame position.

Missing frame
Missing frame
P
P
I
I
P
P
P
time
P
P
time
P
P
P
Missing frame
No MV
Missing frame
P
P
P
P
I
I
P
time
P
P
time
P
P
P
Forward Motion Projection
Backward Motion Projection
38
Outline

• Compressed Video Over Wireless Network
• Prior Arts of Error Concealment
• Proposed Error Concealment
• Motion Vector Correction
• Motion Recovery
• Proposed Error Resilience Based on Motion Vector
  Correction
• Experiments
• Conclusion

39
Schematic of Proposed System
Wireless Network
Encoder with MV Correction
Decoder with Missing Frame Prediction
Error Resilience
Error Concealment
40
Motion Vector Correction

• Spatial-Temporal MV Correction (3-D)

Iterative Correction
time

n-2
MV Field
n
x
n-1
The number of MV correction iteration effects the
result of concealment.

y
The PSNR is the concealment result (foreman
FLR10) with different number of MV correction
iteration
41
Proposed 2-StepMotion Vector Correction
Forward MV correction
Backward MV correction
Iteration
time
42
Concealment PSNRwith Different MV Correction
The PSNR is the concealment result (foreman FLR10)
2-Step 33.4666 dB Bidirectional 33.1826 dB
43
Motion Compensation PSNR
PSNR Drop
With MV Correction 32.0382 dB Without MV
Correction 34.5538dB
(foreman)
44
PSNR Drop

• Effect Increase the bit rate
• Reason Inconsistent Motion
• MV correction is based on motion consistency.
• In this situation,
• concealment is also hard to work.
• We preserve the original MV to avoid MC PSNR
  drop.

45
PSNR Drop Control Factor
If Residual_MADcorrect gt f Residual_MADuncorrec
t MVcorrect MVoriginal Residualcorrect
Residualoriginal
ME
MC
MV Correction
MC
f is the control factor
46
MC_PSNR vs. EC_PSNR
factor MC_PSNR EC_PSNR
inf 32.0382 33.8803
6 32.4231 33.8397
5 32.5454 33.7881
4 32.7421 33.6529
3.5 32.9059 33.5187
3.25 32.9981 33.4395
3 33.106 33.3602
2.75 33.2406 33.2511
2.5 33.4009 33.132
2.25 33.5861 32.8189
2 33.789 32.5876
1.75 34.0185 32.2453
1.5 34.2771 31.5792
1.25 34.5392 31.0878
1 34.6723 30.544
0 34.5538 30.2423
Foreman QCIF MC_PSNR Motion Compensation
PSNR EC_PSNR Error Concealment PSNR (Backward MV
concealment FLR10)
47
MC_PSNR vs. EC_PSNR
Foreman QCIF MC_PSNR Motion Compensation
PSNR EC_PSNR Error Concealment PSNR (Backward MV
concealment FLR10)
48
Error Concealment Results

• Foreman QCIF Backward Concealment FLR10 (63)

MAD Factor inf
MAD Factor 2.75
49
Refinement of Concealment

• MV Temporal Consistency Check

t-2
t

• Frame Spatial Consistency Check

50
MV Temporal Consistency Check
t-2
t
Difft(i, j) MVt(i, j) MVt-2(iB, jB)
,Where (iB, jB) (i, j) 2MVt(i, j) If
Difft(i, j) gt Diff_temporal_TH MVt(i, j)
MVt(i, j) , if Diff_v(i, j) lt Diff_h(i, j),
MVt(i, j) ( MVt(i-1, j) MVt(i1, j) )/2
else MVt(i, j) ( MVt(i, j-1) MVt(i, j1)
)/2 where Diff_v(i, j) MVt(i-1, j) -
MVt(i1, j) and Diff_h(i, j)
MVt(i, j-1) - MVt(i, j1)
MV Refinement
51
Frame Spatial Consistency Check
4x4 block
difference
difference
If difference of the four directions are all
greater then Diff_spatial_TH MVt(i, j)
MVt(i, j)
difference
difference
52
MC_PSNR V.S. EC_PSNR
factor MC_PSNR EC_PSNR EC_r_PSNR
inf 32.0382 33.8803 34.2188
6 32.4231 33.8397 34.1206
5 32.5454 33.7881 34.0954
4 32.7421 33.6529 34.0545
3.5 32.9059 33.5187 34.0389
3.25 32.9981 33.4395 33.9509
3 33.1060 33.3602 33.9143
2.75 33.2406 33.2511 33.8724
2.5 33.4009 33.1320 33.8054
2.25 33.5861 32.8189 33.6826
2 33.7890 32.5876 33.5845
1.75 34.0185 32.2453 33.4763
1.5 34.2771 31.5792 33.2406
1.25 34.5392 31.0878 32.8728
1 34.6723 30.5440 32.5500
0 34.5538 30.2423 32.4962
Foreman QCIF MC_PSNR Motion Compensation
PSNR EC_PSNR Error Concealment PSNR (Backward MV
concealment FLR10) EC_r_PSNR Error Concealment
PSNR with MV refinement
53
MC_PSNR V.S. EC_PSNR
54
MC PSNR (foreman qcif)
MAD Factor 0 34.5538dB MAD Factor 2.75
33.2406 dB MAD Factor inf 32.0382 dB
55
Error Concealment Results
Foreman QCIF Backward Concealment FLR10
MAD Factor inf PSNR 33.8803 MAD Factor
2.75 PSNR 33.2511 MAD Factor 2.75 with MV
refinement PSNR 33.8724
56
Error Concealment Results

• Foreman QCIF Backward Concealment FLR10 (63)

a. b. c. a. MAD Factor inf b. MAD Factor
2.75 c. MAD Factor 2.75 with MV refinement
57
Outline

• Compressed Video Over Wireless Network
• Prior Arts of Error Concealment
• Proposed Error Concealment
• Motion Vector Correction
• Motion Recovery
• Proposed Error Resilience Based on Motion Vector
  Correction
• Experiments
• Conclusion

58
Experiment of Concealment
Encoded Video Sequence
Video Transmission Simulator
Decoder
Error Concealment
Output Video Sequence
59
7 Chih-Heng Ke, Cheng-Han Lin, Ce-Kuen Shieh,
Wen-Shyang Hwang, A Novel Realistic Simulation
Tool for Video Transmission over Wireless
Network, The IEEE International Conference on
Sensor Networks, Ubiquitous, and Trustworthy
Computing (SUTC2006), June 5-7, 2006, Taichung,
Taiwan.
60
Results
FMP Forward MV Projection BMP Backward MV
Projection with MV correction
Foreman QCIF, 15 fps PLR 1 without EC 32.4318
dB FMP 35.2562 dB BMP 35.5058 dB
61
Concealment Results
Foreman QCIF 54, 15 fps PLR 1
Forward MV Projection
Backward MV Projection
62
Results
FMP Forward MV Projection BMP Backward MV
Projection with MV correction
flower QCIF, 15 fps PLR 2 without EC 30.4504
dB FMP 33.4728 dB BMP 34.5167 dB
63
Concealment PSNR
Sequence PLR PSNR Gain
Sequence PLR No EC FMP BMP FMP BMP
foreman 1 32.4318 35.2562 35.5058 2.8244 3.074
2 29.7481 34.5100 34.9532 4.7619 5.2051
5 24.6585 33.5158 34.3082 8.8573 9.6497
flower 1 32.6611 34.1363 34.5167 1.4752 1.8556
2 30.4504 33.4728 34.1768 3.0224 3.7264
5 20.6533 30.2924 32.2664 9.6391 11.6131
stefan 1 31.2693 33.4323 33.4790 2.1630 2.2097
2 26.3288 31.9067 32.0947 5.5779 5.7659
5 18.4289 29.1995 29.5135 10.7706 11.0846
64
Conclusion

• We propose a new motion vector correction method
  to make MV approximate true motion trajectory.
• We propose an error resilient scheme based on MV
  correction.
• We propose a new whole-frame concealment based on
  backward MV projection which is simple to
  implement.

65
Reference

1. Thomas Wiegand, Gary J. Sullivan, Gisle
   Bjontegaard, and Ajay Luthra, Overview of the
   H.264/AVC Video Coding Standard, IEEE Trans. on
   Circuits Syst. Video Technol., vol. 13, No. 7,
   pp.560 576, July 2003
2. Iain E G Richardson, H.264 and MPEG-4 Video
   Compression, John Wiley, 2003
3. Thomas Stockhammer, Miska M. Hannuksela, and
   Thomas Wiegand, H.264/AVC in Wireless
   Environment, IEEE Trans. On Circuits Syst. Video
   Technol., vol. 13, No. 7, July 2003
4. Stephan Wenger, H.264/AVC Over IP, IEEE Trans.
   On Circuits Syst. Video Technol., vol. 13, No. 7,
   July 2003
5. Thomas Stockhammer, Thomas Wiegand, Tobias
   Oelbaum, and Florian Obermeier, Video coding and
   transport layer techniques for H.264/AVC-based
   transmission over packet-lossy networks, Proc.
   ICIP 2003, vol. 3, September 2003.
6. J. Klaue, B. Rathke, and A. Wolisz, "EvalVid A
   Framework for Video Transmission and Quality
   Evaluation", In Proc. of the 13th International
   Conference on Modelling Techniques and Tools for
   Computer Performance Evaluation, Urbana,
   Illinois, USA, September 2003.

1. Chih-Heng Ke, Cheng-Han Lin, Ce-Kuen Shieh,
   Wen-Shyang Hwang, A Novel Realistic Simulation
   Tool for Video Transmission over Wireless
   Network, The IEEE International Conference on
   Sensor Networks, Ubiquitous, and Trustworthy
   Computing (SUTC2006), June 5-7, 2006
2. Network Simulator, http//www.isi.edu/nsnam/ns/
3. Zhang Rongfu, Zhou Yuanhua, and Huang Xiaodong,
   Content-adaptive spatial error concealment for
   video communication, IEEE Trans. On Consumer
   Electronics, vol. 50, Feb. 2004.
4. Olivia Nemethova, Ameen Al-Moghrabi and Markus
   Rupp, Flexible Error Concealment for H.264 Based
   on Directional Interpolation, International
   Conference on Wireless Networks, Communications
   and Mobile Computin, vol. 2, June 2005.
5. Steven Beesley, Andrew Armstrong, Christos
   Grecos, An Edge Preserving Spatial Error
   Concealment Technique for the H.264 Video Coding
   Standard, Research in Microelectronics and
   Electronics 2006, Ph. D., June 2006.

66
Reference

1. W. M. Lam, A. R. Reibman, and B. Liu, Recovery
   of lost orerroneously received motion vectors,
   in Proc. ICASSP93, pp. V417-V420, Apr. 1993.
2. E. T. Kim, S.-J. Choi, and H.-M. Kim, Weighted
   boundary matching algorithm for error concealment
   in the MPEG-2 video bit stream, Signal Process.,
   vol. 73, pp. 291295, Mar. 1999.
3. Yan Chen, Au, O., Chiwang Ho, and Jiantao Zhou,
   Spatio-temporal boundary matching algorithm for
   temporal error concealment, Proc. ISCAS 2006,
   May 2006.
4. Agrafiotis, D., Bull, D.R., Canagarajah, C.N.,
   Enhanced Error Concealment With Mode Selection,
   IEEE Trans. on Circuits and Systems for Video
   Technology, vol. 16, August 2006.
5. M. E. Al-Mualla, C. N. Canagarajah, and D. R.
   Bull, Error concealment using motion field
   interpolation, Proc. ICIP 1998, Vol. 2, pp.
   512516, October 1998.
6. Jinghong Zheng, Lap-Pui Chau, A temporal error
   concealment algorithm for H.264 using Lagrange
   interpolation, Proc. ISCAS 2004, vol. 2, May
   2004.

1. Jae-Young Pyun, Jun-Suk Lee, Jin-Woo Jeong,
   Jae-Hwan Jeong, and Sung-Jea Ko, Robust error
   concealment for visual communications in
   burst-packet-loss networks, IEEE Trans. On
   Consumer Electronics, vol. 49, November 2003
2. S. Gnavi, M. Grangetto, E. Magli, and G. Olmo,
   Comparison of rate allocation strategies for
   H.264 video transmission over wireless lossy
   correlated networks, in Proc. ICME\IEEE Int.
   Conf. Multimedia and Expo, 2003.
3. E. N. Gilbert, Capacity of a burst-noise
   channel, Bell Syst. Tech. J., vol. 39, pp.
   1253-1265, Sept. 1960.
4. E. 0. Elliott, Estimates of error rates for
   codes on burst-noise channels, Bell Syst. Tech.
   J., vol. 42, pp. 1977-1997, Sept. 1963.
5. S. Belfiore, M. Grangetto, E. Magli, G. Olmo,
   Concealment of whole-frame loss for wireless low
   bit-rate video based on multiframe optical flow
   estimation, IEEE Trans. Multimedia, vol. 7, no.
   2, pp. 316-329, Apr. 2005.
6. S. Belfiore, M. Grangetto, E. Magli, G. Olmo, An
   error concealment algorithm for streaming video,
   Proc. ICIP 2003, 2003.

67
Reference

1. Baccichet, P.and Chimienti, A., A low
   complexity concealment algorithm for the
   whole-frame loss in H.264/AVC, IEEE 6th Workshop
   on Multimedia Signal Processing, October 2004.
2. P. Baccicht, D. Bagni, A. Chimienti, L.Pezzoni,
   and F. Rovati, Frame concealment for H.264/AVC
   decoders, IEEE Trans. Consumer Electronics, vol.
   51, no. 1, pp. 227-233, Feb. 2005.
3. Zhenyu Wu and J. M. Boyce, An error concealment
   scheme for entire frame losses based on
   H.264/AVC, Proc. ISCAS 2006, May 2006.
4. A. M. Tekalp, Digital Video Processing. Englewood
   Cliffs, NJ Prentice- Hall, 1995.