Internetbased interactive HDTV

1
Internet-based interactive HDTV

• Bin Yu, Klara Nahrstedt
• Department of Computer Science
• University of Illinois at Urbana-Champaign
• ACM/Springer Multimedia Systems Journal, 9(5)
  March, 2004

2
Interactive HDTV Traditional Approach

• TV cable network set-top box TV set
• High quality video delivery
• Not compatible, proprietary format
• Not scalable, all processing is done in the TV
  studio
• Interactivity, flexibility and customization are
  not easy

3
Interactive HDTV Traditional Approach
4
Interactive HDTV Traditional Approach

• PC Internet
• Easy interactivity, flexibility and customization
• No high quality HDTV display
• Current software solutions are inefficient

5
HDControl Authors Approach

• Internet PC TV set
• Combine good features of previous solutions
• PC function as open set-top box
• Standard video format MPEG2
• Distributed architecture

6
HDControl Architecture
7
Typical Application

• Picture-in Picture (PiP)
• Logo/ticker insertion
• Email browsing

8
Visual Information Embedding (VIE)

• Naïve solution not feasible
• Decode the video stream into raw pixels
• Overlay embedded content
• Re-encode to MPEG2 stream

9
MPEG2 Encoder Diagram Recall
10
Typical GOP Structure
11
Visual Information Embedding

• Inverse Variable Length Coding (VLC) converts
  bit-stream to Motion Compensation (MC) domain
• MC contains motion vectors and prediction errors
  (residual) in quantized DCT format
• Do MC to get reconstructed DCT (RD) domain data
• Replace part of the image with embedded
  information
• Re-encode data to MPEG2 format

12
VIE

• Wrong reference problem
• MB2 use MB1 as a reference for prediction

13
VIE

• Fix reference
• RD domain data of MB2, MB1 and MB0 are required
• Worst case IBBPBBPBBPBBPBB
• Maximum search distance 16 macroblocks
• Potentially all the MBs in I and P frames need to
  be decoded for future use

14
Two Types of Macroblocks

• d-MBs
• MB0 and MB1, their data should be decoded to RD
  domain for future reference
• c-MBs
• MB2, their reference blocks are wrong and their
  MC data have been changed

15
Efficient Motion Compensation

• Only those macroblocks surrounding the foreground
  window are affected by VIE (Chang et al. )
• Future motion prediction pattern is unknown
• Completely reconstruct the reference frame

16
Efficient Motion Compensation

• Introduce delay
• Buffering a GOP
• All c-MBs can be detected by testing whether
  their reference MBs are in the foreground area
• The reference MBs for both c-MBs and d-MBs are
  d-MBs

17
Efficient Motion Compensation

• Back tracking
• c-MB -gt d-MB -gt -gtd-MB
• Perform motion compensation for c-MBs and d-MBs
  to get RD domain data
• Perform motion estimation for c-MBs to get their
  new motion vectors and prediction errors

18
Optimizations

• Bi-direction prediction -gt uni-direction
  prediction
• B frames has two reference frames
• One reference frame is in the foreground area
• Delete one motion vector

19
Optimizations

• Mark out sensitive area
• Foreground window may only occupy a small area
• Define c-sensitive area which may contains c-MBs
  define d-sensitive are which may contains d-MBs.
• For slices in insensitive area, copy them
  directly from input to output without decoding

20
Optimizations

• Shorten the delay
• Select a shorter GOP size
• Start back tracking process earlier
• The sensitive area for the last P frame is mainly
  within the foreground window
• IBBPBBPBBPBBPBB

21
Resynchronization

• MPEG2
• Decoding timestamp (DTS) and presentation
  timestamp (PTS) for each frame are determined by
  sender clock
• Synchronization point (PCR) in multiplexed stream
• Distance between PCR is constant

22
De-multiplexing and Multiplexing
23
Resynchronization

• VIE problem
• The frame size will be changed by VIE process
• The constant spacing of PCR will be violated

24
Two solutions

• Simple padding
• Pad Null packet in the empty space to keep the
  PCR positions
• Can only handle bit-rate reduction
• Waste bandwidth
• Time-invariant bit-rate scaling
• To scale the bit-rate to another constant value
  with scaling factor SF

25
Determine SF

• Measure the maximal ratio (R) in number of
  packets before and after VIE offline
• Monitor the actual ratio (AR) online
• New SF

26
Evaluation

• Experimental environment
• HDTV testbed
• Live high-definition digital TV stream from the
  satellite or storage device is fed into the
  server
• Encode video into MPEG2 format and then multicast
  over LAN
• Player PC decode the stream and send it to TV set
• Video editing servants receive the stream and
  perform editing operation
• Multicast result stream

27
PiP

• FG football_sd.mpg(480256, 30fps)
• BG trees1.mpg(19201088, 30fps)

28
Evaluation

• Distribution of d-MBs

29
Comparison with the previous approach

• The number of MBs to be convert from MC domain to
  RD domain

30
Resynchronization results
31

• Thank you!