DVHDV Tape Drive Synchronization

1
DV/HDV Tape Drive Synchronization

• Stereoscopic Displays and Applications Conference
• 29th 31th January 2007
• San Jose, United States

2
Overview

• Introduction to the Stereoscopic Multiplexer
• Concept and architecture
• DirectShow capture graphs
• Simple tape drive synchronization
• Basic idea
• Missync detection
• Advanced tape synchronization concepts
• Buffering data in system memory
• Recognizing invalid scenes

3
Stereoscopic Multiplexer Concept
Right stream
Left stream
Combined video stream
4
DirectShow

• DirectShow
• Architecture to process media data on the Windows
  platform
• Processing objects are called filters
• Filters expose input and output pins
• Filter pins can be connected to a directed graph
• Each capture device is represented by a capture
  filter
• Stereoscopic Multiplexer
• Not a proprietary capture applications but
• implemented as DirectShow capture filter
• Can be used from any capture application

5
DirectShow Capture Graph
Capture Device
Sink

• The capture application loads
• the capture device filter
• the downstream filters
• Example downstream filters are
• video renderers
• file writer
• network renderer

6
Stereoscopic Multiplexer Capture Graph

• The application loads
• the Stereoscopic Multiplexer
• the downstream filters
• The Stereoscopic Multiplexer loads
• the real physical capture drivers
• video decoders (if required)

7
Stereoscopic Multiplexer Architecture
Application
Application
DirectShow- Filter
Stereoscopic Multiplexer
User Mode
WDM Video Capture Filter
WDM Video Capture Filter
WDM Video Capture Filter
Miniport Driver
Miniport Driver
MiniportDriver
Kernel Mode
8
Stereoscopic Video Recording

• One camcorder optical attachment
• Half resolution per eye
• Degraded image quality
• Simple playback

• Two camcorders remote control
• Full resolution
• Optimum image quality
• No immediate playback possible

9
Dual Tape Playback

• Problem
• Different number of frames on the tapes
• Usually happens when camcorders were turned off
• Playing back dual camcorder recordings
• Starting both tapes simultaneously ? tapes drift
  apart
• Transfer both tapes to harddisk ? splitting
  scenes ?playing each scene separately ? ok but
  inconvenient

10
Tape Synchronization

• Tape synchronization involves
• detecting missync
• correcting missync
• Detecting missync
• Image analysis tracking motion and parallax?
  not reliable, computationally complex
• Sound analysis detecting corresponding events in
  the left and right audio track? does not work
  for silent scenes
• Scene change detection based on metadata
  (recording date and time) in the compressed DV
  stream? reliable, no computational overhead

11
Basic Tape Synchronization

• Resynchronize
• Pause first tape
• Wait for scene changeon second tape
• Pause second tape
• Step forward tosmallest common offset from the
  scene changes
• Send play commandto both tapes
• A tolerance of one frame is good tradeoff between
  playback interruptionsand sync quality

Wait for scene change
Yes
Scene change occurs on the other tape within a
certain tolerance?
No
Resynchronize
12
Basic Tape Synchronization Graph
Tape control
Timecode
Recording date/time, timecode
Microsoft DV Camera and VCR
DV Video Decoder
3dtv.at Metadata Reader
Sink
3dtv.at Stereoscopic Multiplexer
Microsoft DV Camera and VCR
DV Video Decoder
3dtv.at Metadata Reader
Created by Stereoscopic Multiplexer
13
Tape Synchronization Graph with Audio
3dtv.at Audio Slaving Filter
Audio Renderer
Media type A/V interleaved
DV Splitter
Microsoft DV Camera and VCR
3dtv.at Metadata Reader
DV Video Decoder
3dtv.at Stereoscopic Multiplexer
Media type Video only
Microsoft DV Camera and VCR
DV Video Decoder
3dtv.at Metadata Reader

• Audio data are sent to sound card
• Audio data are not available for capturing

14
HDV Support

• HDV standards
• HDV1 720p (1280 x 720 pixels), used by JVC
• HDV2 1080i (1440 x 1080 pixels), used by Sony
  and Canon
• MPEG-2 interframe compression
• Downconversion mode
• Standard definition playback ? low system
  requirements
• Firewire output of camcorders set to DV mode
• Camcorders perform downconversion from HDV to DV
• DV graphs still work
• Native HDV mode
• High definition playback ? high system
  requirements
• Metadata readers parse MPEG streams
• Changes to the filter graph are necessary

15
HDV Graph
3dtv.at Audio Slaving Filter
Audio Renderer
MPEG Audio Decoder
Audio stream
MPEG-2 Demultiplexer
Microsoft AVC Tape Subunit Device
MPEG-2 Video Decoder
3dtv.at Metadata Reader
Video stream Video aux stream
3dtv.at Stereoscopic Multiplexer
Microsoft AVC Tape Subunit Device
MPEG-2 Video Decoder
3dtv.at Metadata Reader
MPEG-2 Demultiplexer

• MPEG transport stream
• PID 0810h Elementary video stream
• PID 0814h Elementary audio stream
• PID 0811h Video auxiliary stream (contains
  metadata)

16
Advanced Tape Synchronization

• Problems with basic tape synchronization
• Frequent playback interruptions
• Frames are multiplexed in the order they arrive
• Precision only approx. 1 frame
• Solution
• Synchronize the multiplexer output, not the
  camcorders
• Buffer data in system memory to delay the stream
  that is ahead
• Resynchronize tapes only if buffer is full
• Precise synchronization based on offset from
  scene change
• Invalid scene recognition
• Detect scenes that are present just on one of the
  tapes
• Pause the other tape until matching scenes have
  been found
• Implementation Monitor difference of left and
  right recording date/time
• Clock time might not be set accurately ? do not
  monitor absolute time

17
Invalid Scene Recognition

• Do not drop the first scene after startup
• Do not drop the first scene after a recording
  break (it is likely that the clocks have been
  adjusted or another pair of camcorders was used)

18
Advanced Tape Synchronization Graph
3dtv.at Audio Slaving Filter
Audio Renderer
DV Splitter
Microsoft DV Camera and VCR
3dtv.at Metadata Reader
3dtv.at Media Sample Buffer
DV Video Decoder
3dtv.at Stereoscopic Multiplexer
Microsoft DV Camera and VCR
DV Video Decoder
3dtv.at Metadata Reader
3dtv.at Media Sample Buffer

• Media sample buffers operate on compressed
  streams
• Buffering one second DV/HDV video requires 3.5 MB
  of memory

19
Stereoscopic Multiplexer Versions

• Version 0.5 (May 2006)
• Basic DV tape synchronization
• Configuration tool

• Version 0.6 (Q1 2007)
• Advanced DV/HDV tape synchronization
• HDV support (HDV2 only)

20
Thanks for Your Attention

• ?

• QA

Visit us at http//www.3dtv.at!