Video Special Effects

1

• Video Special Effects
• Wen-Hung Liao
• 10/3/2006

2
Outline

• Hardware-based video special effects
• Software-based video special effects
• Video content analysis

3
Hardware-based VFX

• Matrox RT.X2 http//www.matrox.com/video/ct/home.c
  fm
• Real-time multi-layer workflows in HD and SD
  Designed primarily for real-time native HDV and
  DV editing, Matrox RT.X2 also provides a
  high-quality MPEG-2 422 I-frame codec so you
  can capture other HD and SD formats using RT.X2's
  analog inputs, and mix all types of footage on
  the timeline in real time.

4
Where to purchase?

• http//www.voxelvision.com.tw/
• http//www.avideo.com.tw/

5
Real-time CPU Effects

• Realtime primary color correction
• Realtime secondary color correction
• Realtime chroma and luma keying
• Realtime speed changes
• Realtime transitions
• Realtime track matte
• Realtime move scale
• Realtime SD clip upscaling in an HD timeline
• Realtime HD clip downscaling in an SD timeline
• Native Adobe Premiere Pro effects

6
Real-time GPU Effects

• Realtime Adobe Motion effect
• Realtime advanced 2D/3D DVE
• Realtime shadow
• Realtime blur/glow/soft focus
• Realtime page curl
• Realtime surface finish
• Realtime pan scan
• Realtime mask
• Realtime mask blur
• Realtime mask mosaic
• Realtime four-corner pin
• Accelerated shine
• Native Adobe Premiere Pro transitions
• Realtime crystallize
• Realtime lens flare
• Realtime old movie effect

7
Graphics Processing Unit (GPU)

• A Graphics Processing Unit or GPU (also
  occasionally called Visual Processing Unit or
  VPU) is a dedicated graphics rendering device for
  a personal computer, workstation, or game
  console.
• Modern GPUs are very efficient at manipulating
  and displaying computer graphics, and their
  highly parallel structure makes them more
  effective than typical CPUs for a range of
  complex algorithms.

8
GPU Operations

• A GPU implements a number of graphics primitive
  operations in a way that makes running them much
  faster than drawing directly to the screen with
  the host CPU.
• The most common operations for early 2D computer
  graphics include the BitBLT operation (combine
  two bitmap patterns using a RasterOp), usually in
  special hardware called a "blitter", and
  operations for drawing rectangles, triangles,
  circles, and arcs.
• Modern GPUs also have support for 3D computer
  graphics, and typically include digital
  video-related functions as well.

9
Applications Example 1

• OpenVIDIA GPU accelerated Computer Vision
  Library, http//openvidia.sourceforge.net/
• The OpenVIDIA project implements computer vision
  algorithms on computer graphics hardware, using
  OpenGL and Cg.
• The project provides useful example programs
  which run real time computer vision algorithms on
  single or parallel graphics processing units.

10
Applications Example 2

• Real-time stereo using GPU
• ... Since the GPU is built to process images it
  is particularly well suited to perform some
  computer vision and image processing algorithms
  very efficiently.  We developed a real-time
  stereo algorithm that runs on the GPU and is
  several times faster than most CPU-based
  implementations.

11
Software-based Video Special Effects

• Examples
• EffectTV http//effectv.sourceforge.net/
• FreeFrame http//freeframe.sourceforge.net/galler
  y.html

12
RGB/YUV Conversion

• http//www.fourcc.org/index.php?http3A//www.fourc
  c.org/intro.php
• RGB to YUV Conversion
• Y (0.257 R) (0.504 G) (0.098 B) 16
• Cr V (0.439 R) - (0.368 G) - (0.071 B)
  128
• Cb U -(0.148 R) - (0.291 G) (0.439 B)
  128
• YUV to RGB Conversion
• B 1.164(Y - 16) 2.018(U - 128)
• G 1.164(Y - 16) - 0.813(V - 128) - 0.391(U -
  128)
• R 1.164(Y - 16) 1.596(V - 128)

13
Types of Special Effects

• Applying to the whole image frame
• Applying to part of the image (edges, moving
  pixels,)
• Applying to a collection of frames (frame-buffer)
• Applying to detected areas
• Overlaying virtual objects
• at pre-determined locations
• in response to users position

14
Compressed-Domain Processing

• Video special effects editing in MPEG-2
  compressed videoFernando, W.A.C. Canagarajah,
  C.N. Bull, D.R
• Fade, dissolve and wipe production in MPEG-2
  compressed videoFernando, W.A.C. Canagarajah,
  C.N. Bull, D.R.

15
Video Content Analysis

• Event detection
• For indexing/searching
• To obtain high-level semantic description of the
  content.

16
Image Databases

• Problem accessing and searching large databases
  of images, videos and music
• Traditional solutions file IDs, keywords,
  associated text.
• Problems
• cant query based on visual or musical properties
• depends on the particular vocabulary used
• doesnt provide queries by example
• time consuming
• Solution content-based retrieval using automatic
  analysis tools (see http//wwwqbic.almaden.ibm.com
  )

17
Retrieval of images by similarity

• Components
• Extraction of features or image signatures and
  efficient representation and storage
• A set of similarity measures
• A user interface for efficient and ordered
  representation of retrieved images and to
  support relevance feedback
• Considerations
• Many definitions of similarity are possible
• User interface plays a crucial role
• Visual content-based retrieval is best utilized
  when combined with traditional search

18
Image features for similarity definition

• Color similarity
• Similarity e.g., distance between color
  histograms
• Should use perceptually meaningful color spaces
  (HSV, Lab...)
• Should be relatively independent of illumination
  (color constancy)
• Localityfind a red object such as this one
• Texture similarity
• Texture feature extraction (statistical models)
• Texture qualities directionality, roughness,
  granularity...

19
Shape Similarity

• Must distinguish between similarity between
  actual geometrical 2-D shapes in the image and
  underlying 3-D shape
• Shape features circularity, eccentricity,
  principal axis orientation...
• Spatial similarity
• Assumes images have been (automatically or
  manually) segmented into meaningful objects
  (symbolic image)
• Considers the spatial layout of the objects in
  the scene
• Object presence analysis
• Is this particular object in the image?

20
Main components of retrieval system

• Database population images and videos are
  processed to extract features (color, texture,
  shape, camera and object motion)
• Database query user composes query via graphic
  user interface. Features are generated from
  graphical query and input to matching engine
• Relevance feedback automatically adjusts
  existing query using information fed back by user
  about relevance of previously retrieved objects

21
Video parsing and representation

• Interaction with video using conventional
  VCR-like manipulation is difficult - need to
  introduce structural video analysis
• Video parsing
• Temporal segmentation into elemental units
• Compact representation of elemental unit

22
Temporal segmentation

• Fundamental unit of video manipulation video
  shots
• Types of transition between shots
• Abrupt shot change
• Fades slow change in brightness
• Dissolve
• Wipe pixels from second shots replace those of
  previous shot in regular patterns
• Other factors of image change
• Motion, including camera motion and object motion
• Luminosity changes and noise

23
Representation of Video

• Video database population has three major
  components
• Shot detection
• Representative frame creation for each shot
• Derivation of layered representation of
  coherently moving structures/objects
• A representative frame (R-frame) is used for
• population R-frame is treated as a still image
  for representation
• query R-frames are basic units initially
  returned in video query
• Choice of R-frame
• first - middle - last frame in video shot
• sprite built by seamless mosaicing all frames in
  a shot

24
Video soundtrack analysis

• Image/sound relationships are critical to the
  perception and understanding of video content.
  Possibilities
• Speech, music and Foley sound, detection and
  representation
• Locutor identification and retrieval
• Word spotting and labeling (speech recognition)
• A possible query could be find the next time
  this locutor is again present in this soundtrack
• Video scene analysis
• 500-1000 shots per hours in typical movies
• One level above shot sequence or scene (a series
  of consecutive shots constituting a unit from the
  narrative point of view)