An Industry Perspective on VVG Research

1
An Industry Perspective on VVG Research

• Oliver Grau
• BBC Research Innovation
• VVG SUMMER SCHOOL '07

2
Overview

• Introduction
• (Computer) Vision, Video Graphics in media
  production
• Integration of (real) Video Graphics
• Production visualisation tools
• Free-viewpoint video for Sport visualisation
• Summary

3
Introduction

• BBC is a public funded broadcaster producing
  different kinds of media. Mainly Radio, TV and
  Online
• Video One of the Main businesses
• Graphics used to
• Add editorial value
• Make programmes (visually) more exciting
• Create new users experience
• Vision Big toolbox to make things happen

4
Introduction

• Added editorial value

5
Introduction

• New user experience 3D Virtual environment

CBBC Adventure Rock www.bbc.co.uk/cbbc/adventurer
ock/
6
Introduction
7
Integration of video graphics

• Classical application Weather forecast
• Studio with chroma-key facility

8
What makes an integrated virtual scene looking
real
Depth perception
9
Integration of video graphics

• Virtual studio
• inserting actor into virtual scene

Camera Tracking
Moving camera
10
3D reconstruction of dynamic scenes in the studio

• Virtual studios have limited features regarding
  optical interaction of real and virtual objects
• More realistic integration requires full optical
  interaction in 3D
• Shadows reflections
• Occlusions
• Free choice of camera angle

11
3D reconstruction of dynamic scenes in the studio

• 12 fixed, calibrated cameras with chroma-keying
  facility
• Developed in the IST-ORIGAMI project

12
Overview
Input images, Camera parameters
Surface model
Volumetric model
Volumetric visual hull computation
Surface computation Marching cube algorithm
Texture mapping
..

• Block diagram

13
Visual hull computation

• Visual hull computation from image silhouettes

Camera-3
Camera-1
Camera-2
14
Visual hull computation

• Visual hull computation from image silhouettes

Camera-3
Camera-1
Camera-2
15
Artefacts in Visual hull-based 3D reconstruction

• Fundamental feature
• only parts of the volume that are visible as
  background in at least one of the silhouette
  images are taken out
• no concavities can be modelled
• Occlusion errors
• Phantom volumes
• Reduced by increasing number of cameras
• Approximation errors
• Sampling errors

16
Visual hull-based 3D reconstruction
12 cameras

• Sampling errors
• Binary volumetric scene representation ?
  Quantisation noise

17
Artefacts in dynamic visual hulls

• Moving edges as effect of discrete voxel grid

object
C1
reconstruction
(moving) edge
C2
18
Improved visual hull computation Super-sampling
Camera-1

• Sub-divide voxels by LSS levels
• Assign voxel a (pseudo-) continuous value

19
Improved visual hull computation

• Gaussian smoothing
• (Optional) complexity reduction

20
Results
3D model using super-sampling (20000 triangles)
Standard visual hull / marching cube
After Gaussian smoothing
21
Results

• Video from ORIGAMI demo production

22
Results

• Non-photorealistic rendering

23
Re-lighting
Original image
irradiance
Studio illumination map
3D scene model
Specular component
Light-neutral image
24
Re-lighting