Performance Capture from Sparse Multi-view Video

1
Performance Capture from Sparse Multi-view Video

• Edilson de Aguiar Carsten Stoll Christian
  Theobalt
• Naveed Ahmed Hans-Peter Seidel Sebastian Thrun
• MPI Informatik, Saarbruecken, Germany
• Stanford University, Stanford, USA
• ACM SIGGRAPH 2008

2
Introduction

• A new marker-less approach to capturing human
  performances from multi-view video
• Jointly reconstruct spatio-temporally coherent
  geometry

3
Outline

• Introduction
• Pre-processing
• Capturing the Global Model Pose
• Capturing Surface Detail
• RESULTS

4
Pre-processing

• Take a full-body laser scan
• 8 cameras , 24fps , 1004x1004 pixels , placed in
  circular
• A coarser tetrahedral version Ttet
• Registered to the first pose of the actor
• in the input footage by ICP

5
Capturing the Global Model Pose

• Pair image features from time t
• Use SIFT feature descriptor
• A set
• pseudo-intersection point of the
  reprojected rays

6
Capturing the Global Model Pose

• Temporal correspondence
• Leordeanu and Hebert 2005
• spectral analysis problem on a graph adjacency
  matrix
• Compute predicted 3D target position
• The best handle vertex vi is the one whose local
  normal is most collinear with the difference
  vector
• A new intermediate target position
• Output

7
Capturing the Global Model Pose

• Re?ning the Pose using Silhouette Rims
• rim vertices it projects into close vicinity of
  the silhouette contour in one of the Ck,t1
• Obtain deformation as same as page 6

8
Capturing the Global Model Pose

• Globally optimizing the positions until good
  silhouette overlap is reached.
• Choose 15-25 key vertices Vk ? Vtet manually

9
Capturing Surface Detail

• Ttet is mapped to Ttri
• check image gradient at the input silhouette
  point has a similar orientation to the image
  gradient in the reprojected model contour image
• If distance between back-projection and original
  position lt
• Then add it as constraint to

10
Capturing Surface Detail

• To recover shape detail, such as folds and
  concavities
• multi-view stereo methodGoesele et al. 2006
• merge the depth maps produced by stereo into a
  single point cloud P
• project points from Vtri onto P Stoll et al.
  2006.
• provide additional position constraints for Eq(3)

11
Result

• 4 different actors
• Feature 200600 each
• After global pose estimation (blue)
• After surface detail reconstruction (green)
• Demo video