Online Model Reconstruction for Interactive Virtual Environments

1
Online Model Reconstruction for Interactive
Virtual Environments

• Benjamin Lok
• University of North Carolina - Chapel Hill

2
Motivation

• Virtual environments (VEs) usually have
• static models
• generic avatars
• input affordance mismatch
• limited interactivity
• Current options to track users
• multiple trackers
• specialized equipment

3
Previous Work

• Object reconstruction
• Image Based Visual Hulls (Watusik, Beuhler,
  Rasker, Gortler, McMilaln, SIGGRAPH 00)
• Virtualized Reality (Kanade, Rander, Vedula,
  Saito)
• Avatars in VEs
• Slater and Usoh 94
• Slater, Sadagic, Usoh, and Schroeder 00.

4
Goal What We Want to Do

• Real-time models of real objects allows

• Rendering in the environment
• Lighting and shadows
• Interaction with virtual objects
• Active participants in the VE

5
Reconstruction Algorithm

• Outside-looking-in camera images
• Generate an approximation to the real objects
  (visual hull)
• Handle dynamic scenes
• Would like to bypass an explicit 3D modeling
  stage.

6
Reconstruction Algorithm

• Image processing
• Calculating volume intersection
• Rendering the visual hull
• Composite with the VE

7
Object Pixels

• Identify new objects
• Perform image subtraction
• Separate the object pixels from background pixels

current image - background image
object pixels
8
Visual Hull Computation

• Visual hull - tightest volume given a set of
  object silhouettes
• Intersection of the projection of object pixels

9
Visual Hull Computation

• Visual hull - tightest volume given a set of
  object silhouettes
• Intersection of the projection of object pixels

10
Volume Querying

• A point inside the visual hull projects onto an
  object pixel from each camera

11
Volume Querying

• Next we do it for a plane

12
Volume Querying

• For an arbitrary view, we sweep a series of
  planes.

13
Camera Images as Textures

• Intersection computation is hardware accelerated
  with projected textures, stencil and frame
  buffers
• Each camera image is used as a texture with full
  alpha at the object pixels.

14
Visual Hull Intersection

• Project each camera texture onto the plane
• Keep points within the intersection of textures

15
Plane Sweeping
16
Visual Hull from a Viewpoint

• To reconstruct the visual hull from a viewpoint,
  render multiple planes (plane sweeping)

17
Incorporating into the VE

• Correctly depth buffered
• Render the virtual environment after
  reconstruction
• Use the reconstruction within the VE

18
Implementation

• 5 wall and an HMD mounted camera
• SGI Reality Monster - ability to obtain many
  video feeds simultaneously
• Computation
• Image subtraction is the most work
• 30000 triangles/sec, 1.7 gigapixels
• 10-15 fps
• Performance will increase as graphics hardware
  continues to improve

19
Results
20
Where to go from here

• Collision detection
• Collision response
• Image space algorithms
• User studies

21
The End

• Thanks to
• Fredrick P. Brooks Jr.
• Mary Whitton
• Samir Naik
• Funding provided in part by
• NIH National Center for Research Resources (Grant
  Number P41 RR 02170)
• NSF
• Demo at 900 PM on Tuesday at UNC
• http//www.cs.unc.edu/Research/eve/arise