Light Field Parameterization

1
MIT9904-14 PIs Prof. Leonard McMillan (MIT
LCS), Prof. Julie Dorsey (MIT LCS), and Dr.
Hiroshi Murase (NTT)
Project Goals
Real-time Acquisition
Low Cost Acquisition
The primary focus of our research effort is to
develop technology to create virtual experiences
that will approach the fidelity of the real
world. In the future, such technologies will have
a dramatic impact on the way we work and play.
They will enable new forms of commerce, bring
together individuals separated by large
distances, and provide us with new forms of
entertainment.
Ultimately we intend to create a device for
capturing and processing dynamically reparameteriz
ed light fields in real-time. We call this device
a synthetic aperture camera array. It is
composed of a two-dimensional array of randomly
accessible image sensors that memory-mapped in
the address space of a host processor. Such a
system will allow images to be synthesized from a
wide range of virtual camera positions in
real-time. We plan to support multiple
simultaneous video streams to support
stereoscopic display as well as multiple viewers.
We have also prototyped two low- cost devices for
acquiring light fields. We have developed two
acquisition systems for acquiring light fields
of static scenes. The first uses a
robotic XY-platform to move a digital
camera. This system allows us to explore
the trade-offs between camera spacing
and resolution in order to estimate the
per- formance of our camera array. This system
uses a precision image sensor, precision optics,
and a motion platform with a travel distance of
approximately one meter squares. It can acquire a
16 by 16 image light field in under 20 minutes,
and it cost approximately 10,000 US to
construct. Our second system is based on
an off-the-shelf flat bed scanner, and an array
of plastic lenses. We have modified the scanner
to operate off of battery power so that this
system can be taken out into the field to acquire
images. Additional processing is required to
correct for shortcoming in the image sensor and
low cost optics. None the less, the system can
acquire an 8 by 12 image light field in under 3
minutes, and cost under 100.
Approach
Light Field Parameterization
We take a non-traditional approach to computer
graphics modeling and rendering, in which a scene
is represented by a collection of images rather
than the geometry and surface properties used in
typical computer graphics. Essentially, we treat
a collection of images as a database of rays. New
views can be constructed from this database on a
ray-by-ray basis by selecting the closest ray to
each desired ray.
A high-level block diagram of our proposed system
is shown above. The cameras host interface will
be an industry standard personal computer bus.
The camera array will be constructed from modular
sensor units mounted on a common motherboard.
Display Technology
We have also developed techniques for direct
auto- stereoscopic viewing of our light fields.
These methods are similar to various lenticular
techniques for viewing stereo images. Our
synthetic aperture generation ap- proach provides
much greater flexibility than tradition optical
approaches. In particular it can overcome many
limitations such a focus control and skewed
frustums. We have demonstrated viewers with true
parallax (both horizontal and vertical), and
variable controlled focus. Our displays have
nearly all of the desired properties of
holograms, yet they are true color and
viewable under normal lights. Furthermore, the
technology is easily adaptable to the display of
dynamic 3-D images. Currently we are
only limited by the resolution of flat panel
displays.
The addressing of sensor modules will be
interleaved in order to maximize the
communication bandwidth between the image sensors
and the host computer. Each sensor pod contains a
CMOS image sensor, buffer memory, and glue logic.
The multi- frame buffer memory is used for two
functions. It is used to store information
for noise cancellation, and it allows the host to
access image rays asynch- ronous to the
image scanning process. This modular design
approach will allow us to upgrade to higher
resolution sensors as they become available.
Our dynamically reparameterized light field
representation allows us to synthesize images
with photographic effects such as variable focus
and depth-of-field. Depth- of-field effects are
created by varying the extent of the
reconstruction filters used on the camera
surface.
The image on the left, when viewed through a
hexagonal lens array, can be seen as a
three-dimensional image of a flower. It can be
simul-taneously seen by multiple viewers. It was
computed from a dy-namically re-parameterized
light field, which allows us to precisely control
the focus at all viewing angles. The inset
provides a magnified view of the image.
A variable focal length can be simulated by
varying the focal plane used in
the reconstruction process. In a synthetic
aperture camera both the aperture and
focal- length settings can be varied from pixel
to pixel. The allows effects that are impossible
with a traditional camera.