Audio, Modeling and Simulation

1
Audio, Modeling and Simulation

• Zack Settel
• SAT / University of Montreal

2
Concepts

• Immersion ??
• Content Immersion vs. Display Immersion

3
ImmersiveContent
4
immersive content
5
Immersive Display
6
Immersive Content Generation

• Perspective rendering from a particular point of
  view in the virtual scene - like sculpture

7
Simulation of Objects in 3D space

• Graphical Systems that model
• Behavior of light and interaction with surfaces

8
Simulation of Objects in 3D space

• Graphical Systems that model
• Behavior of (dynamical systems ) moving bodies in
  gravitational fields (mass/spring coupling),
  kinematics.

9
Simulation of Objects in 3D space

• Audio Systems that model
• Behavior of sound (acoustic ) and interaction
  among audio sources and sinks

10
Sound bussed in 3D volume
11

• Modeling behavior of (dynamical systems ) moving
  bodies in gravitational fields (mass/spring
  coupling) which produce sound, physical
  modeling

12
Content Diffusion

• The way that generated content is immersively
  displayed

13
Graphical Display Systems for

• User-independent rendering
• usually for only one user (content is rendered
  for only one user)

14
Graphical Display Systems for

• User-independent rendering
• VR-style head-mounted display

15
Cave Style Display
16
Desk-top Style Display
17
Graphical Display Systems for

• Group rendering
• The display is shared by a number of users, and
  content is rendered for an ideal user sitting
  directly in front of the display.

IMAX theatre display
18
Conventional Cinema Display
19
Audio Display Systems for

• User-independent rendering

20
Group rendering
Surround Sound System
21
History

• Audio vs. Graphics

22
Display Immersion
Both Audio and Graphics have had various
innovations over the years
Graphics
Panavision and Stereoscopic cinema (1950s)
IMAX (1967-present) Hi-resolution,
hemispheric,3D, projection
23
Display Immersion
Audio
3D-sound (1930s)
24
Display Immersion
Audio
Ambisonics, THX (1980s) Dolby Pro Logic,
muti-channel (1988) Surround AC3 and DTS, DVD
etc. (1990s)
25
Content Immersion

3D Graphics for Cinema and Video Gaming

Sophisticated environments (hardware and
software) for rendering of elaborate graphics
simulations
OpenGL Industry Standard (by SGI
1992) Industry Standardized Hardware Rendering
(1990s) Note first major feature film with
all-simulated graphics Toy Story, Pixar Films,
1995
26
(Not to be confused with SURROUND SOUND)
Libraries and Programming Interfaces for
interactive virtual audio environments for
gaming.DirectX (mid-1990s)OpenAL(2002)MPEG-
4(2002)These systems are not really geared
towards interaction with music and sound
limited flexibility, poor modeling.Based on
short-cutsNO HARDWARE RENDERING!!
Content Immersion for 3D Audio
27
Prototype for Immersive Content Rendering
AudioTwist
3D spatial audio rendering, based on acoustic
modeling (e.g. Doppler, distance, refraction,
reflection, air absorption, reverberation
etc Provides for arbitrary complex audio
scenes, with high degree of specificity for each
connection in the audio scene.
28
Concepts and Artistic Implications
Virtual Camera / Virtual Microphone Subjective
Observer (rendered from one point of
view) Narration of the Microphone
(camera) Instrumental use of Spaces Physical
Mixing- Throwing back to classical recording
techniques, and pre-multi-channel recording
techniques (40s, 50s and early-mid 60s).

29
Active Listening
EXAMPLES
30
Virtual Performance Space
31
Example artwork Menagerie Imaginaire
32
What Next?
33
The importance of Hardware Rendering (Industry
Support) for Immersive Audio Content Adoption of
a Standard (e.g mpeg-4) for 3D audio scene
content and interaction
34
FIN