Virtualized Audio as a Distributed Interactive Application

1
Virtualized Audioas aDistributed Interactive
Application

• Peter A. Dinda
• Northwestern University
• Access Grid Retreat, 1/30/01

2
Overview

• Audio systems are pathetic and stagnant
• We can do better Virtualized Audio (VA)
• VA can exploit distributed environments
• VA demands interactive response

What I believe
Why I care
3
Traditional Audio (TA) System
Performance Room
Listening Room
Performer
Amp
Loudspeakers
Sound Field 1
Sound Field 2
Mixer
Microphones
Listener
Headphones
4
TA Mixing And Filtering
Perception of Headphone Reproduced Sound
Listeners Location and HRTF
Perception of Real Sound
Performance Room Filter
Mixing (reduction)
Microphone Sampling
Headphones
Performer
Perception of LoudspeakerReproduced Sound
Listeners Location and HRTF
Loudspeaker Filter
Listening Room Filter
Amp Filter
5
Virtualized Audio (VA) System
6
VA Filtering, Separation, and Auralization
VA Forward Problem
VA Reverse Problem
7
The Reverse Problem -Source Separation
other inputs
microphone signals
sound source positions
Recovery Algorithms
sound source signals
room geometry and properties
microphone positions
Human Space
Microphones
Reverse Problem

• Microphone signals are a result of sound source
  signals, positions, microphone positions, and the
  geometry and material properties of the room.
• We seek to recover these underlying producers of
  the microphone signals.

8
The Reverse Problem

• Blind source separation and deconvolution
• Statistical estimation problem
• Can unblind problem in various ways
• Large number of microphones
• Tracking of performers
• Separate out room deconvolution from source
  location
• Directional microphones
• Phased arrays

Potential to trade off computational
requirements and specialized equipment
Much existing research to be exploited
9
Transducer Beaming
l gt L
l gtgt L
l L
l lt L
Wave
l ltlt L
L
Transducer
l
10
Phased Arrays of Transducers
Physical Equivalent
Phased Array
11
The Forward Problem - Auralization
sound source positions
Auralization Algorithms
sound source signals
Listener signals
room geometry/properties
Listener positions
Listener wearing Headphones (or HSS scheme)

• In general, all inputs are a function of time
• Auralization must proceed in real-time

12
Ray-based Approaches To Auralization

• For each sound source, cast some number of rays,
  then collect rays that intersect listener
  positions
• Geometrical simplification for rectangular spaces
  and specular reflections
• Problems
• Non-specular reflections requires exponential
  growth in number of rays to simulate
• Most interesting spaces are not rectangular

13
Wave Propagation Approach
2p/2t 2p/2x 2p/2y 2p/2z

• Captures all properties except absorption
• absorption adds 1st partial terms

14
Method of Finite Differences

• Replace differentials with differences
• Solve on a regular grid
• Simple stencil computation (2D Ex. in Fx)
• Do it really fast

pdo i2,Y-1 pdo j2,X-1
workarray(m0,j,i) (.99) (
Rtemparray(j1,i)
2.0(1-2.0R)temparray(j,i)
Rtemparray(j-1,i)
Rtemparray(j,i1)
Rtemparray(j,i-1) -
workarray(m1,j,i) ) endpdo endpdo
15
How Fast is Really Fast?

• O(xyz(kf)4 / c3) stencil operations per second
  are necessary
• fmaximum frequency to be resolved
• x,y,zdimensions of simulated space
• kgrid points per wavelength (2..10 typical)
• cspeed of sound in medium
• for air, k2, f20 KHz, xyz4m, need to perform
  4.1 x 1012 stencil operations per second (30 FP
  operations each)

16
LTI Simplification

• Consider the system as LTI - Linear and
  Time-Invariant
• We can characterize an LTI system by its impulse
  response h(t)
• In particular, for this system there is an
  impulse response from each sound source i to each
  listener j h(i,j,t)
• Then for sound sources si (t), the output mj(t)
  listener j hears is mj (t) Si h(i,j,t) si(t),
  where is the convolution operator

17
LTI Complications

• Note that h(i,j) must be recomputed whenever
  space properties or signal source positions
  change
• The system is not really LTI
• Moving sound source - no Doppler effect
• Provided sound source and listener movements, and
  space property changes are slow, approximation
  should be close, though.
• Possible virtual source extension

18
Where do h(i,j,t)s come from?

• Instead of using input signals as boundary
  conditions to wave propagation simulation, use
  impulses (Dirac deltas)
• Only run simulation when an h(i,j,t) needs to be
  recomputed due to movement or change in space
  properties.

19
Exploiting a Remote Supercomputer or the Grid
20
Interactivity in the Forward Problem
sound source positions
Auralization Algorithms
sound source signals
Listener signals
room geometry/properties
Listener positions
Listener wearing headphones
21
Full Example of Virtualized Audio
other inputs
microphone signals
sound source positions
Recovery Algorithms
sound source signals
room geometry and properties
microphone positions
Human Space
Microphones
Reverse Problem
other inputs
microphone signals
sound source positions
Recovery Algorithms
Combine
sound source signals
room geometry and properties
microphone positions
Human Space
Microphones
Reverse Problem
other inputs
microphone signals
sound source positions
Recovery Algorithms
sound source signals
room geometry and properties
microphone positions
Human Space
Microphones
Reverse Problem
22
VA as a Distributed Interactive Application

• Disparate resource requirements
• Low latency audio input/output
• Massive computation requirements
• Low latency control loop with human in the loop
• Response time must be bounded
• Adaptation mechanisms
• Choice between full simulation and LTI
  simplification
• number of listeners
• Frequency limiting versus delay
• Truncation of impulse responses
• Spatial resolution of impulse response functions

23
Conclusion

• We can and should do better than the current
  state of audio
• Lots of existing research to exploit
• The basis of virtualized audio
• Trade off computation and specialized hardware
• VA is a distributed interactive application

VA forward problem currently being implemented at
Northwestern