Audio design for interactive systems

1
Audio design forinteractive systems

• Camille Goudeseune
• Integrated Systems Laboratory,
• Beckman Institute

2
Outline

• Sound libraries
• music instruments
• Digital Audio
• music theory
• Roles of sound
• composition

3
high-level multiplatform (linuxwindows at least)
sound libraries

• Java Sound API
• VSS
• FMOD
• SDL
• SEAL
• Housemarque

4
Java Sound API

• Painless for Java. Very basic. Avoid if you're
  ambitious. http//java.sun.com/products/java-media
  /sound/

5
VSS Virtual Sound Server

• www.isl.uiuc.edu/software/software.html
• the 500-pound gorilla if you need stuff like
  8-channel output or linux/irix/windows
  compatibility
• steep learning curve for advanced stuff

6
FMOD

• Syzygy (the Cube) uses it for sound.
• CPU-miserly
• finite but featureful API
• www.fmod.org

7
SDL

• large user community
• open source.
• Good if you want to integrate sound and graphics
  tightly.
• www.libsdl.org

8
SEAL Synthetic Audio Library

• many C compilers supported
• Wide support for hardware acceleration
• www.sonicspot.com/sealsdk/sealsdk.html

9
Housemarque

• Multichannel with certain PC soundcards
• www.s2.org/hmqaudio/

10
low-level sound libraries linux

• OSS, www.linux.org.uk/OSS/
• Most distros include the free basic version
• 30 for fancy multichannel soundcard drivers
• ALSA, www.alsa-project.org
• religiously open-source alternative
• 0.5.10 is the stable version
• 0.9.0 development version is fine too(part of
  the upcoming 2.5 linux kernel).

11
low-level sound libraries windows

• MMIO
• simple, universal
• waveOutGetDevCaps(), waveOutWrite(), ...
• DirectSound
• part of DirectX
• LPDIRECTSOUNDBUFFER, CreateSoundBuffer(), Lock(),
  Unlock(), ...
• faster but more awkward use a wrapper

12
Basics of Digital Audio

• File formats
• 16-bit or 8-bit?
• These days, 8-bit is embarrassing.Pro gear uses
  32!
• Stereo or mono?
• Panning mono is faster simpler than stereo.
• 8 kHz? 44 kHz?
• .WAV, .AIFF, .AU, .MP3
• converters sox and dozens of others

13
Basics of Digital Audio

• Two tradeoffs

14
Basics of Digital Audio

• Debugging
• Gaps in sound ? excessive CPU use
• stuttering ? CPU starvation(CPU fast enough
  but poorly scheduled)
• Different from graphics! The frames per second
  cant degrade if the CPU is taxed.

15
Basics of Digital Audio

• Debugging
• Electric-guitar distortion ? clippingToo
  quiet hissToo loud clippingJust right
  almost clipping
• For every stage in the audio pipeline,both
  software and hardware.Every place you can set
  the volume level!

16
Where to get sounds

• Buy fx music libraries
• Build record it yourself
• Build synthesize it yourself
• adjust an existing synth patch, a little or a lot
• Steal websearch
• 8-bit might still suffice while prototyping

17
What to do with the sounds,once you have them

• common roles
• alerts acknowledgements ambience
• sound vs. image
• speech vs. non-speech
• synchronization with visual events
• combining sounds
• tips for spatializing

18
Short and subtle is best!

• Graphical excellence gives to the viewer the
  greatest number of ideas in the shortest
  time with the least ink in the smallest
  space. (Edward Tufte, The Visual
  Display of Quantitative Information)
  
• This applies to alerts, acks, and ambience.
• Clutter is worse with sound no earlids!

19
Sound or image for a message?

• Use sound if
• Simple
• Short
• Standalone
• Temporal
• Demands immediate action

• Use visuals if
• Complex
• Long
• Referred to later
• Spatial
• Can deal with it later

20
Speech vs. non-speech

• Main advantage precise, simple! But...
• Carries extra connotations
• Tone of voice, imprecise word choice, can confuse
  or distract
• Non-speech doesnt interfere with conversation

21
Synchronizing to other things

• Often 100 msec is accurate enough,when passively
  observing a sonic anda visual event
• Sync with input action might needto be tighter,
  30 or even only 4 msec.

22
Moving a probe through a dataset

• Trigger sounds on button-click, or when crossing
  thresholds
• Or, play a continuous sound.
• Vector-valued data is trickier.
• Nominal data one sound per name.
• Scope of probe point, ball, shell.
• Auto-size based on probe speed.
• Granulation

23
Navigating a VR world

• Dense world
• azimuth click rate varies with turning rate
• altitude high/low beeps (rate
  varies with climb rate)
• speed vehicle-engine metaphor
• Large world quiet continuous ambiences
  localized to individual parts of the world.
• Play only the nearest two or three.

24
Combining sounds

• It can get only so loud before clipping.
• Spatial separation (panning).
• Temporal separation (one sound at a time).
• A party of drunks / a good dinner conversation.
• Frequency separation.
• Each layer gets its own tempo.
• Heavily layered techno or orchestral music.

25
Spatialized sound

• Steady tones are worst.Bird chirps are best
  (they should know).
• wide frequency band, complex attack
• Loudspeaker array
• Headphones with HRTF
• Motion-tracked headphones
• In the real world you move your head slightly to
  tell where a sound comes from.