Audio

1
Audio

• Sound
• Audio synthesis

2
Audio Perception

• Sound Pressure waves in frequencies between
  50Herz - 22,000Herz
• Lower frequencies more felt by the whole body
  than heard
• Sounds can be perceived as coming from a location
• Not terribly accurate
• Cone of confusion

3
3D Audio Perception

• Cone of confusion
• Cone-shaped zones in front of and behind head
• 3D Audio cues
• Interaural Time Difference
• Interaural Intensity Difference
• Pinnae filtering
• Body filtering

4
3D Audio Perception

• Goal for 3D sound is Spatialization
• The sense that the
• Sound originates outside your head
• Sound has a direction
• Interaural Time Difference
• The more extremely left or right, the greater the
  difference
• Time difference lt 5ms

5
3D Audio Perception

• Interaural Intensity Difference
• Head absorbs and reflects sound energy
• The first ear to get sound gets loudest sound
• Head Shadow
• Cone of confusion
• Time difference too small to detect
• Intensity is similar in both ears

6
Pinnae Filtering

• Outer ear (Pinna) shape filters sound based on
  its direction
• Childhood learning trains brain to associate
  filtering effects with direction
• Unique per person
• Record directional white noise
• Microphone in ear canal
• Sounds from speakers located about head

7
Pinnae Filtering

• A Generic Pinna can be simulated
• Record directional white noise received by dummy
  head
• Body filtering
• Reflection and absorption
• Included in Pinna model

8
Head-Related Transfer Function

• HRTF is the general term
• Transformation of real sound to spatialized
  sound
• Best delivered by earphones

9
Environmental Effects

• Sound exists in an environment
• Bounces off objects
• Is absorbed by objects
• Simple effects
• Reverb Simulate the environmental echo
• Echo is the attenuated signal
• Gives a richer room-like feeling
• Larger room has longer time delay

10
Audio signals

• Nyquist limit
• Must sample signal at least twice as frequently
  as highest reproducible frequency
• Audio 44.1KHz (CD)
• 22KHz
• 11KHz (Analog AM Radio)
• 8KHz (Telephone)

11
Audio - Digital Implications

• 44,100 Hz
• 44,100 Samples/sec
• 16-bit samples
• Stereo
• 172KBytes/sec
• Specialized hardware - Sound card

12
Reproduction

• Sampling
• Record sounds by whatever means
• Synthesis
• Analog Synthesis
• FM Synthesis
• Wavetable Synthesis

13
Control

• MIDI - Musical Instrument Digital Interface
• Developed to control music synthesizers
• Details of synthesis are controlled by
  synthesizer
• MIDI data
• Sets synthesis parameters
• Sets music sequence

14
Synthesis

• Analog Synthesis
• Simple sum of frequencies
• Select from a palette of source frequencies
• Sum of frequencies is filtered
• FM
• One frequency is controlled by another
• Wavetable
• Digitize audio signals

15
Analog Synthesis

• Fouriers observation
• Any signal can be created as the sum of sine
  waves
• Square wave Infinite sum
• f 2f 4f
• Synthesizer
• Collection of oscillators

16
Frequency Domain

• A perfectly periodic signal plotted in the
  frequency domain

(Time domain plot)
17
Spectrum

• Spectrum represents the set of frequencies
  present in the signal

18
Filters

• Eliminate part of the signal by removing certain
  frequencies
• Analog filters dont have these square response
  shapes
• Band pass
• Bandwidth

19
FM Synthesis

• Modulate the frequency of a wave
• Carrier frequency is modulated by Modulator
  signal

20
FM Synthesis for Synthesizers

• The greater the Modulator amplitude, the greater
  the Carrier frequency variation
• Higher Carrier bandwidth
• DX Carrier and Modulator are musically-tuned
  frequency
• Depends on the note you are playing
• Controls the harmonic content of a note

21
Wavetable Synthesis

• Collect a sample of the real sound
• Issues
• Reduce memory load by looping sample
• Shift pitch instead of sampling each individual
  note
• Apply interpolation techniques to make pitch
  shifting work right

22
Audio Path
Reverb, Environment. Spatialization
Envelope Loudness Control
Mixing/ Combination
23
Wavetable Synthesis Example

• Leyanda (Guitar)
• Leyanda (CDShaw)

24
Interactive Sound

• Goal
• Want to enhance the interactive experience
• Give the user a sense of presence
• Add to the emotional content of the game
• Make it more fun

25
Interactive Sound

• Music
• Sound effects
• Noises
• Commentary - Sports
• Narrative
• Conversations

26
Interactive Problems

• Regular music composition has
• Beginning
• Middle
• End
• Interactive user control makes this difficult
• Some genres have this structure

27
Interactive Music

• Game genres with order
• Sports
• Racing
• Fighting
• Semi-Ordered
• Puzzle
• Adventure

28
Music Genres

• The Infinite Loop
• Theme and variation plays forever
• Pomp Circumstance
• Diablo
• Problems
• 30 second piece repeated over 6 hours!
• 720 repetitions!
• Diablo example 12 Repetitions/hour

29
Repetition Solutions

• Make the Dominant theme hard to find
• No catchy theme!
• Create a variety of textures
• Make only transitions stand out
• Where repetition is small
• Dont repeat musical phrases

30
Music Strategies

• Play Win or Lose music
• Music must be long enough to be meaningful
• Music may be so long that the game situation
  changes before completion
• Very short music makes little sense
• Interrupt current music
• Sounds jarring

31
Modules

• Modular chunks
• Each segment of the game plays independently of
  others
• Some thematic relation
• Disjointed

32
Music Strategies

• Compose many themes in parallel
• Switch between themes
• Connect modular components together

33
Analogy Parallel Trains

• N trains of music running in parallel
• Each train serves an emotional purpose
• Train A Calm
• Train B Rising Excitement
• Train C Climactic moments
• Train D Falling Excitement
• Generally, Train A would be most commonly played

34
Within Each Musical Train

• Each Car contains a few bars of music
• Switch between trains when a Car is complete
• Dont switch in the middle of a Car
• Simple version
• Each musical phrase ends on last bar of Car
• Complex
• Notes at end are carried over to next

35
Parallel Trains
36
Parallel Trains Shuffle Cars

• Shuffle cars
• Instead of playing cars in order
• Problem Random cars sound like random radio
  tuning
• Must determine
• Appropriate car pairings
• Reasonable paths

37
Repetition In Trains

• Use repeated phrases carefully
• Maybe use a statistical tool to analyze paths
• Bayesian nets
• Endings
• Use transitions as an opportunity to End
• Use next Car to Begin new series

38
Composing a Train

• Create a piece with all layers
• Piece can probably survive a layer or two removed
• Variation piece with layer removed
• Be careful with prominent instruments
• Fallback Use instruments with similar acoustical
  properties
• Piano, Organ, Woodwinds
• No Trumpets, Drums or Screaming guitar!