Creating Compelling Audio Content Using DirectX8 Audio

1
Creating Compelling Audio Content Using DirectX8
Audio

• Phil Scott Nvidia Developer Relations
• pscott_at_nvidia.com

2
Agenda

• Choosing the API
• Expectations
• Voices
• Memory, Quart into a pint pot
• Soundscapes
• Sounding better

3
Why are Nvidia doing an audio talk?

• nForce, platform processing architecture
• Total DirectX8 Audio Acceleration
• Fetching our graphics expertise to audio
• Great audio will complement great visuals
• Not our first audio processor remember NV1?

4
Making the right decisions

• Planning the audio at an early stage
• Audio planning decisions are as important as the
  ones made in design, graphics and networking
  architecture
• Can be a major gameplay element
• Look at Thief for a fine example
• Enhance the whole experience
• Half-Life, Undying
• Good decisions will help performance

5
Choosing the API for the task

• Involve the team, each member has their own
  agenda
• Audio engineer
• Designer
• Musician
• Audio programmer
• Rest of the programming team
• Producer

6
Choosing. II

• Every game has different requirements
• Game type
• Speed of response
• Evolving audio
• Memory footprint
• CPU power
• Target platform / Multi-platform

7
Choosing III

• Break it down, into smaller components
• Music interactive or canned?
• DirectMusic or CD Redbook audio
• 2D/3D sound effects
• DirectSound
• Soundscape modeling
• DirectMusic w/scripting or DirectSound
• Cutscene sounds
• DirectMusic

8
ChoosingIV

• The Direct audio path
• DirectX8 Audio contains a great solutions for all
  of these areas
• Hardware acceleration
• High quality
• Loads of flexibility
• Low latency
• 2D Mono/Stereo
• 3D virtualization

9
The API cut

• Both DirectSound and DirectMusic are very capable
  of doing the work
• DirectSound
• Low Latency a handful of milliseconds response
• Direct control over the audio buffers
• DirectMusic
• Branching audio
• Script driven
• Native support for compressed formats
• High level script API
• Less code at the highest level

10
You arent just tied to one or the other either

• Create the DirectSound8 object.
• DirectSoundCreate8(guid_SelectedGuid,lpDS,NULL)
• Using that lpDS object to create the DirectMusic
  Performance
• Hr CoCreateInstance( CLSID_DirectMusicPerformanc
  e,
• NULL, CLSCTX_INPROC,IID_IDirectMusicPerformance8
  ,
• (void)lpPerformance )
• Hr lpPerformanceInitAudio(NULL,lpDS,NULL,
• DMUS_APATH_DYNAMIC_3D, DMUS_AUDIOF_ALL,NULL )
• Now you are enabled for both

11
Expectations from hardwareI

• Older Hardware
• 16 mono or 8 stereo voices
• Most of the mixing done on the host CPU
• Bandwidth or memory limited
• Slower, lower quality 3D HRTF encoding usually
  done on the host CPU

12
Expectations II

• Modern Hardware
• Full DirectX8 Audiopath hardware acceleration
• Up to 256 stereo voices
• 64 3D Full HRTF virtualized voices
• 5.1 AC3 output, both encoded real time or
  streamed
• Vast bandwidth
• Hardware voice decompression
• I3DL2 reverb, with Occlusion and Obstruction
  processing
• Independent effects processing on per voice

13
2D Voices

• Most commonly used
• Supported on all hardware
• Simple in concept
• Very low processing overhead
• Mono or Stereo
• No math needed
• Plenty of good examples of this I dont need to
  show you

14
3D HRTF encoded voices

• Supported by all hardware
• Good hardware has no CPU hit
• Choice of HRTF encoding algorithm
• More complicated in concept
• Involves math
• Results are excellent

15
Creating a 3D voice

• Creating 3D voices is easy! Same as 2d voices
  except
• Add DSBCAPS_CTRL3D to the creation flags
• Create the voice with CreateSoundBuffer()
• Choose a HRTF encoding algorithm
• DS3DALG_HRTF_FULL
• DS3DALG_HRTF_LIGHT
• DS3DALG_NO_VIRTUALIZATION (Pan Volume)
• QueryInterface the object to be a
  LPDIRECTSOUNDBUFFER8
• QueryInterface to be a LPDIRECTSOUND3DBUFFER8

16
You also need a listener

• This is the concept of the audio camera
• This is easy to create also
• Create a sound buffer with DSBCAPS_CTRL3DDSBCAPS_
  PRIMARY
• Query the interface with the GUID
  IID_IDirectSound3DListener8
• Use GetAllParameters() and SetAllParameters() to
  modify listener position, orientation etc.

17
How much math?

• Your 3D graphics guru has all this vector math
  available Ask for it and dont replicate work
• view vectors and positions are readily
  available
• Listening cones are at your discretion
• Update your voices
• Relative to listener, DS3DMODE_HEADRELATIVE or
• Listener uses view vectors and the API does the
  transformation into listener space
  DS3DMODE_NORMAL

18
Advanced voices

• 5.1 AC3 Encoded voices
• 6 discreet pre-mixed voices
• Simple to implement
• Examine the WAVEFORMATEX part of the WAVE header
• A WAVEFORMATEXTENSIBLE stream will contain !0 in
  cbSize (this is the number of bytes extra in
  the header
• Create the voice as normal
• If it fails to create, cascade back to
• WFE Quad encoded (front pair Rear pair) HRTF
  pre-encoded
• Standard Stereo HRTF pre-encoded

19
Advanced Voices II

• ADPCM decoding is also available this way in some
  hardware
• Use WAVEFORMATEXTENSIBLE again to drive this
• Again the right header information can be pulled
  from the WAVE FILE, no need to work it out.
• 41 compression ratio
• Good Quality compression
• CAVEAT
• On NVIDIA nForce hardware the ADPCM file must be
  compressed with our compression tool

20
Advanced voicesIII

• Fallback time! Older hardware is limited!
• Use ADPCM as a medium for higher fidelity, rather
  than more samples, for example samples_at_ 44-48khz
• Try to create the voice
• If it works, Great! Its on good hardware.
• If it fails,
• Decode the voice to ram
• Re-sample the voice
• Create a normal sound buffer with the re-sampled,
  lower resolution voice
• NVIDIA will supply library code for decompression
  soon!

21
Further reading

• Further reading on Multichannel formats
• "Enhanced Audio Formats for Multi-Channel
  Configurations and High-Bit Resolution",
  http//www.microsoft.com/hwdev/audio/multichaud.ht
  m

22
Scalability

• Have a sensible cut back scheme for 3D voices
• Use ADPCM!-Lower grade non compressed voices
• Fallbacks for 5.1-Virtualized Stereo
• Use DS3DALG_NO_VIRTUALIZATION for lower grade
  hardware. Pan and Volume still sounds good, and
  you only need 1 codepath

23
Just do this!!!

• Configure your speakers. Its now very important
  to the end user!
• Use SetSpeakerConfig(DWORD dwSpeakerConfig)
• DSSPEAKER_5POINT1
• DSSPEAKER_SURROUND
• DSSPEAKER_QUAD
• DSSPEAKER_STEREO
• DSSPEAKER_MONO
• DSSPEAKER_HEADPHONE
• NO EXCUSES! 1 line of code to enable 5.1 encoding

24
Pre-Processing

• An interseting idea to save CPU/Bandwidth
• Pre-spacialize or Pre-Virtualize sounds
• Especially effective to provide a vast soundscape
• Creates a nice hole for the 3d sounds to live in
• Saves the CPU on all systems, and it can be HRTF
  FULL encoded, or another spacialization algorithm
• Bass frequencies are more or less directionless.
  Pre-encoding these may save the CPU or free up a
  3d voice on the APU

25
Memory expectations

• Todays games need high quality samples
• Usually 22khz _at_ 16bit 44k per second of audio
• In the future we expect this to go higher 44khz
  or more, and maybe even 24bit audio
• The budget that most audio programmers/engineers
  get is usually limited
• Audio is sometimes perceived as bad ROI in memory
  terms

26
MemoryII

• Multiple instantiation of sounds is currently a
  problem with DirectSound8
• The obvious way is to use DuplicateSoundBuffer
• Its advised to stay away from this, since you
  cant derive any new interfaces from a duplicate
  buffer
• May be software mixed
• DuplicateSoundBuffer 3D in the past have not
  mixed well

27
Memory III

• Just create multiple copies of the same sound
  buffer
• Great, if youve got the memory for it but I bet
  youre over budget now!
• Create and Destroy sound buffers on request
• This is not a quick historically
• Will fragment memory and loose performance for
  the whole game
• Not good for twitch sounds, such as gunfire
• Get clever

28
Memory IVGetting clever!

• For one shot sounds, just create them as normal
• For multiple instantiated sounds,
• create container voices a concept of channels
• Load the wave, but not into the voice, just keep
  it as data
• Stream it into the voice, filling one half of the
  voice while the other half is playing.
• Typically you will need about a 1 second voice
  just in case of starvation
• The data doesnt need to be in memory, you
  could be streaming it from HD or CD for one shot
  sounds

29
MemoryConclusion

• Be aware that you will probably not have the
  memory budget for what you want
• Modern hardware will make good quality samples
  sound great, but cant fix low quality samples so
  quality is needed
• Get Clever! Only keep in memory what is twitch
  sounds if you need to. If the game is scripted or
  events are predictable Fetch from HD or CD on
  demand
• Cache some of these demand sounds if you have to

30
Soundscapes

• Soundscapes are what I call ambient sounds, the
  movie industry calls this FOLEY
• Soundscapes should,
• Set the mood for the game
• Reflect the overall look of the screen to
  involve, scare or excite the player
• Not be triggered sounds such as alien speech,
  footsteps, gunfire, tank sounds etc
• Movies do this all the time, the classic example
  ALIEN

31
SoundscapesII

• Drive them with DirectMusic scripting
• Lets the audio engineer mix the sounds just
  right
• Can take cues from game code, monitor game
  variables
• Imagine changing the mood of the game based on
  how well you do, or whether you are about to be
  attacked
• Sounds can be 3D
• Scary whispers that move past youre head
• DirectMusic handles decompression of sounds and
  also can do the streaming for you

32
Adding FX to voices

• DirectSound has the FX interface for adding
  effects such as distortion, flange, chorus and
  I3DL2 Reverb
• Add FX control by using DSBCAPS_CTRLFX to youre
  buffer creation flags
• GOTCHA! At the moment DSBCAPS_CTRLFX is mutually
  exclusive with DSBCAPS_CTRLFREQUENCY
• OUCH! No car engines in tunnels cant use the
  reverb engine! How can I do this then???

33
I3DL2 Reverb

• Use PropertySets!
• Create a single buffer object
• QueryInterface for a properyset
• Check the caps what each properyset supports
• All hardware that supports hardware reverb
  supports I3DL2
• www.iasig.org is the place to get the headers
• Nvidia will post an environmental audio library
  soon

34
Occlusion and Obstruction

• I3DL2 is how you add occlusion and obstruction to
  your games, quickly and easily
• OCCLUSION
• Dont even try to represent the geometry that the
  graphics are using, youll just suffer
• Concider the room as a series of portals, if the
  portal is open its audible This will be
  available from the graphics programmer
• OBSTRUCTION
• Quick and dirty bounding boxes will do just fine
• Again graphics programmer to the rescue!

35
Making it sound good!

• The trend, is for the game to throw a load of
  sound effects at the audio engine
• You need to gate these, otherwise, youll be in
  phase cancellation hell
• Instead, gate sounds by group and only allow 2
  or 3 to be playing at a time
• While this the sound is less congested on the
  ear, it also improves performance
• You will not be the favorite person on the team
  if the audio drops the fps by a significant amount

36
Common sense

• Do I need to say it
• 3D sounds can only be MONO sourced
• DirectMusic can do a lot more besides music
• 3D voices are a valuable resource, dont do 3d
  voices where 2d works voice overs
• Dont hard code anything into your engine, keep
  it in script or config files
• Code your voices as 3D and if the hardware is
  limited, use NOVIRTUALIZATION, dont try and
  write a different path

37
Conclusion

• Modern hardware is extremely powerful
• Enable features that only take 1 line of code
• Use high fidelity samples, as we move on PCs are
  going to be plugged into very good speakers. 5.1
  decoders, so this is very important
• Use teamwork, to solve problems. Your graphics
  programmers will have all the 3D math, occlusion,
  visibility determination already calculated,
  dont do work that you dont need to do!

38
Question time!

• ?
• Questions
• Pscott_at_nvidia.com
• Phil Scott Nvidia Developer Relations