Composition of Sound with Image

1
Composition of Sound with Image        

• Week 10
• IAT-204, Spring 2007SFU Surrey

2
Todays topics

• Digitization of Audio
• Some Basics of MSP
• Audio Control of Video
• Writing Audio Signal into Video

3
Learning Objectives

• Explain how digital audio works (esp. adc, dac,
  and Nyquist theorem)
• Understand how the signal network function and
  basic objects of MSP
• Elaborate on how to detect events in the sound
• Understand how to grab the soundtrack from a
  QuickTime movie using jit.qt.movie/spigot
• Elaborate on how to write MSP audio signal into a
  Jitter matrix using jit.poke

4
Digitization of Audio
5
Digital Representation of Sound

• Sampling and quantizing a sound wave
• take discrete samples of the sound waves
  instantaneous amplitude
• store that information
• reproduce those amplitudes at the same rate to
  create the illusion of a continuous wave

6
Devices for Digitization

• ADC
• converts the sampled voltages into discrete
  numbers quantization
• DAC
• converts the numbers to voltages at the same rate
  at which they were recorded

How can the quality of the original audio be
reproduced faithfully?
7
Nyquist Theorem

• Sample rate at least twice as the highest
  frequency of the analog wave
• E.g. 22,050 Hz ? 44,100 samples/s
• Computer can only accurately represent
  frequencies up to half the sampling rate
• E.g. 16,000 samples/s ? up to 8,000 Hz
• Nyquist rate half of the sampling rate

How do you deal with frequencies in the sound
higher than Nyquist rate?
8
Low- Pass Filter

• before ADC
• low- pass filter filter out any frequencies above
  Nyquist rate
• after DAC
• samples may contain spurious high frequencies
  created by the sample process ? low- pass filter
  to filter out

9
Some Basics of MSP

• Difference between MSP and Max
• In Max, patch codes allow messages to be sent at
  a specific moment in response to
• user action (mouse click, key stroke, )
• events scheduled to occur (by metro, delay, ...)
• In MSP, patch codes establish a signal network
• calculate the audio info at any particular instant

10
Difference between MSP and Max (contd)

• MSP signal network ? patch that runs at a faster
  (audio) rate than Max
• In Max, scheduling is based on control rate
• commonly 1000 times/s
• In MSP, each object calculates all the numbers
  at
• audio rate ? 44,100 times/s

11
Some Basic MSP Objects

• cycle cosine waveform oscillator ? generating
  signals at the frequency specified

• phasor sawtooth wave oscillator ? fade-in effect

• dac audio output, which can be turned on/off
  by messages like start/stop, 1/0

• Signal multiply operator
• can multiply a signal with another signal or a
  constant

12
Objects Linking Max and MSP

• line ? smooth out a sudden change in amplitude

• Objects like sig, line ? inlets take Max
  messages, but outlets connect to the signal
  network
• Objects like snapshot ? behave conversely

• sig converts a constant (float/int) to a signal

• snapshot outputs the current value from an
  input signal when it receives a bang

13
Objects for Audio Input Play

• startwindow ? turns audio on only in the Patcher
  window that contains it (different from start)

• adc receives signal from the audio input
  jacks, and sends the incoming signal out its
  outlets

• selector switch among several input signals
  based on the input to its left inlet
• 1 - the 2nd inlet
• 2 the 3rd inlet

• sfplay play opened audio files. When playing
  done (or stopped)? sends a bang out its right
  outlet

14
Track Amplitude of MSP Audio Signal

• To track the sound's amplitude for use as control
  data in Max
• snapshot ? the instantaneous amplitude of the
  sound
• avg ? the average magnitude of the signal since
  the last time it was checked
• peakamp ? obtain the peak magnitude of the
  signal since the last time it was checked

15
Amplitude Envelope

• Tell us about the general evolution of the
  loudness of the sound over time

• Amplitude envelope the shape of the overall
  change in amplitude

• The portion increasing from silence to a peak
  amplitude ? attack of the sound

16
Track Peak Amplitude

• You can trigger peakamp to report a peak by
• a bang in the left inlet,
• at regular intervals setting by a number in
• the right inlet

17
Using Decibels

• We actually perceive the intensity of a sound as
  a function of its relative level in decibels (dB)

A0 - a reference amplitude A - the amplitude
being measured

• takes amplitude and converts it to dB ?
• 1 0 dB
• lt1 lt 0 dB

18
Audio Event Detection

• Audio Events
• notes in a piece of music, words in spoken text,
  etc
• need to detect
• when the amplitude passes certain threshold, and
  keeps increasing till a turning point ?
  signifying the attack of the event
• when the sound has gone below the threshold for a
  sufficient time ? consider the event to be over

19
Audio Event Detection (2)

• Min. Note Duration ? a time to wait before
  looking for a note-off level - level that goes
  below the threshold

• Done in detectevent subpatch -gt main patch
  provides three parameters for it

• Min. Off Time ? the amount of time that the
  level must remain below the threshold for the
  note to be considered ended

• Note-on Threshold ? above which the sound
  signifies an event or note

20
(No Transcript)
21
Using MSP Audio in a Jitter Matrix

• Copy an MSP audio signal into a Jitter matrix
  using jit.poke
• Grab soundtrack from a QuickTime movie using
• jit.qt.movie ? Sound Output Component (_at_soc ltsoc
  namegt)
• spigot ? ltsoc namegt

22
Grab Soundtrack From Movie

• set named soc for jit.qt.movie object ? specify
  a sound output component that can acquire the
  soundtrack of the movie

• spigot object with attribute of the same name
  as the soc ? allow you to access the audio signal
  in MSP

23
Some Notes on spigot

• spigot object has two outlets ? the left and
  right audio channels of the movie soundtrack
• You can use as many spigot objects as you like
  to grab the audio from multiple QuickTime movies
• But one spigot object per sound output component

24
send/receive tapin/tapout

• The send object sends its input signal to all
  receive objects that share its name

• tapin signal is written into a delay line that
  can be read by a tapout object
• argument max delay time in ms default 100
• tapout signal passing through it will
  experience continuous delay

• The three audio signals ? received by named
  receive objects ? summed into the gain object

25
Write Audio Signal into Jitter Matrix

• This named jit.matrix object ? portray an
  oscilloscope view of the movie soundtrack, with
  the dry and two delayed signals ? red, green,
  blue

26
Write Audio Signal into Jitter Matrix (2)

• scope Jitter matrix is generated by three
  jit.poke objects ? write MSP audio signals into
  cells in the matrix
• arg 2 number of dims in the matrix
• arg 3 Plane number

27

• rightmost inlet sets the vertical (dim 1)
  coordinate ? based on the soundtrack signal

• middle inlet sets the horizontal (dim 0)
  coordinate ? based on signals generated by a
  phasor object

• 1st inlet ? provide the value to write into the
  matrix cell specified by the other two inlets,
  which take signals to specify the cell location

28
Composition of both Matrices

• you only see the movie image appear in those
  cells where the jit.poke objects have traced the
  waveform

• Image from the jit.qt.movie and oscilloscope
  drawn by jit.poke ? composited into a final
  matrix by the jit.op

29
Summary of Todays Topics

• How digital sound works (esp. adc, dac, and
  Nyquist theorem)
• The signal network and basic objects of MSP
• Detect discrete events in the sound
• Grab the soundtrack from a QuickTime movie using
  jit.qt.movie and spigot
• Write MSP audio signal into a Jitter matrix using
  jit.poke