Affective Effects

1
Affective Effects

• Ambient Display of Facial Affect using Musical
  Effects

Luke Barrington 10th October 2005
2
Outline

• Ambient Technology
• Musical Displays
• Facial Affect Model
• Effect Evaluation
• System Prototype

3
Acknowledgements

• Dr. Michael Lyons Dominique Diegmann
• Intelligent Robotics and Communication
  Laboratories
• NSF / JSPS East Asia and Pacific Summer
  Institutes (EAPSI) program

4
Ambient Technology

• Information overload places demands on our
  attentional resources.
• Professional, home community roles intersect
  with each other and distract.
• Ambient or calm technologies seek to present
  information in a low stress fashion in the
  periphery
• salient data is brought to your attention
• insignificant details dont register

5
Keeping Calm
6
Ambient Technology at Work
The Dangling String
Heat Sink
Prada RFID Closet
7
Musical Displays

• Music is a natural and flexible medium for an
  ambient display.
• It does not interfere with and often enhances
  unrelated activities
• working, relaxing, cleaning, chatting, driving,
  exercising,
• Music is complex enough to provide a display
  substrate with considerable bandwidth.
• The relation of music to emotions suggests that
  it may be well-suited for communicating
  information about human affect.

8
Related Work

• Xerox PARC Audio Aura Mynatt et. al. 1998
• Weakly Intrusive Ambient Soundscape (WISP)
  Kilander Lonnqvist 2002
• Adding phrases to a musical piece Butz Jung
  2003
• Limitation all these systems use pre-selected
  sonic components.
• To overcome this we use audio effects, applied to
  music of the users choice, to encode the
  information to be displayed.

9
System Design
10
Project Motivation

• Care of a dependent elder watching television
• Other ATR projects (Intention Detection,
  MemoryScape)
• More applications
• Monitor a classroom
• Observe a baby
• Watch a pet

11
Demo Design
12
Feature Extraction

• Input is from a web camera mounted above TV 15
  frames / second
• Face detection algorithm from Viola Jones
  2001
• Use optic flow in detected area to calculate
  motion in 7 facial regions.
• 3 binary features
• face detected 0,1
• rigid motion of the head 0,1
• non-rigid motion of facial features 0,1

13
Affective State

• Russel 2003 proposes an alternative to the 7
  emotions Core Affect.
• 2 dimensions Activation and Pleasure.
• Pleasure is more difficult to quantify
  automatically.
• Discretize activation into 3 levels
• agitated
• normal
• relaxed
• 4th possible state is absent.

14
Activation Levels

• The discrete binary signals from the vision
  system are converted into activation levels,
  evaluated at a constant interval T ( 0.5s)
• where n number of detections in interval
  t-T, t
• d 0.05 is an accumulator constant
• If the presence level falls below a threshold,
  the absent state is triggered indicating no face
  has been detected for a set interval (15 seconds).

15
Affective State Model

• Given presence, affective state is determined by
  the rigid and non-rigid motion levels which vary
  between 0, 1.
• Threshold levels for each activation value
  determine the source's location in 2D state
  space
• While these indicators could be used to
  continuously vary musical display parameters, for
  simplicity we choose a discrete model.

16
Musical Effects

• Rather than creating new music, our musical
  display seeks to modify existing music in a
  meaningful way akin to a DJ or producer.
• We modulate the music using effects filters,
  reverb, delay, time shifts,
• In designing the effects, we take inspiration
  from existing musical genres

17
Beat Tracking

• Synchronizing effect onsets and durations with
  the beat (phase) and rhythm (frequency) of the
  music makes the effected output sound more
  musical.
• Beat synchronization also allows finer control of
  the sparseness of the display.
• We used Jehan 2004s beat tracking algorithm
• Frequency-domain derivatives of the music signal
  excite a bank of tuned resonators (comb filters).
• The resonator with frequency matching the
  periodic modulation of the music phase-locks.
• The resonator with the maximum response
  corresponds to the tempo and the peak phase
  points correspond to the beat onsets.

18
Musical Effects

• The affect model and music effects processing are
  implemented in Max/MSP.
• A music file is loaded from a playlist and,
  depending on the affective state, is routed
  through a particular musical effect patch.
• Music with effects is sent to the listeners
  speakers.
• The effects used to represent each affective
  state are selectable according to user
  preference.
• Patches are synchronized to a master buffer
  position to allow smooth switching between
  effects that depend on the current playback
  position in the song (e.g. skips, rewind).

19
Musical Effects Description
20
Example Code
21
Musical Effects Evaluation

• A user study had 10 subjects listen to 20 second
  music clips.
• Subject hear the original followed by 14
  effect-modified versions in random order.
• Clips from 3 different songs, representing Bossa
  Nova, Classical (solo piano), and Electronica
  musical genres were rated for
• Activity Level (1relaxed, 2normal, 3agitated)
• Effect awareness (1No effect, 2Just noticeable,
  3Detectable, 4 Obvious, 5Dominant)
• Enjoyment (1Very pleasant, 2Pleasant,
  3Neutral, 4Unpleasant, 5Very unpleasant)

22
Effect Ratings
State 1 (relaxed) 3 (agitated) Awareness 1
(obvious) 5 (ambient) Unpleasantness 1 (nice)
5 (nasty)
23
Correlations
Responses average over songs
r 0.96
r 0.89
r 0.76
24
Observations

• Some effects beat the overall trends by
  maintaining desirable levels of pleasure and
  ambience for relatively agitated states.
• These findings guided the choice of effects used
  in the prototype
• low-pass filter (LP), dub effect plus low pass
  filter (dub LP), or filter sweep (FS) were chosen
  to represent the relaxed state
• high-pass filtered dub (dub HP) or rewind (RW)
  effects represent the agitated state
• absent state was simply indicated by a gradual
  fade-out of the volume.
• The correlation of effect awareness and
  displeasure strongly supports the notion that
  ambience enhances the pleasure of an affective
  information display.

25

• Want to see the demo at work?
• Want to hear the effects output?
• Want to ask some questions?