The Future of eInteraction

1
CS 6905 Designing for Multimodality Multimodalit
y Guidelines
Dr. Jo Lumsden NRC IIT e-Business 46 Dineen
Drive Fredericton, N.B., E3B 9W4 tel
506-444-0382 e-mail jo.lumsden_at_nrc.gc.ca
2
relatively little guidance currently available
for designing multimodal interfaces so going
to focus this lecture on guidelines for
designing and combining earcons within a
graphical user interface i.e. audio-visual
interfaces
3
introduction

• sound suite collection of earcons each
  annotating a different element in the user
  interface
• just like a visual style should be consistent
  across the windows etc of an application (e.g.
  color schemes, border etc) so too should a sound
  suite be consistent in terms of audio style
• need to consider low- and high- level design
  goals when designing audio-feedback
• low-level role of individual earcons
• high-level characteristics the sound suite
  should possess

4
low-level design goals earcons (1)

• need to design audio representations for widgets
  the events widgets generate
• widgets
• interaction techniques that include objects (e.g.
  button) interface functionality (e.g. drag and
  drop)
• each has distinct visual appearance establish
  context
• each has distinct position within graphical
  display
• distinguish between multiple instances
• audio enhancement can extend usability by
  reinforcing context or, when users visual
  attention is not focused on the widget, by wholly
  communicating context

5
low-level design goals earcons (2)

• events
• interface messages that communicate action or
  state
• short term communicate immediate result of
  users interaction
• associated with discrete foreground tasks that
  require some degree of visual attention
• audio cues can be added to visual cues to ensure
  all events are communicated successfully
  unambiguously without overloading visual sensory
  channel
• long term communicate progress of ongoing task
• associated with secondary/background tasks which
  could ultimately be monitored audibly other than
  when visual attention is required for initiation,
  acknowledgement, abortion etc

6
low-level design goals earcons (3)

• widgets events 2D information space of the
  graphical UI
• on widget axis visual audio cues communicate
  type
• on event axis visual audio cues communicate
  status
• widget/window position (partial dimension)
  visual audio cues communicate instance

7
high-level design goals sound suite (1)

• designing audio feedback/signatures for
  individual widgets in isolation ? guarantee of
  successful UI
• when embedded in single UI, earcons from
  different widgets may interfere with each other
• sounds appear disassociated from source
• annoyance
• user fatigue
• identify set of goals for the sound suite as a
  whole

8
high-level design goals sound suite (2)

• minimize annoyance
• avoid excess intensity (volume) variations
  overall loudness of audio feedback
• main causes of audio-related annoyance
• audio feedback must keep pace with events
  visual cues
• prevent user confusion fatigue

9
high-level design goals sound suite (3)

• simplify mapping
• minimize total number of different earcons in one
  sound suite
• cluttered audio-enhanced user interfaces users
  disable sound
• achieve this by
• ensure mapping between sounds assoc.
  widget/widget behaviour simple and obvious
• ensure overall number of concurrently playing
  sounds is not excessive

10
high-level design goals sound suite (4)

• facilitate segregation
• earcons assoc. with a particular widget must
  always be perceived as emanating from that widget
• when sequences of earcons are perceived as coming
  from the one widget forms elemental association
  which speeds up recall
• Acoustic Sound Segregation users perceive
  sounds as forming coherent groups if they are
  similar proximal
• characteristics of earcons strongest criteria
  by which users group sounds
• e.g. 2 notes with same timbre more likely to be
  thought similar (grouped) than the same note
  played with different timbre
• proximity of sounds perceived along time or
  frequency axes

11
guidelines

• based on empirical studies
• practical advice on how earcons should be
  designed for use with widgets in 21/2D graphical
  UI
• grouped into 3 sets
• how humans perceive sound its impact on earcon
  design
• designing earcons for a specific widget
• combining multiple earcons

12
human perception of sound (1)

• Guideline G1
• sounds used to identify widgets must be
  absolutely distinguishable (i.e. without
  reference to comparison scale)
• earcons cannot be assigned a unique pitch or
  loudness (humans can only make relative judgments
  about these cues)
• timbre uniquely distinguishable
• amenable to the representation of events timing
  parameters
• allows for changes in musical pitch which can be
  used to create spatial cues illusion

13
human perception of sound (2)

• Guideline G2
• carefully match widget characteristics sound
  sources
• allows auditory feedback requirements of widgets
  to exploit auditory features of sounds
• e.g. annotate events with rapid onset with sound
  sources which have correspondingly sharp attack
  features
• e.g. sustained sound sources (e.g. violin
  organ) should be used to represent continuous
  event messages impactive sound sources (e.g.
  piano drum) should be used to represent
  discrete messages

14
human perception of sound (3)

• Guideline G3
• rhythmic motives can be best used to encode
  events which communicate a time-varying parameter
• pitch can also be used but is less effective
• rhythm one of the most powerful factors in
  pattern recognition

15
human perception of sound (4)

• Guideline G4
• keep earcons within narrow intensity range
• ensures that if user changes overall volume of
  the audio output on his/her computer, no sounds
  are lost and no sounds will stand out (and be
  annoying)
• suggested range
• max 20dB above background threshold
• min 10dB above background threshold

16
human perception of sound (5)

• Guideline G5
• design earcons so that they can be played at
  different tempos
• ensures earcons can keep pace with events
  regardless of users skill levels
• minimize earcon duration by
• minimizing sound duration of each note (min
  0.03sec)
• playing only begin/end components of long earcons
  during rapid manual user input
• playing earcons in parallel to speed up
  presentation for fast/experienced users

17
human perception of sound (6)

• Guideline G6
• ensure widget signatures are distinct
• when using musical timbres, remember that
  non-musicians can easily differentiate between
  the following families of timbre but their
  intra-family timbre recognition is considerably
  weaker
• piano piano, harp, guitar, celestra, and
  xylophone
• organ organ and harmonica
• wind trumpet, french horn, tuba, trombone, and
  saxophone
• woodwind clarinet, english horn, pan
  pipes,piccolo, oboe, bassoon, and flute
• strings violin, cello, and bass
• drums drums

18
human perception of sound (7)

• Guideline G7
• make the audio feedback for each individual event
  sound like a complete unit in its own right
• accentuate (i.e. play slightly louder) the first
  note (or part thereof) and elongate the last note

19
human perception of sound (8)

• Guideline G8
• ensure synchronicity between sensory modalities
• ensures perceptual binding that exists when one
  event generates stimuli in several sensory
  modalities
• audio-visual synchronicity can suffer from audio
  lead and audio lag
• audio lead more detectable than audio lag
• audio lead gt 40msec are detectable avoid leads gt
  90msecs
• audio lag gt 120msec are detectable avoid lags gt
  180msecs

20
designing specific earcons (1)

• Guideline E1
• the absence of sound where sound is expected is
  sufficient feedback to alert a user to a problem
  if the expected sound would have been generated
  as the direct result of a user action and not as
  a piece of background information
• i.e. where a user would anticipate sound upon the
  performance of an action, the absence of that
  sound is enough to alert the user to a problem
• unsure whether this will always hold true if
  multiple audio sources are playing in the same UI

21
designing specific earcons (2)

• Guideline E2
• the absence of a discrete sound which is not the
  result of a direct user action (e.g. a piece of
  discrete background information) may not be
  sufficient feedback to alert a user to a problem
• i.e. where a user does not anticipate auditory
  feedback because he/she has not taken some direct
  action, the absence of sound is unlikely to be
  noticed

22
designing specific earcons (3)

• Guideline E3
• rank the sounds used to inform users of events
  according to importance
• when it is not possible to play all sounds, it is
  possible to identify and only play the most
  appropriate

23
designing specific earcons (4)

• Guideline E4
• do not only map sounds to events directly related
  to user actions also map them to changes in the
  system model
• proven usefulness of sounds to indicate status of
  user interaction
• equally useful to map sounds to events that occur
  in the data model as a result of user interaction
• sounds might be alterations to interaction sounds
  or completely different

24
designing specific earcons (5)

• Guideline E5
• limit the number of different sounds used
• carefully closely analyze task/interaction
  requirements rather than naïvely mapping
  different sound to each different event
• reuse event signatures across widgets to minimize
  total number of mappings users must learn
  reinforces the meaning of each
• where possible, use the same (at least similar)
  audio feedback across different widget types
  when conveying the same information
• too many sounds will confuse users
• naïve solution is not particularly scalable

25
designing specific earcons (6)

• Guideline E6
• ensure all sounds provide useful information that
  users cannot adequately obtain from other, less
  intrusive sources
• if sounds are used to provide information that
  the users are not interested in or can access
  easily from other sources annoying due to
  intrusive nature

26
designing specific earcons (7)

• Guideline E7
• blend sounds together rather than play them in
  isolation
• creating a cohesive audio ecology ensures that
  audio feedback is less intrusive
• intrusive sounds are desirable sometimesbut not
  for background sounds

27
designing specific earcons (8)

• Guideline E8
• use instruments and rhythm in way analogous to
  structure used in music when creating complex
  earcons

28
designing specific earcons (9)

• Guideline E9
• avoid using gt 6 notes per second when using note
  repetition as means to convey information
• users can find more than 6 notes per second hard
  to differentiate

29
designing specific earcons (10)

• Guideline E10
• ensure speech feedback is designed so it can keep
  pace with rate of user interaction when including
  it in earcon design
• also, enable users to silence the speech at any
  time

30
combining multiple earcons (1)

• Guideline C1
• if several sounds are playing simultaneously, the
  absence of any one sound may not be sufficient to
  alert a user to a problem
• especially true when, under these circumstances,
  the user does not anticipate audio feedback
  because he/she has not taken some direct action
  (see E1 and E2)

31
combining multiple earcons (2)

• Guideline C2
• if audio-enhanced widgets are to be included in a
  UI such that their feedback may be played
  simultaneously, prioritize their associated
  earcons
• enables, where necessary, only the most important
  feedback to be played
• linked to E3

32
combining multiple earcons (3)

• Guideline C3
• moderate the feedback of audio-enhanced widgets
  when they are included within the same UI
• prioritize the functions of the widgets and
  moderate the intensity of feedback of each widget
  (as a whole) in accordance with this ranking
• prioritize the earcons for each widget (see C2)
  and moderate the intensity of feedback in
  accordance with this prioritization for each
  widget as well as in relation to the importance
  of the widget as a whole within the UI (see above)

33
combining multiple earcons (4)

• Guideline C4
• moderate the audio feedback for simultaneous
  foreground and background tasks so neither masks
  out the other
• where applicable, the audio feedback for
  background tasks should be sufficiently demanding
  that it will not be missed by the user
• particularly relevant when the graphical
  representation of the background task may be
  obscured by that of the foreground task

34
combining multiple earcons (5)

• Guideline C5
• ensure the audio feedback for a background task
  is complete in its representation if the
  background activity is likely to be obscured by
  the graphical representation of the foreground
  task
• see C2 and C3
• where this is not possible if the background
  task is only represented graphically users are
  likely to miss most/all of the background activity

35
combining multiple earcons (6)

• Guideline C6
• spatialize earcons to allow users to
  differentiate multiple instances of the same
  widget type
• can prevent need to modify audio feedback design
  for each instance when used collectively within
  the same UI

36
evaluating earcon design

• new amended audio designs for widgets and the
  combined use of audio-enhanced widgets must all
  be thoroughly evaluated/tested (see lectures 7
  11)
• complex task
• especially need to examine effectiveness of
  moderation or prioritization of feedback
• bad audio design is counter-productive to
  interaction advantages presented by
  audio-enhancement
• bad/ad hoc audio design annoying and biases
  users against using audio-visual interfaces

37
discussion