The Future of eInteraction

1
CS 6905 Designing for Multimodality Interaction
Paradigms
Dr. Jo Lumsden NRC IIT e-Business 46 Dineen
Drive Fredericton, N.B., E3B 9W4 tel
506-444-0382 e-mail jo.lumsden_at_nrc.gc.ca
2

• what is an interaction paradigm ?

a model of how humans interact with a computer
3

• what is the most common interaction paradigm ?

the standard design of most desktop PC-based
applications
4

• what are the distinguishing features of the
  desktop design paradigm ?

input via keyboard and mouse output is typically
graphical
5

• to what extent is this paradigm multimodal ?

depends on your interpretation of multimodal!
6

• can anyone suggest alternative interaction
  paradigms ?

audio-based, audio-enhanced, gestural, etc.
7
focus on designing for mobility
exemplifies the need for multimodal interaction
8
the status quo of m-interaction (1)

• experts predict that 21st century decade of
  mobile computing
• hype gt reality Urbaczewski et al., 2003
• recent international survey
• m-commerce is difficult to use
• quality of service is low
• technology is dominating users Jarvenpaa et al.,
  2003
• users forgiving of technical limitations
  
• users ? forgiving of flawed interfaces Sarker
  Wells, 2003

9
the status quo of m-interaction (2)

• devices
• more diverse
• shrinking size weight
• ? portability but ? usability
• hard to use with one hand
• level of frustration high for mobile technology
• biggest cause low usability Venkatesh et al.,
  2003
• worsen if not paid sufficient appropriate
  attention
• difficulty of use wasted time error

10
so why research m-interaction? (1)...

• widespread acceptance of mobile devices
  essential for promise commercial benefit of
  mobility (m-commerce) to be realised
• level of acceptance will not be realised if
  users interaction experience with mobile
  technology is negative

11
so why research m-interaction? (2)...

• need to design use the right types of
  m-interaction to make m-commerce a desirable
  facility in the future
• prerequisite ensure users experience meets
  both sensory functional needs Venkatesh et
  al., 2003

12
so why research m-interaction? (3)...

• resource disparity
• successful e-commerce interface design ?
  successful m-commerce interface design
• success of m-commerce services innovative users
  uses (not necessarily technology)

13
complexities of designing for mobility (1)

• m-interaction ? e-interaction
• but, direct migration of desktop to mobile
• is common.and unsuccessful!
• issues?

• disparity in available physical resources
• mobility of users
• desktop interaction designed for stationary users
• devotion of attentional resources
• stationary users can typically devote all to
  interaction
• mobile users need to remain focused on primary
  task for safety
• complexity of context of use (see lecture 6)

14
complexities of designing for mobility (2)

• so what is the result?
• frustration
• perceived lack of usability
• abandonment of technology services

15
goals for evolving m-interaction (1)

• The great advantage the telephone possesses over
  every other form of electrical apparatus consists
  in the fact that it requires no skill to operate
  the instrument Alexander Graham Bell, 1878
• where applicable, assume no skill or training
  available
• better understand effect of mobility on
  interaction
• design to accommodate these influences
• better align m-interaction with user behaviour
  social conventions

16
goals for evolving m-interaction (2)

• use mix of techniques to overcome physical device
  limitations (e.g. screen restrictions)
• hard to design purely visual interfaces that
  accommodate users limited attention
• issues with head-mounted displays Barfield
  Caudell, 2001
• obtrusive
• hard to use in bright daylight
• occupy visual resource Geelhoed et al., 2000
• customizability flexibility

17
so what is available?
18
speech-based audio (1)

• input user issues command/input by speaking
• output synthesised or pre-recorded speech
• Beasly et al., 2002 Lai Yankelovich, 2000
• vocabularies can be
• constrained Beasly et al., 2002
• unconstrained Lai Yankelovich, 2000
• sophistication v. accuracy
• natural
• hands- and/or eyes-free interaction

19
speech-based audio (2)

• environmental issues
• ambient noise levels
• privacy
• when used for both input output, monopolises
  auditory resource
• can listen to non-speech audio while issuing
  speech-based input
• hard to listen to and interpret speech-based
  input and output simultaneously
• appropriate when used contextually in
  combination with other paradigms

20
non-speech audio (1)

• proven very effective at improving interaction on
  mobile devices Brewster, 2002 Brewster et al.,
  2003 Holland Morse, 2001
• Pirhonen et al., 2002 Sawhney Schmandt,
  2000
• maintain visual focus on environmental navigation
• language independent typically fast
• two categories
• earcons musical tones combined to convey
  meaning relative to application objects or
  activities
• auditory icons everyday sounds used to
  represent application objects or activities

21
non-speech audio (2)

• can be multidimensional
• data conveyed
• spatial location in which it is presented
• most humans good at streaming audio cues
• allows for positioning of sounds in 3D space
  around head
• users can ID direction of sound source and take
  appropriate action (e.g. select the source)

22
non-speech audio (3)

• supports eyes-free interaction
• leaves speech channel free for other use
• primarily a feedback/output mechanism
• best used in combination with other modalities

23
gestural interaction (1)

• naturally very expressive
• can be multidimensional
• 2D hand drawn Brewster et al., 2003 Pirhonen et
  al., 2002
• 3D hand generated Cohen Ludwig, 1991
• 3D head generated Brewster et al., 2003
• simple, natural gestures can be used for range of
  input on mobile devices Harrison et al., 2003
• head-based gestures already used successfully in
  applications for disabled users

24
gestural interaction (2)

• no need to look at display to interact with it
• supports eyes-free interaction
• hands-free if head gestures
• combined use of audio gesture has most
  potential

25
audio-enhanced gestural interaction (1)

• true multimodal interaction
• some examples
• Audio Windows Cohen Ludwig, 1991
• data glove used to point to items represented
  audibly in 3D space around head
• powerful creation of a rich, complex
  environment without need for visual display
• Nomadic Radio Sawhney Schmandt, 2000
• personal messaging system that uses speech and
  non-speech audio to deliver information
• planar 3D audio rep. allows users to listen to
  multiple sound streams but distinguish separate
  each (Cocktail Party Effect) spatial
  positioning conveys info about time of occurrence

26
audio-enhanced gestural interaction (2)

• more examples
• Compaq iPAQ MP3 Player Pirhonen et al., 1991
• small set of metaphorical gestures rep. control
  functions of the player drag finger over touch
  screen to enter
• end-of gesture audio feedback to confirm action
• significantly better than the standard graphical
  interface
• extended by Brewster et al. to incorporate
  dynamic audio feedback about progress of gestures
  Brewster et al., 2003
• improved accuracy awareness of accuracy of
  gestures
• simpler the audio feedback design the better
• Bullseye Menus Fiedlander et al., 2000
• menu concentric rings around cursor non-speech
  audio cue indicates when move between items
• users able to select items using only the sounds

27
audio-enhanced gestural interaction (3)

• more examples
• 3D Auditory Radial Pie Menu Brewster et al.,
  1991
• menu items displayed in 3D space around users
  head at level of the ears
• select items by nodding in direction of item
• tested different audio designs

28
(No Transcript)
29
audio-enhanced gestural interaction (5)

• demo
• Audio-Enhanced Unistroke Alphabet Lumsden
  Gammell., 2004
• run demo

30
graphical interaction

• visual or graphical displays are possible but
  difficult
• cannot simply migrate to the smaller screen
• leads to navigational problems
• look at segmentation/reduction of data
• focuscontext techniques
• fisheye lens
• etc.
• can make widgets smaller but maintain usability
  if audio feedback is added to their interaction
  properties

31
conclusions

• the future of mobile devices and therefore
  m-commerce depends on getting m-interaction right
• need to be innovative
• imaginative, combined use of mix of techniques
• users control technology not technology control
  users
• mobile technology as easy to use as the
  old-fashioned telephone!

32
references
Barfield, W., Caudell, T. (2001), Fundamentals
of Wearable Computers and Augmented Reality,
Mahwah, New Jersey Lawrence Erlbaum
Associates. Beasly, R., Farley, M., O'Reilly, J.,
Squire, L. (2002), Voice Application
Development with VoiceXML, SAM Publishing. Brewste
r, S. A. (2002), Overcoming the Lack of Screen
Space on Mobile Computers, Personal and
Ubiquitous Computing, 6(3), 188 - 205. Brewster,
S. A., Lumsden, J., Bell, M., Hall, M., Tasker,
S. (2003), Multimodal 'Eyes-Free' Interaction
Techniques for Mobile Devices, in Human Factors
in Computing Systems - CHI'2003, April 5 10, Ft
Lauderdale, USA. Cohen, M., Ludwig, L. F.
(1991), Multidimensional Audio Window Management.
International Journal of Man-Machine Studies,
34(3), 319 - 336. Fiedlander, N., Schlueter, K.,
Mantei, M. (1998), Bullseye! When Fitt's Law
Doesn't Fit, in ACM CHI'98, Los Angeles,
CA. Geelhoed, E., Falahee, M., Latham, K.
(2000), Safety and Comfort of Eyeglass Displays,
in P. Thomas H. W. Gelersen (Eds.), Handheld
and Ubiquitous Computing (pp. 236 - 247), Berlin
Springer. Harrison, B., Fishkin, K., Gujar, A.,
Mochon, C., Want, R. (1998), Squeeze me, hold
me, tilt me! An Exploration of Manipulative User
Interfaces, in ACM CHI'98, Los Angeles,
CA. Holland, S., Morse, D. R. (2001), Audio
GPS Spatial Audio Navigation With a Minimal
Attention Interface, in Mobile HCI 2001 Third
International Workshop on Human-Computer
Interaction with Mobile Devices, 10 September.
Lille, France. Jarvenpaa, S. L., Lang, K. R.,
Takeda, Y., Tuunainen, V. K. (2003), Mobile
Commerce at Crossroads, Communications of the
ACM, 46(12), 41 - 44. Lai, J., Yankelovich, N.
(2000), Conversational Speech Interfaces, in The
Human Computer Interaction Handbook (pp. 698 -
713) Lawrence Erlbaum Associates
Publishers. Pirhonen, P., Brewster, S. A.,
Holguin, C. (2002), Gestural and Audio Metaphors
as a Means of Control in Mobile Devices, in
ACM-CHI'2002, April 20 25, Minneapolis,
MN. Sarker, S., Wells, J. D. (2003),
Understanding Mobile Handheld Device Use and
Adoption, Communications of the ACM, 46(12), 35 -
40. Savidis, A., Stephanidis, C., Korte, A.,
Crispien, K., Fellbaum, C. (1996), A Generic
Direct-Manipulation 3D-Auditory Environment for
Hierarchical Navigation in Non-Visual
Interaction. in ACM ASSETS'96, Vancouver,
Canada. Sawhney, N., Schmandt, C. (2000),
Nomadic Radio Speech and Audio Interaction for
Contextual Messaging in Nomadic Environments, ACM
Transactions on Computer-Human Interaction, 7(3),
353 - 383. Urbaczewski, A., Valacich, J. S.,
Jessup, L. M. (2003), Mobile Commerce
Opportunities and Challenges, Communications of
the ACM, 46(12), 30 - 32. Venkatesh, V., Ramesh,
V., Massey, A. P. (2003), Understanding
Usability in Mobile Commerce, Communications of
the ACM, 46(12), 53 - 56.
33

• ideas discussion