Language, graphics and gesture in a mapbased dialogue

1
Language, graphics and gesture in a map-based
dialogue

• John Lee
• Human Communication Research Centre
• University of Edinburgh

2
Gesture and Languagethe emerging consensus?

• Gesture and language are integrated parts of a
  multimodal communication system
• Gesture is (typically) neither epiphenomenal,
  redundant nor unrelated, but complementary
• Information provided only in gesture is often
  essential to understanding a communication
• Both speakers and hearers are able to exploit
  this (at least subconsciously)
• A considerable literature on these points now
  exists, cf. Goldin-Meadow, 2003.

3
Gesture and information

• Gesture naturally often conveys spatial
  information
• E.g. direction or nature of a movement mentioned
  (but not described) in speech
• Nature of gesture can depend on language
  structure
• Gesture can be of major assistance in explanation
• examples of kinship, fish traps etc. in Laos
  (Enfield 2005)
• which it achieves by allowing the use of
  spatial structures as cognitive artifacts
• But gesture in diagrammatic reasoning tasks is
  not greatly studied

4
Our study

• Collaboration between Edinburgh and Stanford
  (Jean Carletta, Tracy McLeod Barbara Tversky,
  Julie Heiser)
• Map-based task recovery of people from Stanford
  campus after a hypothetical earthquake
• Conducted in pairs video-recorded
• Two conditions co-present and remote
• Videotapes annotated using Noldus Observer and
  NITE toolkit
• Data analysis still in a fairly early stage
• emphasis here is on planning phase of the task

5
The map (or one like it)
6
The two conditions
7
Observations

• Theres a lot of gesture in the remote condition
  
• often more than in co-present, unexpectedly
  (Alibali et al 2001)
• which perhaps emphasises that even co-present
  is not face-to-face
• Very tight focus of interaction on the map itself
• Absence of iconic gestures and beat gestures
• (in both conditions)
• If beat gestures are perhaps fundamental
  elsewhere, deictic and focussing gestures may
  be in our situation

8
Observations (contd.)

• In co-present, gesture without speech is unusual
  
• except when continuing with an action already
  discussed
• as though unexplained gesture would be
  inappropriate
• and so is speech without gesture
• heavy use of deixis and ellipsis high
  multimodality
• Results in a powerful imperative to interact
• produces a close and explicit focus on
  information
• produces a faster and much more successful
  performance on the task than in remote (cf.
  Heiser et al 2004)
• Clear benefit of shared access to the map,
  mediated by shared gesture

9
Observations (contd.)

• In remote, deixis being impossible, descriptive
  phrases are frequent
• Conventions need to be agreed for identification
• e.g. place names, people numbers, etc.
• Task progress is slowed
• Lack of fine-grained feedback leads to poor
  convergence
• sketches in final stage are low quality (Heiser
  et al 2004)
• cf. MAGIC results

10
Speculation

• Role of gesture in betraying processes of thought
  (cf. Goldin-Meadow 2003) is important
• evident in remote in absence of speech probably
  also a factor in co-present
• but hard to disentangle from communication
• Grounding is an important process with 2 levels
• linguistic (cf. Clark) to establish shared
  reference
• reasoning to ensure shared strategy and allow
  confident progress based on mutual contributions
• both levels more difficult to achieve in remote
  condition
• Continuity between use of ephemeral gestural
  cognitive artifacts (Enfield) and concrete
  diagrams

11
Moral for technology?

• In systems to support distant task
  collaborationtry to reproduce co-presence!
• Hardly a new idea
• But maybe some thoughts here about focusing
  multimodality on areas such as gesture around
  task representations, rather than e.g.
  reproducing the face-to-face experience