Referring to Objects with Spoken and Haptic Modalities

1
Referring to Objectswith Spoken and Haptic
Modalities

• Frédéric LANDRAGIN
• Nadia BELLALEM
• Laurent ROMARY
• LORIA Laboratory
• Nancy, FRANCE

2
Overview

• Context Conception phase of the EU IST/MIAMM
  project (Multidimensional Information Access
  using Multiple Modalities - with DFKI, TNO, Sony,
  Canon)
• Study of the design factors for a future haptic
  PDA like device
• Background Interpretation of natural language
  and spontaneous gestures.
• A model of contextual interpretation of
  multimodal referring expressions in visual
  contexts.
• Objective To show the possible extension of the
  model to an interaction mode including tactile
  and kinesthetic feedback (henceforth Haptic)

3
MIAMM - wheel mode
4
MIAMM - axis mode
5
MIAMM - galaxy mode
6
Reference domains and visual contexts
7
Reference domains and visual contexts
 these three objects 
? , ,
8
Multimodal fusion architecture
user model
referential expression
request
language module
domains
under-specification
history
dialogue manager
request
visual module
domains
history
referential gesture
request
task module
referent
domains
reference domain(s)
history
9
Haptics and deixis

• Haptic gestures can take three classical
  functions of gesture in man-machine interaction
• semiotic function select this object
• ergotic function reduce the size of this
  object
• epistemic function save the compliance of this
  object
• How can the system identify the function(s)?
• linguistic clues (referential expression,
  predicate)
• task indications (possibilities linked to a type
  of objects, affordances)
• Deictic role to make the object salient,
  whatever the function, in order to focus the
  addressees attention on it.

10
Haptics and perceptual grouping

• Interest formalism for the focalization on a
  subset of objects
• Grouping factors
• objects which have similar haptic (in particular,
  tactile) properties (shape, consistency, texture)
• objects that have been browsed by the user (the
  elements of such a group are ordered)
• objects that are stuck together, parts of a same
  object...

11
Haptics and perceptual domains

• Can visual and tactile perceptions work together?
• simultaneous visual and tactile perception
  implies the same world of objects (and
  synchronized feedbacks)
• a referring expression can be interpreted in
  visual context or in tactile context
• How can the system identify the nature of
  perception?
• for immediate references, the visual context
  gives the reference domain and haptics gives the
  starting point in it
• for ieterating references, each type of context
  can provide the reference domain (so both
  hypotheses must be tested)

12
Haptics and dialogue history

• Interpretations that require an ordering within
  the reference domain the first one, the next
  one, the last one
• in visual perception, guiding lines can be
  helpful (if none, an order can always be built
  with the reading direction)
• in haptic perception, the only criterion can be
  the manipulation order
• Some referring expressions that do not need an
  order may be interpreted in the haptic
  manipulation history
• the big one (in the domain of browsed objects)
• them (the most pressured objects)

13
Architecture for speech-haptic referring
user model
referential expression
request
language module
domains
under-specification
history
visuo-tactile module
dialogue manager
request
domains
history
haptic gesture
request
task module
referent
domains
reference domain(s)
history
14
Summary

• What does not change from deictic to haptic
• the status of speech and gesture in the
  architecture
• the repartition of information among speech and
  gesture
• the need of reference domains
• the use of salience and the use of orderings in
  domains
• the algorithms for the exploitation of all these
  notions
• What does change
• some unchanged notions can have more than one
  cause
• objects must be browsed to be grouped in a haptic
  domain
• one aspect of the architecture the visual
  perception module becomes the visuo-tactile
  perception module

15
Application - MMIL design

• MMIL - Multimodal Interface Language
• Unique communication language within the MIAMM
  architecture
• XML based representation embedded in a web
  service protocol (SOAP)
• Oriented towards multimodal content
  representation (cf. Joint ACL/SIGSEM and TC37/SC4
  initiative)
• Example VisHapTac notification to Dialogue
  Manager

16
Overall structure
e0
Visual haptic state
set1
description
Participant setting
s1
s2
s2-1
set2
Sub-divisions
inFocus
s2-2

s25
s2-3
inSelection
17
MMIL format for VisHapTac - 1

• ltmmilcomponentgt
• ltevent ide0gt
• ltevtTypegtHGStatelt/evtTypegt
• ltvisModegtgalaxylt/visModegt
• lttempSpan
• startPoint2000-01-20T141206
• endPoint2002-01-20T141213/gt
• lt/eventgt
• ltparticipant idset1gt
• lt/participantgt
• ltrelation typedescription sourceset1
  targete0/gt
• lt/mmilcomponentgt

18
MMIL format for VisHapTac - 2

• ltparticipant idset1gt
• ltparticipant ids1gt
• ltNamegtLet it belt/Namegt
• lt/participantgt
• ltparticipant idset2gt
• ltindividuationgtsetlt/individuationgt
• ltattentionstatusgtinFocuslt/attentionstatusgt
• ltparticipant ids2-1gt
• ltNamegtLady Madonnalt/Namegt
• lt/participantgt
• ltparticipant ids2-3gt
• ltattentionStatusgtinSelectionlt/attentionStatusgt
• ltNamegtRevolution 9lt/Namegt
• lt/participantgt
• lt/participantgt
• lt/participantgt

19
Future work

• Within the dialogue manager module, domains may
  be confronted, using a relevance criterion
• The way the linguistic contraints of the
  referring expression apply in the different
  domains may be such a criterion.
• Validation in the MIAMM architecture
• The transition from deictic to haptic may not be
  an additional cost for the development of a
  dialogue system, both from the architecture point
  of view and the dialogue management point of
  view.