Measuring Human Multimodal Behavior and Developing Multimodal Interfaces

1
Measuring Human Multimodal BehaviorandDeveloping
Multimodal Interfaces

• Jean-Claude MARTIN
• LIMSI-CNRS, FRANCE
• LINC, University Paris 8, FRANCE
• martin_at_limsi.fr
• www.limsi.fr/Individu/martin

2

• Is it possible to identify a minimal set of
  generic types of cooperation between
  communication modalities that
• are independent of the modalities
• are observed in human behavior
• should be used in computer I/O behavior

3
CONTENT

1. Framework
2. Measuring Behavior
3. Futures applications

4
1. FRAMEWORKTYCOON (Martin 93)
5
1. FRAMEWORKDefinitions
6
2. MEASURING BEHAVIOR SRI WOZ Toronto Map (Cheyer
et al. 98)

• How to analyze the multimodal behavior ?
• How to specify a multimodal system ?
• Find a common grid ?
• 10 annotated subjects

7
2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer
et al. 98)
GRAPHICS Royal ontario museum selected
SPEECH Can you give me more information about the museum
GESTURE -
8
2. MEASURING BEHAVIOR SRI WOZ Toronto Map

• Kehker et al. 98

9
2. MEASURING BEHAVIORSRI WOZ Toronto Map (Cheyer
et al. 98)

• Annotating and managing ambiguity
• Annotate cooperation at a finer level
• Nb modalities, nb of attributes

GRAPHICS Campbell House not selected but name displayed
SPEECH What is there in the Campbell House?
GESTURE Draws an arrow towards Campbell House
10
2. MEASURING BEHAVIOR IST-ISLE Project
http//isle.nis.sdu.dk/

• 2000-2002
• Natural Interaction and MultiModality
• NIS, URome, IMS, DFKI, MPI, ELRA, LIMSI
• Survey on Data Resources
• 225 on gestures speech
• Survey on Coding Schemes
• 103 on gestures speech

11
2. MEASURING BEHAVIOR XML DTD Java
12
2. MEASURING BEHAVIORXML DTD Java
For each object o For each reference r If the
target of r is o Then sum salienceOf o in
r Rate red-compl sum / nbReferences
13
3. FUTURE DIRECTIONSApplication to teaching

• How do teachers modify their mm communication
  behavior depending on student level, time slots
• Towards mm pedagogical agents
• BQR Université Paris 8

14
3. FUTURE DIRECTIONSSpeech Therapist Corpus

• Multiple tasks
• Goals
• Who types ?
• Improvement
• Multimodal
• Mediated
• Collective gesture

15
3. FUTURE DIRECTIONSApplication to home
automation

• Goals
• Usability of mm for in-home application
  (interactive TV)
• Applying / Adapting the TYCOON methodology
• RNRT Project
• 2 years
• Partners
• MediaLive (simulation room of in home
  environment)
• Laboratoire Conception de Produits Nouveaux et
  Innovation (Ecole Nationale Supérieure dArts et
  Métiers)

16
Measuring Human Multimodal BehaviorandDeveloping
Multimodal Interfaces

• Jean-Claude MARTIN
• LIMSI-CNRS, FRANCE
• LINC, University Paris 8, FRANCE
• martin_at_limsi.fr
• www.limsi.fr/Individu/martin