Mixed Reality: A Model of Mixed Interaction

1
Mixed Reality A Model of Mixed Interaction
User Interface Engineering Team CLIPS-IMAG
Laboratory, University of Grenoble

• Céline Coutrix, Laurence Nigay

2
Introduction Mixed Systems

• Merging Physical and Digital Worlds
• In many application fields

1
3
Introduction Designers Needs
A framework for
2
4
Outline

• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

3
5
Outline

• Mixed Interaction Model
• Mixed object
• Mixed interaction
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

4
6
Example
5
7
Mixed Interaction Model Mixed Object
mixed object
tool
task object
6
8
Mixed Interaction Model Mixed Object

• Interaction modality m (d, l)
• speech (microphone, natural language)

7
9
Mixed Interaction Model Mixed Object

• Linking modality

mixed object
input device
input language
output device
output language
8
10
Mixed Interaction Model Mixed Object
mixed drawing
computer vision
camera
image
shapes
projector
selection
image
9
11
Mixed Interaction Model Mixed Object
a music bottle
screen table
audio sequence
identity
10
12
Mixed Interaction Model Mixed Object
mixed object
physical properties
digital properties
Linking modalities
tool
task object
11
13
Mixed Interaction Model Mixed Interaction
tool
elementary task
action
response
reaction
feedback
12
14
Mixed Interaction Model Mixed Interaction

• Interaction modality m (d, l)

13
15
Mixed Interaction Model Mixed Interaction
mixed tool
input language
input device
output language
output device
14
16
Mixed Interaction Model Mixed Interaction
15
17
Mixed Interaction Model Mixed Interaction
mixed tool
computer vision
mixed tool
camera
image
button pressed word, place on drawing
paper button
translation
drawing
16
18
Outline

• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

17
19
Descriptive Power

• Schemes for classification and better
  understanding of mixed systems

18
20
Outline

• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

19
21
RAZZLE Case Study
20
22
RAZZLE Case Study
mixed piece
geometric transforma
location sensor
numeric values
geometric transforma
orienta sensor
numeric values
3D projection
image
21
23
Generative Power
mixed tool
?
?
?
?
?
22
24
RAZZLE Gestural Modality
23
25
RAZZLE Gestural Modality
mixed tool
computer vision
camera
image
physical properties
recognized gestures
translation
24
26
RAZZLE Tactile Modality
25
27
RAZZLE Tactile Modality
mixed tool
input language
touchpad
data
physical properties
click
translation
26
28
Outline

• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

27
29
Comparative Power

• Criteria within the model for assessing multiple
  design alternatives
• Observability
• Spatial continuity

28
30
Comparative Power Observability
mixed tool
input language
input device
output language
output device
task object
input language
input device
output language
output device
29
31
Comparative power Observability
mixed tool
input language
camera
image
physical properties
recognized gestures
translation
30
32
Comparative Power Observability
mixed tool
input language
touchpad
data
physical properties
click
translation
31
33
Comparative Power Spatial Continuity
mixed tool
input language
input device
output language
output device
task object
input language
input device
output language
output device
32
34
Comparative Power Spatial Continuity
mixed tool
computer vision
camera
image
physical properties
recognized gestures
translation
mixed piece
geometric transforma
location sensor
numeric values
geometric transforma
orienta sensor
numeric values
3D projection
image
33
35
Comparative Power Spatial Continuity
mixed tool
input language
touchpad
data
physical properties
click
translation
mixed piece
geometric transforma
location sensor
numeric values
geometric transforma
orienta sensor
numeric values
3D projection
image
34
36
Outline

• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work

35
37
Conclusion

• Definition of a Mixed Interaction Model which
• Capitalizes existing approaches to describe mixed
  systems
• Allows designers to create new interaction
  modalities
• Allows them to choose between multiple design
  alternatives

36
38
Future Work

• Improving comparative power by identifying more
  criteria

37
39
Future Work

• Providing a development environment based on
  ICARE

38
40
Mixed Reality A Model of Mixed Interaction
User Interface Engineering Team CLIPS-IMAG
Laboratory, University of Grenoble
Céline Coutrix, Laurence Nigay