CUIML:A language for generating multimodal humancomputer interface

1
CUIMLA language for generating multimodal
human-computer interface

• Christian Sandor
• Thomas Reicher
• ???
• try

2
Presentation Agenda

• DWARF??
• Introduction
• UIML??
• Requirement Analysis
• Related Work
• Design
• Example
• Future Work
• Value to our research

3
Reference

• 1Christian Sandor,Thomas Reicher(2001),CUIML
  A language for generating multimodal
  human-computer interfaces, Proceedings of the
  European UIML.
• 2Marc Abrams,et al.,UIMLAn Appliance-Independe
  nt XML User Interface Language,Garvin Innovation
  Center,VA.
• 3Micah Dubinko(2001),What Are XForms
  ,http//www.xml.com/pub/a/2001/09/05/xforms.html.
• 4Just van den Broecke(2000),Pushlets Send
  events from servlets to DHTML client
  browsers,http//www.javaworld.com/javaworld/jw-03
  -2000/jw-03-pushlet_p.html.
• 5Martin Bauer, Bernd Bruegge, Gudrun Klinker,
  Asa MacWilliams, Thomas Reicher, Stefan Riß,
  Christian Sandor, Martin Wagner (2001),Design of
  a Component-Based Augmented Reality Framework,
  ISAR 2001

4
DWARF??
5
DWARF A Wearable Computer
6
Augmented Reality
????????????
7
DWARF ???
CUIML
8
Introduction

• CUIMLCooperative User Interfaces Markup
  Language.
• CUIML is part of DWARF project
• The DWARF is to develop a framework for
  applications on wearable computers which
  consists numerous devices.
• The interaction with the user should be
  multimodal in this situation.

9
Multimodal System

• Multimodal System supports communication with the
  user through different modalities such as
  voice,gesture and typing
• Modal means Modality and Mode
• Modality
• Communication channel used to convey information.
• Mode
• State that determines the way information is
  interpreted to extract meaning.

10
Multimodal System(2)

• In the multimodal HCI,system should support user
  input and system output by various ways.
• Examples
• Inputspeech,gesture,keyboard
• Outputvoice(VoiceXML),3D graphics(VRML),2D
  graphics,Text(HTML).
• In UIML,we want to describe HCI using an abstract
  language,so this language should allow us to
  define the behaviors of various input / output
  components.

11
UIML ??

• Create natural separation of UI design from
  programming Job.
• Allow many to one relationship between the two.
• Transform to native markup language by XSL
• A measure of the power of UIML is that it can
  replace hand-coding of Java AWT/SWING user
  interfaces.
• (?SALT?????)

12
Where UIML Fits In
Application database
Content (XML, audio,)
Device Adaption UIML
WML
VoiceXML
XHTML
cHTML
13
UIML Perspective
Device-independent markup
Device-dependent markup
Scripting languages
High-level language (C, Java)
Assembly language
Machine language
14
Features of UIML

• UIML can be rendered to many device-dependent
  language like VoiceXML,HTML or WML.
• Web-based
• Interfaces generated with UIML can send events
  but can not receive events.(?????,?????????)
• UIML only generate views that do not interact
  with each other.

15
CUIML Overview

• CUIML use UIML as a starting point.
• CUIML is a unified user interface description
  language to describe multimodal user interface in
  an abstract way.
• CUIML suggests a MVC(Model-View-Controller)
  framework to solve the synchronization problems.

16
CUIML Overview(2)

• By XSL transformations,these descriptions are
  converted into various markup language for
  various devices.
• The controllers internal structure is a
  DFA(deterministic finite automaton),it map CUIML
  to various input and output device

17
Requirements analysis

• Modeling information at a high level of
  abstraction
• Mapping generic information to device-dependent
  presentations
• Synchronization of the different channels
• Controller configuration

18
Related Work

• UIML
• Petri Net
• S.Riss Work Flow Engine
• XForms
• Mozquito

19
Design of CUIML
20
Runtime behavior of the components

• The client request a HCI by sending a HTTP
  request.
• Server renders the components and send back to
  client.
• The rendering is done by applying XSL
  transformations.

21
Initializing the Client
Controller Config / Information for Manipulator
22
Generation of HCI components
23
The generated Components(1)

• Controller
• Synchronize the view components
• Keep track of current state of HCI
• Use WFE implementations in prototype

24
The generated Components(2)

• View
• Display markup languages
• Send event to controller
• The concept of describing and rendering views is
  adopted form UIML.

25
The generated Components(3)

• Manipulator
• An enhancement over UIML.
• Changes to the view is done by manipulator
  instead of rendering new view.
• View is accessed by manipulator by the Document
  Object Model(DOM).(???View????)

26
The generated Components
27
Structure of CUIML
28
Create Multi User Interface with CUIML
29
ExampleSticky
Edit this sticky
Selected
Edit here
????????????
30
DFA of the Controller
31
Define the interface
32
Define the events
33
(No Transcript)
34
Future Work

• To develop a CUIML IDE
• Security concerns of the CUIML component sent to
  client side
• Internal structures of Controller using DFA is
  not sufficient for more complex situations.One
  possibility will be Petri Nets or even Colored
  Petri Nets.

35
The value to our research

• The idea of synchronize VRML ?VoiceXML and HTML
  with MVC Model.(Instead of time tag)
• To change views at runtime by Manipulator.
• The flow control language may be substitue by
  XLANG or Petri Net.

36
Problems may solved using CUIML

• Voice /Animation synchronization.
• Protocol integration.
• ASR/TTS integration and its performance issues.
• Virtual user autonomy.
• The Voice propagation range issues.

37
System Design Prototype(Before)
38
System Design Prototype(After)
39
Q A