Multi-modal Interfaces

1

• Multi-modal Interfaces
• The focus of IST in FP6 is on the future
  generation of technologies in which computers and
  networks will be integrated into the everyday
  environment, rendering accessible a multitude of
  services and applications through easy-to-use
  human interfaces
• IST WP 2005-2006

2
Our Mission

• The challenge is
• To make interfaces as simple as possible, and
  interaction as easy as possible
• For citizens to be able to access, receive and
  use information in their own language
• We support the research, development and
  integration of
• Advanced technologies for interfaces
  interaction
• Integrated multilingual information systems and
  services
• We focus on
• Technology-mediated interaction and communication
  between people, between their devices, and
  between people and content
• Combining traditional human language services,
  like human translation and interpretation, and
  advanced language technologies and resources

E1 Interfaces
3
What do we mean?

• Human-Computer Interaction
• is concerned with the design, implementation and
  evaluation of interactive computer-based systems,
  as well as with the multi-disciplinary study of
  various issues affecting that interaction. The
  aim of HCI is to ensure the safety, utility,
  effectiveness, efficiency, accessibility and
  usability of such systems.
• So its all about radical changes in the way
  people work and interact with each other and with
  information
• and systems conceived and designed as
  community-centred, sharable, expandable,
  cooperative, collaborative and responsive media,
  catering through user and environmental
  monitoring, for a board range of human needs

Constantine Stephanidis, ICS-FORTH, ERCIM News,
July 2001
Constantine Stephanidis, International Journal of
HCI, 1999
4
Focus so far

• Multilingual systems
• unrestricted spontaneous speech-to-speech
  translation for specific application scenarios
• statistical/hybrid (integrating linguistic
  knowledge) approaches to automatic translation

• Interaction
• intuitive multi-modal interfaces that are
  autonomous and capable of learning and adapting
  to the user environment in dynamically changing
  contexts
• recognise emotive user reaction
• robust dialogue capability with unconstrained
  speech and language input

Human-to-human technology mediated human
communication Human-to-things virtual and
physical Human-to-self health,
well-being Human-to-content information
retrieval/browsing/navigation Device-to-device
human mediated device-device communication Human-
to-embodied robots assistance, autonomous,
doing-things
E1 Interfaces
5
The story so far!

• International Scientific Forum Towards and
  Information Society for All, International
  Journal of Human-Computer Interaction, Vol.
  10(2), 1998 and Vol. 11(1), 1999
• Special Theme Human Computer Interaction, ERCIM
  NEWS, No. 46, July 2001
• Coordination and Fusion in Multi-modal
  Interaction - a research roadmap, Dagstuhl
  Seminar 01441, Nov. 2001
• Interface Technologies Workshop, Luxembourg, May
  13-14, 2002
• IST Work Programme 2003-2004, Strategic Objective
  2.5.7
• Internal Reflection Group on Interface
  Technologies multi-sensorial, multilingual
  interactivity and virtual environments, 28 May
  2002
• Impact Study of IST section on HLT, contract
  C28262, March 2004
• Multi-modal Interfaces consultation meeting,
  Luxembourg 23 April 2004
• IST Work Programme 2005-2006, Strategic Objective
  2.5.7
• Make it simple - a survey of information
  technology, The Economist, Oct. 30, 2004
• Building the Europe of Knowledge a proposal for
  7th Framework Programme, COM(2005) 119 final, 6
  April 2005

6
Main challenges

• Need to provide information artefacts to be used
  by increasingly diverse user groups
• Expanding context of use, from scientific and
  business use to residential and nomadic use
• Increasing variety of media and devices used to
  access a community-wide pool of services and
  information resources
• Demonstrate added-value of multi-modality and
  performance advances through multi-modal
  interaction
• Go beyond system integration for demonstration
  purposes and target real world solutions
• Emphasise multi-user scenarios and non-business
  applications
• Stronger focus on defining the expected impact
• Need to promote cooperation between small
  European companies
• Promote research that could lead to spin-offs

7
An RD agenda

• Key interaction technologies (speech, natural
  language input, haptics, vision, sensors,
  gesture, agents, , including unconventional
  modalities) and their modelling and integration
  into interaction platforms
• Interface architectures, tools for cost effective
  integration, e.g. fusion, and toolkit
  interoperability
• Computational models of multi-modality, and
  advanced corpora for research
• Re-develop a serious effort on automatic
  translation
• Reinforce work on robustness and acceptability of
  speech recognition and research on
  speaker-independent software
• New learning paradigms in (spoken) language
  translation and portability to new languages,
  including greater focus on language resources as
  computer processable and reusable resources
• Design processes, methods and tools that support
  scalability and modality independence
• Empirical science base for understanding
  end-users, e.g. computational models, test suites
  and benchmarks

8
Future orientations?

• Focus more on adaptability, learning capabilities
  and re-configurable interfaces
• Collaborative technologies and interfaces in the
  arts, including games, design, new media
• Less explored modalities, e.g. handwriting,
  haptics, bio-sensing
• Affective computing, including character and
  facial expression recognition and animation
• Innovative aspects of user centred design and
  ergonomics, and more generally design theory as
  an enabling discipline, e.g. product design
• Multilingualism and machine translation
• A real need for more product innovation and
  industrial impact

Extracted from the Call 1 evaluation, 2004
9
Multi-modal Interfaces

• Objective natural and easy to use interfaces
  that use several modalities or are multilingual
• Natural interaction between humans and the
  physical or virtual environment
• Multilingual communication systems

IST WP 2005-2006
10
Multi-modal Interfaces

• Natural interaction between humans and the
  physical or virtual environment
• Interfaces that are autonomous, and learn and
  adapt to user intentions and behaviour, in
  dynamically changing environments
• Featuring unconstrained, robust interaction,
  recognise user reactions and respond to them in a
  natural way
• Selecting the right combinations of modalities
  according to the preferences and context
• Looking at both the fusion of information related
  to different modalities and their channelling to
  multiple modalities

IST WP 2005-2006
11
5-year Targets

• Natural interaction between humans and the
  physical or virtual environment
• interfaces must demonstrate their ability to hide
  complex technologies, improve ease of use and
  accessibility, and support much more diverse and
  less expert user communities
• making key enabling technologies (speech, vision,
  haptics, ...) much more tolerant of
  inconsistencies and sufficiently robust for wide
  usage
• ensure that voice recognition based upon natural
  language understanding becomes a routine form of
  user-computer interaction

draft
12
Multi-modal Interfaces

• Multilingual communication systems
• For unrestricted domains, including
  task-oriented, real-time understanding of
  spontaneous spoken and gesture input
• Address novel learning paradigms that exploit
  contextual information, human and linguistic
  knowledge in a more effective way than currently
  done
• Portability of new languages taking advantage of
  methods and techniques developed for languages
  already covered is a further challenge to be
  addressed, e.g. in the context of new EU languages

IST WP 2005-2006
13
5-year Targets

• Multilingual communication systems
• establish a productive research community on
  language understanding and machine translation
• move from manual encoding of linguistic
  information to automatic acquisition, analysis
  and annotation of information supporting autonomy
  in the way systems develop linguistic abilities
• develop interoperable written and spoken
  resources and new applications supporting the
  seamless integration of the new languages, within
  the context of EU policies on multilingualism and
  enlargement
• focus on systems that handle multiple languages

draft
14
Overall Context

• And in addition
• Leadership Ambition your project should be the
  most important one in the field in Europe, if not
  in the world
• Success tangible, easy to understand and
  measurable indicators, with highly visible
  results and project products
• Industrialisation strong focus on exploitable
  results, well structured industrial
  participation, and planned and sustainable
  commercialisation
• Standardisation increased effort to bring
  together industrial players, academia, user
  groups and policy makers
• Visibility and dissemination fewer, more
  valuable and visible deliverables, workshops,
  products, demonstrators
• Instruments pick the right one an IP is not an
  inflated STReP

15
The Evaluation Criteria

• European added-value
• Show added value to existing national and
  European programmes
• Compliment ongoing efforts of most partners
• ST excellence
• Position the project vis-à-vis wider strategic
  objectives
• Define clear challenging but attainable
  objectives
• Describe state-of-the-art and go beyond (no catch
  up)
• Commit to some key deliverables/impact measures
• Justify balance between research and development
• Stress visible industry-scale validation through
  experiments, prototypes, demonstrators
• Show that you can scale to industrialisation
• Dont just be interesting, be manufacturable

draft