Multimodal Affective Technologies

1

• Multimodal Affective Technologies
• a concrete answer from the perspective of a large
  company

Diego Arnone Engineering Ingegneria Informatica
2
Multimodal Interfaces vs. Digital Media

• Digital Media (iTV, podcasting, iPod, weblog,
  smartmobs,) as a Dynamic two-ways Participative
  Process calling for improved and more affective
  human-human and human-machine interaction
• User in the loop from a merely passive
  consumer to active member of a networked and
  participated community fully immersed in an
  intelligent pervasive environment
• Emotional Multimodal Interfaces as the key
  enablers for allowing a more effective
  many-to-many two-way communication and
  interaction
• Affective Multimodal Interaction supporting
  Digital Art Entertainment

3
Conveying Affectiveness in Leading-edge Living
Adaptive Systems
4
The CALLAS project experience

• CALLAS aims to define and develop a multimodal
  architecture including emotional aspects,
  targeted to digital media applications,
  essentially in an ambient intelligence
  paradigm.
• CALLAS aims to be the creative laboratory for
  experimenting new concepts for a more
  participative and affective-enriched user
  interaction in digital media applications
• The key assumption for CALLAS will be to allow a
  rapid and incremental prototyping, where some
  basic components, sub-assemblies or initial
  prototypes will be made available for extensive
  re-use

5
CALLAS Objectives

• To develop a set of software components dedicated
  to the processing of individual modalities,
  their fusion and semantic interpretation (the
  CALLAS Shelf), with input modalities (speech,
  sound, non-verbal behaviour and basic physical
  interaction) and output modalities (speech,
  non-verbal behaviour, affective ECA, music,
  haptics)
• To propose a Software Engineering Framework,
  based on the customised integration of the above
  components into an architecture with authoring of
  application-specific control, supporting rapid
  prototyping and development
• To implement prototype demonstrators (CALLAS
  Showcases), targeted to Multimodal interaction in
  the context of new Media (Digital Arts and
  Entertainment), with emphasis on emotional aspects

6
Working Areas
7
Advancing the state of the art

• The project will investigate an interface
  paradigm in which multimodal input will be
  analysed in terms of representation mediating the
  interaction process rather than in terms of
  interface commands or information access.
• a new way for investigating emotions in
  multimodal interfaces tailored to New Media
  environments. The advent of new media will be the
  enabler for a new way of conceiving emotions in
  multimodal interfaces, since these could be
  considered as social objects, emerging from a
  diffuse mediated interaction between group of
  users
• Emotions as the key driver for allowing New Media
  to unleash yet partially unexplored
  potentialities for a more active two-ways
  communication and natural user interaction
• From a technological perspective the project will
  design a software engineering approach supporting
  the development of multimodal applications

8
Mediated Interaction
9
Physical paradigm for Multimodal Interaction

• Multimodal interaction in Mixed Reality
  environments tends to focus on commands/informatio
  n access targeting objects (in the CALLAS
  proposal this has been referred to as physical
  paradigm)
• One of the objectives of CALLAS is to support
  Multimodal interaction with environments as a
  whole (e.g. artistic installations), or with new
  Media content to better capture user experience
• This should be achieved by
• An integration of Multimodal input over larger
  fragments of user interaction
• The interpretation of Multimodal input in terms
  of emotional representations, which will mediate
  the installation response
• Multimodal interpretation should capture the user
  experience through her relation to the multimodal
  application (e.g. measure empathy), going
  beyond the measuring of the traditional emotions

10
Framework Architecture An Example
11
CALLAS Showcases

• Augmented Reality for Art, Entertainment, and
  Digital Theatre
• Interactive Installations for Public Spaces
• Next-Generation Interactive Television           

12
Augmented Reality for Art, Entertainment, and
Digital Theatre

• AR installations actually constitute an ideal
  environment for Multimodal interaction for the
  following reasons
• Because AR environments are already designed to
  accommodate a range of sensors for tracking the
  user, the same environment can track the users
  non-verbal behaviour, to be interpreted in terms
  of emotional response.
• The sensing devices that support most
  implementations of AR systems can similarly
  support seamless interaction within the artistic
  installation.
• This explains that more recently, digital artists
  have used professional AR environments, such as
  the ARToolkit, to develop interactive
  installations (Maia Engeli).

13
Augmented Reality for Performing Arts

• Every year throughout Europe new AR Artistic
  Installations are being developed, although
  individual production costs remain high
  (100k-700k)
• Traditional solutions to this problem have been
• the development of these installations as part of
  mixed Art and Technology structures, for instance
  academic centres in Digital Arts
• the use of software development kits that
  facilitate the integration of the installation
  devices, such as Max/MSP (used for instance to
  support the development of Magali Desbazeilles
  you think hence I follow you, or some of Studio
  Azzurros performances).

14
Augmented Reality for Performing Arts Digital
Scenographies (1)

• The emerging affective multimodal interfaces
  allow today new yet partially unexplored ways of
  conceiving, experience, living and enjoying
  pervasive and immersive theatre-based
  performances (dance, drama, plays, opera, etc.),
  which are more and more interactive, immersive,
  emotional, and open to new forms of relationships
  between the humans (actors, audience, stage
  director) and the adaptive interactive spaces.
• This showcase will demonstrate how the real time
  detection of the affective state of the people
  involved in a live performance (directors,
  actors, audience) can improve the naturalness
  of the experience, paving the way for unexplored
  ways of broader and more pervasive participation
  of the users (audience).

15
Augmented Reality for Performing Arts Digital
Scenographies (2)

• The showcase aims at allowing all the performing
  arts practitioners (director, stage manager,
  actors, playwrights, scenic designer, costume
  designer) to use and master the latest affective
  and multi modal for interactively conceiving,
  designing, authoring, sharing and experiencing
  naturally interactive performing arts works.
• It will also provide innovative natural immersive
  interfaces and devices for creating or living
  interactive performing art works within
  multidimensional (virtual, augmented, mixed)
  realities.
• The real challenge will be just to move forward
  from an off-line design of a virtual stage to a
  real-time virtual scenography.

16
A Prototype example

• Augmented Reality for live performances
• 1. Components analysis and identification
  Storyboard, interaction concept, modality
  analysis (month 6)
• 2. Proof of Concept AR technology baseline
  Basic components integration and fusion,
  interpretation of components speech, gesture
  recognition, sound capture (month12)
• 3. First trials with prototype installation,
  evaluation of interaction concepts support (month
  18)
• 4. Extended prototype (at month 24) including
• Extended emotions representation
• additional modality combinations
• upgraded versions of the selected shelf
  components
• Additional components from the framework (i.e.
  ECA)
• 5. Complete installations supporting users
  trials (month 36)

17
Overall Exploitation strategy

• The main business aspect for CALLAS will be the
  commercial exploitation and selling of the CALLAS
  framework, as well as consultancy to vertical
  application developers, service providers,
  producers, event organisers and network operators
  to create new interactive entertainment formats,
  accessible content and e-training applications.
• The market for CALLAS framework will be mainly
  constituted by
• CALLAS customers, players (artists, technology
  providers) which are aimed at developing
  multimodal applications
• CALLAS users, organisations/institutions which
  are interested in committing such multimodal
  applications (local authorities for festivals,
  theatres, etc.)
• The CALLAS showcases will provide tangible
  evidence of the commercial potential offered by
  the technology and will be used by the supplier
  partners to focus their industrial offering.
• The showcases demonstrated within the project are
  also expected to directly generate revenues in
  the final year of the project, and these would be
  rolled out by the user partners and user group
  members (Studio Azzurro, Teatro Massimo and Ars
  Electronica) to wider audiences.
• The CALLAS consortium intends to provide the
  framework as open source.

18
Multimodality fostering innovation in the New
Media Market

• Emotional multimodal interfaces as enablers of
  innovation in the Digital Media domain
• Multimodal technology is pushing Process
  Innovation at two levels
• Emerging technologies are changing the way in
  which the same things are made
• New services/applications are emerging (mobile
  tv, internet TV, Digital Art, Digital Media,
  Co-operative Geo-based Videogames,)

19
Business Cases (1)

• CALLAS business is based on a wide range of
  different business cases
• Cost reduction for
• CR1 SMEs developing digital interactive
  installations (e.g. Studio Azzurro, Humanware)
• CR2 Traditional theatres (Teatro Massimo) using
  multimodal applications complementing the
  physical scenographies
• CR3 Local authorities, Digital Arts Museums
  setting-up interactive live performances (white
  night events, etc.)
• Revenue increase for
• RI1 Large Service Providers enlarging their
  market to Digital Entertainment Business
  (Engineering, Metaware)
• RI2 Companies providing new services or tools in
  the growing interactive TV market (DigitalVideo,
  XIM, BBC)
• RI3 Spin-offs based on providing advanced mobile
  services in festival (VTT, SNS)
• RI4 Theme Parks using Mixed Reality
  Installations

20
Business Cases (2)

• Business case Cost reduction in live performance
  
• Description large re-using of CALLAS components
  and/or sub-assemblies in the preparation and
  setting-up of a digital interactive opera
• Analysis of estimated costs of two real cases
  Michael Nyman opera Facing Goya and Neither
  opera in Stuttgart Staatsopera
• Key assumption
• CALLAS will reduce costs and time for designing
  and implementing High-quality multimodal
  applications for artistic mixed-reality digital
  installations, since powerful and suitable
  software engineering techniques will be used
• Conclusion A 20 average cost reduction is
  expected in the production of a digital opera

21
Engineering Business Case

• Business case 1. Revenue Increase in Digital
  Entertainment Market
• BC description increase the market share for
  industry, telecommunication and services business
  line (currently 22) throughout the penetration
  in the digital entertainment market
• A strong commitment from the CEO (see Engineering
  internal magazine, home page)
• Key financial assumptions
• growth rate for Engineering in last years of 17,
  as in the corporate accounts
• Analysis of revenue and market share in last
  three years
• Extrapolation of data considering a changing in
  the market share, thanks to a larger penetration
  in the digital media one (despite with a
  decreeing of the finance one)
• Conclusion an expected value creation for 21
  millions of Euro after 2007 (average) in term of
  revenue increasing and of 1,06 millions (average)
  in term of net profit.

22
ENG Business Case Figures
23
iTV Business case

• Creation of new iTV services
• Problem
• saturation of traditional content (DTV in 75
  European homes and broadband in 60 by 2010)
• Many delivery models competing for consumer
  revenue (IPTV, 3G)
• At the same time, large scale investment from
  network operators in modernising and extending
  infrastructure
• Opportunity
• Users becoming increasingly sophisticated while
  current DTV UIs are first generation
• Set top box hardware and network performance is
  evolving to enable advanced applications and
  interfaces
• Need for new services to grow the market
• CALLAS will provide a framework to enable iTV
  developers to create empathic, emotion-sensitive
  interactive services to increase revenue and
  consumer loyalty

24
iTV Business case

• Cost reduction for iTV adoption of
  affective-based multimodal applications
• Problem
• Deployment of affective interfaces would
  currently require complex bespoke development,
  resulting a specific application
• The high volume of end users requires that
  hardware must be as low cost as possible with
  maximum use of efficient software/firmware and
  intelligent reuse of standard hardware
• In order to be deployed to millions of homes, new
  iTV applications must be resilient, robust and be
  based on well-tested reusable components
• Opportunity
• Callas is creating a set of reusable components
  which spread development costs across multiple
  new media applications, platforms and
  entertainment domains
• Callas will therefore reduce costs and time for
  designing and implementing affective-based
  multimodal applications
• Conclusion Callas will help to make
  affective-based multimodal applications viable
  for iTV developers by sharing costs with other
  domains and maximising reusability

25
Open source

• CALLAS intends to provide the framework as open
  source software
• The benefit of open source licence and an open
  architecture are
• The free availability of the framework and the
  architecture enables every interested user to be
  part of the CALLAS system with low investment
  costs.
• New or improved functionalities can be added by
  the user community also after the lifetime of the
  project.
• The CALLAS system can be ported to new platforms
  which are not supported by the project or which
  cannot be foreseen at the moment.
• The CALLAS results will be sustainable as the
  maintenance can be done by the user community as
  well as by companies.

26
Technology Transfer for Multimodal Interfaces

• Many RD institutions involved in multimodal
  interfaces domain
• Plenty of dynamic and innovation-led SMEs
• Multimodal interfaces market very risky and
  uncertain
• Incubator Model for transferring technology
• Large IT Service Provider as hub for
  leveraging the research and the innovation to the
  market

27
The Consortium
Large Industrial Players Engineering, BBC,
Metaware
New Media SMEs XIM, Nexture, Digital Video,
Studio Azzurro, Humanware
Research and Academic VTT, UOA, ICCS, FPMS, TEES,
TKK, PAR8, SNS, UOR, HIT, TMAS
28
Technology Transfer Model
RD
Innovation-led SMEs
Large Industrial Players
Market (Application Developers, Third-Party
Services, Artists,)
29

• THANK YOU
• Diego Arnone
• Engineering Ingegneria Informatica
• Contacts and references http//www.eng.it
• RD Direction
• Roma Italy
• Tel. [phone]
• Mob 39 335 1038042
• Email massimo.bertoncini_at_eng.it