ICT Networking Session

1
ICT Networking Session SEMANTIC ADAPTATION IN
AFFECTIVE INTERACTION

• STEFANOS KOLLIAS
• National Technical University of AthensComputer
  Science Division
  School of Electrical and Computer Engineering
• Lyon, France, November 25, 2008

2
Introduction

• Affective computing is of major importance in
  Human Machine Interaction it involves
  perception, interpretation, cognition,
  expression.
• Related International conferences include
  ACII2005, ACII2007, LREC2007, LREC2008, ACII2009.
• EU IST projects and networks (Ermis, Interface,
  Safira, Humaine, Callas, Semaine, Feelix-Growing,
  Metabo) investigate(d) different issues of
  affective computing theories and models of
  emotional processes, computational modelling,
  emotional databases, input signal analysis,
  emotion recognition, generation of embodied
  conversational agents.

3
State-of-the-art

• Affective computing systems
• model and analyse single or multi-modal affective
  cues
• extract and use statistical information and
  rules
• create data sets and environments to perform user
  state detection
• include Embodied Conversational Agent (ECA)
  synthesis and interaction.

4
Requirements and achievements

• Common frameworks
• i) discrete, dimensional, component,
  appraisal emotional affective models,
• ii) cognitive goal-based interaction
  computational models
• iii) Facial Action Coding System (FACS) and
  MPEG-4 models for facial, viseme and body motion
  analysis and synthesis,
• iv) features extracted from paralinguistic
  speech analysis,
• v) affect indicating words from linguistic
  speech recognition,
• vi) physiological characteristics and
  cues,
• vii) multi-modal integration through
  feature or decision fusion.
• Need to address emerging requirements humanlike
  interactions, less constrained environments,
  adaptive artificial systems

5
Problems encountered

• Synchronisation, contradiction, co-existence
  lack, evolution.
• Fusion of different signals and cues requires
  inferring on contradictory or ambiguous emotional
  cues
• Assuring that interaction is acceptable and
  appropriate in terms of user experiences.
• Lack of common knowledge framework consistently
  representing taxonomies, rules and correlations
  in affective computing.
• Recognition performance and fusion of multi-modal
  inputs is not satisfactory across varying and
  different environments,
• Lack of interaction cycles where, in real time,
  the analysis loop feeds the synthetic one with
  info which is then reused further through users
  affective feedback,
• Including the cognitive component and the context
  of interaction in the loop makes the situation
  much harder and systems almost impossible to
  benchmark.

6
Problems encountered (2)

• Test databases utilize sets of well-defined
  multi-modal (aural, visual, biosignal) data,
  captured in controlled or partly uncontrolled
  environments, depicting humans engaged in
  predetermined tasks.
• Moreover, knowledge obtained from one specific
  dataset may only be efficiently reused in the
  same environment. This refers to
• low-level requirements, e.g., similar lighting
  conditions or absence of occlusion in visual
  signal analysis,
• user-oriented restrictions, e.g., same subjects
  with same physical characteristics and similar
  expressivities,
• similar cognitive or interaction tasks, e.g.
  reading or speaking to an artificial listener,
  classifying hand gestures in predefined
  categories.

7
SAFE Concept

• Generate a common adaptable semantic
  representation framework, i.e., a consistent
  environment to serve as knowledge substrate for
  affective interaction in real, uncertain,
  environments.
• Basis
• state-of-the-art imperfect (uncertain,
  incomplete, vague, inconsistent) knowledge
  representation formalisms,
• novel alignment, mapping and reasoning tools,
• semantic adaptation and learning.

8
SAFE Concept (2)

• Generate a learning, evolving and adapting
  cognitive model
• Start with basic knowledge about the nature of
  possible interactions, users and the environment,
  
• Include powerful sensing and reasoning
  mechanisms, along with the ability to infer from
  expert knowledge reinforced by accumulated
  experiences,
• Result in a system gradually evolving its
  knowledge to incorporate its observations along
  with its own or the users evaluation.

9
SAFE Model
10
SAFE Technologies

• Formal Knowledge
• It stores the terminology, axioms,
  assertions, constrains that describe affective
  interaction.
• - HCI Ontologies module (formal ontological
  description representing the concepts and
  relationships of the field, providing formal
  definitions and axioms that hold in every HCI
  environment.
• abstract descriptions of users
  affective states
• low level descriptions of multimodal
  users info
• context description (user expressivity,
  goals of
• interaction, environmental
  characteristics).

11
SAFE Technologies (2)

• Real environments cause inconsistencies in the
  Formal Knowledge
• For example, the personality and expressivity of
  the specific user make some of the axioms and
  constraints of the HCI Ontology non-applicable or
  even wrong, according to logical entailments or
  user feedback.
• These inconsistencies make the formal use of
  knowledge that the SAFE Reasoner provides rather
  problematic.
• How to solve

12
SAFE Technologies (3)

• The reasoner detects the inconsistency
• It follows a paraconsistent reasoning approach
  taking into account that the interaction with the
  user is assumed to be continuous and the user
  feedback will provide the system with additional
  important information needed in order to resolve
  the inconsistency.
• The Knowledge Adaptation component of SAFE
  resolves the inconsistency through a recursive
  learning process.

13
SAFE Technologies (4)

• Knowledge adaptation
• It determines the minimal set(s) of axioms
  that cause the inconsistency,
• - Inconsistency Handling module
• The minimal sets get represented in
  connectionist models and, with the aid of
  learning algorithms, are adapted and then
  re-inserted in the knowledge base. -
• Some parts of the knowledge represent
  properties of objects that are rigid, while
  others are highly dynamic.
• For the latter, the adaptation will be
  performed more drastically, with high learning
  rates, while for the former a more careful
  strategy will be followed.

14
SAFE Target

• Semantic adaptation in affective interaction.
• Key words are learning and knowledge.
• Successfully combine these main components of
  cognition and machine intelligence.
• Adaptation and evolution of ontological knowledge
  to effectively handle the context of
  interactions, i.e., specific user
  characteristics, goals behaviours, or
  environmental changes.

15
SAFE Target (2)

• The SAFE accomplishments will follow and closely
  relate to current state-of-the-art developments
• in W3C (RIF, OWL Development groups, EMOXG, MMI,
  Emotion Incubator)
• in the Humaine Association (http//emotion-researc
  h.net/association), so that the SAFE system and
  technologies can be easily distributed and shared
  by the RD community for affective interaction.

16
Advancing the State-of-the-Art

• Computational models of Affect
• ?Taxonomies of affective user states, related
  to computational models of affect and emotions
  (Scherers proposal for distinguishing classes of
  affective states
• Emotions (e.g., angry, sad, joyful,
  fearful, ashamed, proud, elated, desperate)
• Moods (e.g., cheerful, gloomy,
  irritable, listless, depressed, buoyant)
• Interpersonal stances (e.g., distant,
  cold, warm, supportive, contemptuous)
• Preferences/ Attitudes (e.g., liking,
  loving, hating, valuing, desiring)
• Affect dispositions (e.g., nervous,
  anxious, reckless, morose, hostile)

17
Advancing the State-of-the-Art (2)

• Interpretation of Affective User Behaviours
• Context Modelling
• Imperfect Knowledge Representation and Reasoning
• Connectionist model for ontology adaptation

18
Applications

• Robotics (Emotion aware, Knowledge Aggregation,
  Context Analysis) FEELIX-GROWING
• Human Computer Interaction in Emotional
  Environments (Knowledge and Emotion) CALLAS
• Analysis of status of children/students in
  e-learning Environments (Behaviour Analysis,
  Knowledge of User States) AGENT-DYSL
• Analysis of status of car driver (monitoring
  safety) METABO

19
Prospects

• Requirement for Novel Cognitive Systems
  interweaving Knowledge Learning/Adaptation
  Technologies
• Theoretical and Technological Advancing of the
  State-of-the-art
• Novel Applications Emotion, Context and
  Knowledge Aware Robots, Tutors, Systems,
• New FP7 ICT Call for funding.

20

• Thank you for your attention.
• contact details
  stefanos_at_cs.ntua.gr