Communicating Expressiveness and Affect in Multimodal Interactive Systems

1
Communicating Expressiveness and Affect in
Multimodal Interactive Systems
2

• Joshua Mau
• Kyle Leneau
• Andrew Meyer

3
Main Topics of Discussion

• What is Multimodal Interactive Systems?
• How could it affect us?
• What applications is it being used for?
• Our reactions to the article.

4
Multimodal Interactive Systems

• MIEE - (Multisensory Integrated Expressive
  Environments)
• Background information and general knowledge
• Purpose To adapt new mediating technologies to
  basic human forms of communication. (i.e..
  nonlinguistic movement, and gestures)
• Main Goal Establish a framework that accounts
  for the relationship between the current state of
  the art in the stream-based audio-visual
  technology with humanistic theories on
  expressiveness actions and aesthetic experience.
• Artistic applications vs. scientific aspects

5
Multimodal Interactive Systems Cont.

• Layered Conceptual Framework.
• 3 Layers
• Linguistic-based (audio based, valence vs.
  activity)
• Valence positively of negatively valued affects
• Activity force of theses affects
• Gesture-based Maps and spaces (mapping audio
  cues with expressive feelings)
• Signal-based (motion phase, movement stroke, on
  time durations of a few seconds)

6
Multimodal Interactive Systems Cont.
7
Multimodal Interactive Systems Cont.

• Algorithms for cross-model mappings
• Mathematical representations of showing/proving
  how the systems functioned.
• Legato for instance can be measured at the Nth
  note by using such a formula
• L(n) DR(n)/IOI(n)
• L is Legato
• DR is time-related parameters
• IOI is inset-onset interval
• I is intensity
• O is onset duration

8
Multimodal Interactive Systems Cont.

• Implementation using EyesWeb http//www.eyesweb.or
  g
• Interactive Concerts
• Interactive Museums
• Interactive expressive Hi-Fi systems
• Mapping of varieties of music into computer
  animations.
• Other
• Games
• Toys
• Security?

9
How could it affect us?

• With this new technology there are now more ways
  for individuals to communicate with others.
• For example the invention of the electric guitar
  has made recording and reproducing music easier
  than it has ever been. A child could now record
  his/her self playing on their computer and then
  burn a CD and reproduce it for friends and family
  in a matter of a few hours.
• How do you think this could affect your life?

10
What is it being used for today?

• People are using what is known as a conductors
  jacket to write and perform music for large
  audiences.
• They are creating technologies that enable
  communication with computers in the absence of
  today's standard input devices (i.e. mouse,
  keyboard, touch screens, etc)
• Playstation has a game that uses your body
  movement to control the characters actions on
  the screen (EyeToy http//www.us.playstation.com/E
  yeToy.aspx)

11
Reactions and Opinions

• Didnt talk enough about of the challenges of the
  MIEEs. Can artificial intelligence really tell
  me how I am feeling?
• Didnt talk about the creative potential
  implementations, mostly showed on how it is done.
• The article was way to technical, as people who
  implement interactive multimedia solutions it
  didnt talk to much about the possibilities of
  these processes.

12
Questions?

• What is to come in the future?
• Would you use something that expressed your
  emotions?