Meeting of Distant Minds: Remote Interaction in Sentient Spaces

1
Meeting of Distant MindsRemote Interaction in
Sentient Spaces

• Cham, Tat-Jen
• Associate Professor / SMA-CS FellowSchool of
  Computer EngineeringNanyang Technological
  University

2
Remote Human-to-(Human?) Communication

• Intuitive notions of remoteness
• Spatial distance
• Individuals are far apart
• Temporal distance
• Individuals do not receive information live
• More precisely
• Semantic Bandwidth
• how much of what an information producer means
  actually gets communicated to the far end
• Interactive Querability
• how easily can an information consumer retrieve
  desired information in a random access fashion

3
Semantic Bandwidth versus Interactive Querability
Rosetta Stone /Mars Beagle 2
Decreasing Semantic Bandwidth
High-end video- conferencing facility
Back of LT
Front of LT
VulcanMind Meld
Decreasing Interactive Querability
4
A. Maximizing Semantic Bandwidth

• Case Abundant resources
• Augment with missing semantic information
• e.g. accurate visualization for non-verbal cues,
  body language, immediate physical environment
• Case Restricted resources
• Prioritize and transcode only important semantic
  information
• e.g. iconic representation for a soccer game,
  versus poor quality video

5
Consistent Frames of Reference

• Claims
• Incorrect semantics worse than no semantics
• Inconsistent user reference frames distort
  non-verbal semantics
• Consistency of Reference Frames
• Intuitively similar to our natural experience
• Same across all remote users
• Important Reference Frames
• Environment
• Human Interaction
• Data Manipulation

6
I. Environment Reference Frame

• Users experience a shared physical environment
• consistent with intuitive understanding of
  physical space and properties
• Approaches
• Geometrically-correct fusion of multiple remote
  environments (window)
• Virtual transportation of users into
• a remote environment (teleportation)
• a virtual, surrogate environment (Matrix)
• Key Component
• User-aware Immersive Displays

7
Instant Sentient Spaces

• Sentient Spaces
• intelligent indoor environments, embedded with
  sensors and effectors
• dynamically aware and responsive to occupants

• Our Research Vector
• Context aware
• Omni-interactivity, focused on both user input
  and output
• Off-the-shelf equipment
• Robust, casual configuration
• Portable technology to quickly awaken existing
  spaces

• Existing Research Efforts
• Location aware
• Specific user interfaces, focused on user input
• Fixed, expensive equipment
• Often brittle, require careful configuration
• Pre-designated rooms

8
Interesting Research Problems in Sentient Spaces

• Robust User Awareness
• Graceful degradation in user sensing and services
  under adverse conditions, sensor limits, etc.
• Easy Reconfigurability
• Easy visual scripting to define layout and
  functionality of space
• Visualization
• HCI technology involve user input and output
• Current research typically focused on input
• What about output?

9
Projector-Camera Systems

• Projector-Camera Systems as HCI Technology
• Cameras as sensors
• Projectors as effectors
• Projectors areduals of cameras
• Unified analysis
• Good synergy
• PROCAMS 2003
• First IEEE International Workshop on
  Projector-Camera Systems, Nice France, October
  2003
• http//www.procams.org

10
Instant Display WallsCasual Multi-Projector
Stitching

• Goal Combine outputs of multiple casually-placed
  overlapping projectors into a seamless image

• Issues
• Geometric and radiometric distortion due to
  oblique projection
• Color non-uniformity in projector output
• Output mixing issues (i.e. soft-partitioning
  problem)

Display Surface
SEAMLESS
Presentation Server
Projector
Camera
11
Instant Display Walls

• Existing Research Results
• Geometric self-calibration of projectors and
  cameras without grids
• Online photometric modeling of projectors,
  cameras and walls
• Preliminary Results
• P. Song T.J. Cham, 2003
• L.K. Tang T.J. Cham, 2003

12
User-Adaptive, Persistent Displays

• Remove shadows due to user occlusion and
  eliminate blinding light falling on user
• T.J. Cham, J.M. Rehg, R. Sukthankar G.
  Sukthankar, CVPR 2003

13
II. Interaction Reference Frame

• Desirables
• a common viewpoint relationship established among
  users
• natural modes of face-to-face interaction
  correctly represented
• Remote users body language to be conveyed in an
  accurate and intuitive context
• Approaches
• 3D stereo video-conferencing that allows
  geometrically-corrected rendering
• Monocular methods for manipulating head-poses /
  gaze directions

14
Eye Contact in Video-Conferencing
Desired (Star Trek)
Actual
15
Gaze Correction Framework

• Analogous View Transfer
• Learn geometric-photometric analogy mapping for
  face model parameters
• Advantages
• Able to generalize across identities, facial
  expressions, etc.
• Robust to non-rigid-body motion
• Easy to obtain reference views

16
Gaze Correction Preliminary Result

• T.J. Cham, S. Krishnamoorthy M.J. Jones, ICARCV
  2002

Camera view (morphable face modelsuperimposed)
Gaze-corrected view
17
III. Data Manipulation Reference Frame

• Collaboration in a shared data space
• Users can create and understand hand-drawn text,
  glyphs and figures in the same context
• permits data creation, manipulation and
  presentation through natural, intuitive means

18
Shared Space for Real and Virtual Figures

• PROCAMS-augmented Whiteboard
• Shared space for real and digital hand-drawn
  figures
• M.A. Husada T.J. Cham, 2004
• Subsequent Research Directions
• Real-time digitization of hand drawings
• Simultaneous users e.g. disambiguating laser
  pointers
• Hand-based interaction e.g. pointing,
  manipulating virtual knobs and sliders

19
B. Maximizing Interactive Querability

• General idea
• Construct intelligent agent that act as
  interaction proxy at remote site
• But
• While doable for highly restricted scenarios with
  limited interaction, e.g. phone answering
  systems, Eliza
• what about more general interaction scenarios?

20
Learning Probabilistic Scripts from Observed
Interactions?

• Can we learn a model of interaction?
• e.g. instructor agent based on video recorded
  sessions, that takes on the personality of the
  instructor
• ignore user interface issue
• Learn probabilistic scripts that
• Translate sensor inputs into a limited set of
  interaction elements (int-els?)
• Encode joint probabilities for groups of
  interaction elements in particular temporal or
  interval sequences
• Implement a decision tree for how a class of
  interactions should progress

21
Concluding Remarks

• Sentient spaces are great platforms to research
  remote interaction systems
• Framework to think about the problem
• Maximizing semantic bandwidth and interactive
  querability
• Semantic bandwidth related to consistent
  reference frames
• Environment, Human Interaction and Data
  Manipulation
• Huge hotbed of potentially interesting and useful
  research!