Scalable Immersive Enviroments

1
Scalable Integrated Environments
Team E. Chew, P. Georgiou, C. Kyriakakis, M.
McLaughlin, G. Medioni, S. Narayanan, R.
Nevatia,C. Papadopoulos, A. Sawchuk, R.
Zimmermann NSF ERC Meeting December 1,
2006 Scalable Integrated Environment
connecting people and information through the
integration of collaboration tools and sensory
technology for high quality representation of
reality
2
Outline

• Project Introduction
• ScenarioRemote Assistance for Aircraft
  Maintenance
• ScenarioAudioPeer Support for Interactive,
  Participatory Learning
• Conclusions and Future Plans

3
Business Application DomainRemote Problem
Solving
Engine Disposition
Engine Event
Inspect Engine
Report Condition
Order QAR
Remote Conference
KAL Main Base (Gimpo, Korea)
Pratt Whitney (24 Help Desk, USA)
Airport ABC
Partnership
4
Remote Problem SolvingSystem and Test
Infrastructure
Collaborative Site 1 E.g., Airport Maintenance
Manager
Collaborative Site 2 E.g., Manufacturer Help Desk
Expert
HD video rendering
HD video rendering
Camera
Camera
Low latency protocols
Mics
Mics
Internet
Immersive audio rendering
Immersive audio rendering
PC
PC
Wireless
Mobile station E.g., borescope, video and audio
transmissions
Storage and Retrieval
Collaborative Site 3 E.g., Field
Service Technician
5
Research Dimensions
Mobility Seamless wired andwireless environments
Scalability Peer-to-peer versus client-server
topologies
SIE
Scalability Heterogeneous devices, HD SD media
streaming
Archiving Session recording, storageand retrieval
MultimodalData Analysis Metadata extraction
andprocessing
Storage Retrieval
6
Technical Challenges
Storage and Retrieval

• Low latency, high quality transmissions
• Error correction and concealment
• Mobile, wireless environments
• Multimodal recording dataarchiving and
  management
• Information sharing
• Optimization for asymmetric and heterogeneous
  conditions and devices
• Multi-channel echo cancellation
• Audio-visual localization
• Human tracking and identification

7
July 2005
USC-Inha Two-way ExperimentsUSC (Powell Hall of
Engineering)
8
P2P Distributed Virtual Env.
Proximity Audio

• Server-less, distributed virtual environment
  based on p2p topology
• Users are represented as avatars
• Torque engine ACTIVE networking
• Unique ACTIVE protocol provides multiple stream
  channels
• Audio conferencing
• Game message distribution
• Video (planned)
• Broadcast and interactive modes (proximity audio)

CCNC 2006
9
Video
SIE Demo Video
10
Outline

• Project Introduction
• ScenarioRemote Assistance for Aircraft
  Maintenance
• ScenarioAudioPeer Support for Interactive,
  Participatory Learning
• Conclusions and Future Plans

11
Support Interactive, Participatory Learning

• Motivation
• Current DEN infrastructure allows excellent
  delivery of multimedia streams from instructors
  to students.
• Channel from students to instructors/TAsonly via
  face-to-face communication (e.g.,office hours)
  or telephone (one-to-one).
• Participatory learning can be enhancedthrough
  interactive communicationamong students and
  between students and teachers.

There is a need for a collaborative, interactive
toolthat supports group voice communication.
12
Support Interactive, Participatory Learning

• AudioPeer ? a Multiuser Audio Chat System
• Goals Requirements
• Enable discussions and QAfor courses, labs,
  office hours
• Support participation of students,TAs, LAs, and
  instructors
• Be scalable 100 groups(classes), 1000
  students
• Integrate with the existing DENinfrastructure
  (e.g., single sign-on)
• Be extensible, e.g., speech transcripts of
  sessions to allow text search
• Support for on-campus network and off-campus
  broadband network technologies (cable modems and
  DSL).
• Integrated floor control via an adaptive
  algorithm to allow the number of speakers to grow
  as the number of participants increases.

13
Support Interactive, Participatory Learning
AudioPeer, Second Generation

• Web based, simple interface
• Can be easily integrated with DEN
• For large, moderated groups
• Allows floor control by moderator (e.g., TA)
• For small groups (e.g., projects discussions)
• Automatic floor control to allow
  completeinteractivity

ParticipantClient
Administration Interface Visualization of large
groups Monitoring of network conditions, etc.
14
Support Interactive, Participatory Learning

• Accomplishments
• AudioPeer implemented
• Support for on and off-campus students (e.g.,
  with DSL andcable modem connections)
• Software tested on Windows 2000 and XP
• Limited session management implemented(to handle
  discussions of different courses)
• Pilot classroom program use during the fall 2003
  2004 CSci585 Database Systems course

• Virtual office hours for final exam preparation.

Sampleconversationfrom CSci585 Student Mishari
Almishari (A)
(Note audiorecorded viacamcorder,hence
theA/C noise)
15
Demonstration
AudioPeer Demo Video
16
ACTIVE VE Benefits

• Purpose creates a logical shared space with
  objects and participants to execute training,
  maintenance and other procedures
• Server-less P2P architecture (physically fully
  distributed)
• No large up-front capital investment, suitable
  for small organizations, but at the same time
• Scalable to large number of participants
• Low bandwidth works with DSL, cable modems,
  wireless(environments where video might not
  always work)
• Open stream transport architecture that allows us
  to carry game messages, audio traffic, haptic
  messages, video (if desired), and more
• Dynamic optimization of end-to-end latency
  through adaptive topology reorganization
• Integrates with other HYDRA components

17
Status

• 2004
• Two-Way Live HD Streaming between LA and Hawaii
• DIP v.2 Two-Way Baseline User Studies
• One-Way Live 4-channel HD Streaming on Internet2
  (Austin, TX)
• 2005
• WCA 2005 First wireless HD demonstration
  (Washington DC)!
• HYDRA tests between Korea and USC
• 2006
• 3 HD station installations at Korean Air (Korea)
• Recording capabilities implemented in HYDRA
• HYDRA server integrated in SIE and GeoDec
• DIP experiments between USC and Stanford

18
Conclusion and Summary
Engine Disposition
Engine Event
Inspect Engine
Report Condition
Order QAR
Remote Conference
Laptop PC applications allow - Voice Video
over IP - QAR Data Transfer - Remote
Video Boroscope - Session Data Analysis -
Data Sharing
Interactive and Collaborative Process
19
Accomplishments Highlights

• 3 HYDRA HD streaming stations installed in Korea
• HYDRA server integration with SIE and GeoDec
• Internet2 Idea Award nomination of RMI
• HYDRA NSF Collaborative Systems grant
• Patents Multi-threshold Smoothing (6,910,079)
  issued,several filed
• DIP experimentswith Stanford, June 06

• ACTIVE protocol design and implementation
• Example application P2P Virtual Environment
• AudioPeer pilot classroom program use with CSci
  585 and ISE 515courses (2003/04/05)
• Extension to mobile devices Mobisys 05
• Example application Tele-rehabilitation
  (Zumberge 06 grant)

20
Public Awareness Highlights

• Engineering researchers develop next-generation
  videoconferencing technologiesPE Magazine,
  NSPE, March 06
• Publications in CCNC 06, MMCN 06, ACM
  Multimedia 05, MMSJ, MTAP,

• US Finalists in ImagineCup 2006
• 2006 University RiceBusiness PlanCompetition
  participants
• AudioPeer software licensedto USC startup
• Invited talks at
• VON, October 06
• NUS, April 06
• Schloss Dagstuhl, March 06
• Keynote, ACM MSC workshop, Singapore, Nov. 05

sponsored by
21
Related Work

• Streaming
• U. of Washington/ResearchChannel (M. Wellings)
  Uncompressed HD and Sony HDCam (270 Mbps)
• GIST (J. Kim) HDV integration with AccessGrid
• ISI East (L. Gharai, C. Perkins) UltraGrid - HD
  uncompressed (1 Gbps)
• McGill (J. Cooperstock and J. Roston)
  Ultra-Videoconferencing system (SD), audio
  streaming
• Shared Spaces High Definition
  Ultra-Videoconferencing, (2005) McGill, U. of BC
  in Vancouver, 3 HD-SDI video streams (3.5 Gbps)
  funding by Canarie, CFI, Valorisations Recherche
  Quebec, Panasonic, Cisco
• We have shown 4 HD streams (lt 100 Mb/s combined)
  in September 2004!
• HP Halo conferencing system (SD)
• Standfords CCRMA group
• KDDI Japan NAB 2005 press release - Low-Delay
  JPEG2000 HDTV Codec (JH-2000N)
• Polycom VSX8000 H.264 HD? conferencing (press
  release April 26, 2005)
• We are part of the HDTV Working Group of APAN
  (Asia-Pacific Advanced Network Consortium) we
  collaborate with CCRMA and ISI East.

22
Business Application DomainRemote Problem
Solving

• Example Pratt Whitney Institute for
  Collaborative Engineering (PWICE)
• Goals and Objectives
• Improve Korean Airs (KAL) Line and
  BaseMaintenance operations
• Improve personnel and data communicationsusing
  the latest technologies
• Streamline processes for aviation
  aircraftmanagement, maintenance, engineeringand
  support
• Develop and apply new technologies in creative
  ways
• Significantly improve efficiency of processes
• Decrease cost
• Enable high quality, user-centric communication
  and information sharing between distributed
  participants
• KAL maintenance, engineers managers in
  distributed locations including with PW
  personnel
• Trainees and trainers

Partnership
Inha University
23
SIE Functional Diagram
Users
Clients / Applications
A/V acquisition
A/V rendering
Distributed VE
Query/API Interface
Multimodal Processing
HYDRA Stream Engine
Network Protocols
Stream Events
Error Correction
Storage Retrieval
Localization
NAT Traversal
Gesture Recog.
P2P Overlay
???
A/V Indexing
24
HD Two-Way Live Streaming
HD video rendering
HD video rendering
Low latency protocols
Camera
Camera
IMSC net
IMSC net
Mics
Mics
Audio rendering
Audio rendering
Tutor Hall (RTH 217)
Powell Hall (PHE 106)
PC
PC
End-to-end latency 350 ms

• HD resolution 1280x720, 30 fps, 20 Mb/s
• Audio interleaving FireWire IP protocols
• Software rendering MC and iDCT on GPU
• Challenges
• Precise timing with low latency decoding time
  for I, P, and B frames varies
• Error resilience for transmission
• Independent source/destination clocks
• Multiple stream synchronization

Unique, flexible, integrated HYDRAstream
management architecture