conference vi99, Trois-Rivieres, 19 mai 99

1
SSGRR - 2000International Conference on Advances
in Infrastructure for Electronic Business,
Science and Education on the Internet,
July 31 - August 6, LAquila, Italy
2

• Designing Virtual Environments for Critical
  Transactions and Collaborative Interventions
  the VERTEX / APIA Framework for Networked,
  Physics-Compliant Objects

Denis Poussart Denis Laurendeau François
Bernier Martin Simoneau
Nathalie Harrison Denis Ouellet Christian Moisan
poussart_at_gel.ulaval.ca
www.gel.ulaval.ca/vision/
Supported, in part, by grants from NSERC, FCAR,
the Institute for Robotics and Intelligent
Systems and the Canadian Foundation for Innovation
3

• Critical Interventions represent cases where
• errors, delays, lack of optimization
• may have very negative consequences
• for safety
• for the environment
• for health
• for costs

• In the future, as more and more complex
  situations arise, we may anticipate that
  operational support from Virtual Environments
  will become paramount and prevalent in the
• planning
• training
• execution phases of delicate tasks

4

• The inspection, maintenance and repair of
  hydroelectric facilities is just one example

5
Virtualizing reality
6

• But something is missing ...

• For critical tasks, visual illusion is not
  sufficient.

• There is more to real things than just
  shape, or forms, even if they are augmented
  with some behaviors.

Reality includes PHYSICS!

• Accurate physical modeling, laws, and
  simulation capabilities must be integrated
  within the virtual environment.

7
Virtualizing reality
8
This opens up a huge question space!

• what is relevant to be physically modeled?

• what would be appropriate forms of models?

• what level of detail?

• perhaps multiple levels of detail, depending ...

• how to develop scenarios ?

and brings about many integration issues ...
9
We are exploring this approach in
VERTEX Virtual Environments from 3D
Representations to Task planning and EXecution

• A project of Phase 3 of the Institute of Robotics
  and Intelligent Systems (IRIS) of the Network of
  Centers of Excellence program of Canada.

• Objective is to optimize the execution of
  delicate tasks by combining the accurate
  simulation of actual scenes, tools and processes
  with advanced human machine interfaces.

10
VERTEX
11
VERTEX
12
VERTEX
13
VERTEX
Training
Simulated scenarios
Planning
Optimized scenarios
Task simulation in VR mode Reactive Interaction
Predictive evaluation
Task decomposition
14
VERTEX
Task supervision Teleoperation in augmented VR
mode
Execution
Training
Real time control of robot and tools
Simulated scenarios
Planning
Optimized scenarios
Task simulation in VR mode Reactive Interaction
Predictive evaluation
Task decomposition
On site acquisition
15
VERTEX
Task supervision Teleoperation in augmented
VR mode
Execution
Training
Real time control of robot and tools
Simulated scenarios
Planning
Optimized scenarios
Task simulation in VR mode Reactive Interaction
Predictive evaluation
Task decomposition
On site acquisition
16
Who is the user? What are his / her needs?
Design issue

• Actually, complex interventions typically
  involve several users, of various types, with
  different needs, perspectives and internal
  models

• These users, acting cooperatively, might
  very well be in different locations

User-centric design
Hix, D., Swan, E., Gabbard, J., McGee, M.,
Durbin, J., King, T. (1999) User-Centered Design
and Evaluation of a Real-Time Battlefield
Visualization Virtual Environment. In Proceedings
of IEEE Virtual Reality '99
17
VERTEX
On site Acquisition
Modeling off-line
18
A key aspect of physics relates to the handling
of time. Real time????
Design issue
In (critical) VEs, time has many different
flavors

• it might just flow out of the action loop,

• it may relate to factors which impact upon the
  users sense of interactivity, such as latency
  jitter,

• during strategic planning activities, it blends
  with predictive evaluation,

• and during the direct, immediate supervision.
  control, execution of the task (Augmented
  Reality), timing accuracy is mandatory this is
  the realm of hard real time.

• But different physical components may require
  different time resolution run time optimization
  requires fine grain control of time.

19
The (physics) simulation engine is at the core of
the system
Design objectives
Beside its predictable real-time behavior, the
engine should be capable of supporting

• dynamic extensibility (non - stop)

• internal coherence, robustness

• precisely known degradation

• multi-resolution behavioral modeling

20
Design objectives (cont)
From an implementation point of view, the engine
should seek

• modularity, reusability

• networked deployment (multiple users,
  geographica extent of tasks)

• capability to operate from heterogeneouscomponents
  with run-time binding

• close match to current and foreseeable trends

- Moores law - high-speed networking
21
Implementation choice
Use the Common Object Request Broker Architecture
(CORBA) as the software bus - the glue - in
assembling the Vertex system.
Why? CORBA is heavy, with significant overhead
...
True, but as time will unfold
22
APIA
network
Actors Properties Interactions Architecture
23
Physics engine
APIA
network

• Maintains an on-going representation of the
  world

• A Lego-like approach, with hierarchical
  capabilities

• Implemented on a cluster of COTS (à la Beowulf)

• Runs (preferably) on hard real-time OS (OSs)

• May include heterogeneous components

24
Controller
APIA
network

• Overall management

• Scenario authoring

• Repository of model objects

25
Sensors Actuators
APIA
network

• I/O links to the actual physical world

26
HMIs
APIA
network

• Multiple and different views / interactions
  easily implemented

• To suit the representation levels required by
  different types of users

27
APIA
network

• Provides the physical glue between the components

• Geographically - distributed computing, users

• RT-CORBA provides the logical glue

• Designed to fully exploit high performance
  networking QOS, CaNet3

28
(No Transcript)
29
Implementation choice
Vertex uses ACE and TAO, a CORBA
implementation under development at the Center
for Distributed Object Computing, Washington
University. ACE / TAO is designed to support
real-time networked applications, with rigorous
control of task priorities and QOS.
Douglas C. Schmidt, Center for Distributed
Object Computing,Washington University
30
A generic approach ...
VERTEX / APIA is currently being deployed in
other areas, such as breast and liver cancer
treatment through cryosurgery.

• minimally invasive surgery shares many
  aspects of telerobotics,

• a collaborative project with the Imaging
  Research Unit of Hopital St-François dAssise
  (Dr. C. Moisan) and the Finite Element
  Research Group at Laval.

31
VERTEX
Execution
Task Supervision Teleoperation in AR mode
Training
Real-Time Cryogenic Probe Control
Simulated scenarios
Planning
Optimized Scenarios
Task Simulation in VR mode Reactive Interaction
Predictive Evaluation
Task Decomposition
MRI Acquisition
3D Geometrical and Tissue Modeling
Models of Augmented Scenes
Behavior Modeling
Modeling
Tools, Materials and Processes
32
Real-time display of (simulated) cold front
spatial distribution
On-going 3D visualization of cryoprobelocation
and orientation
33
Current status andfuture work
Video