Review97

1
4D Modeling and Mobile VisualizationJune, 2002
William Ribarsky and Nickolas FaustGVU Center
and GIS CenterGeorgia Institute of Technology
2
Research Goals

• What do we do with ever-expanding collections of
  3D data that are being automatically collected
  and modeled from multiple sources?
• Scalable, hierarchical, fused data organizations
• For interactive visual exploration
• For other applications
• Mobile interfaces
• Mobile applications

3
Accomplishments and Collaborations

• Development of mobile situational visualization
  applications
• User studies of multimodal interface for mobile
  and pervasive computing
• Development of multiresolution techniques for
  interactive visualization of high detail urban
  scenes (with Avideh Zakhor).
• Application of transparent dynamic objects and
  new hierarchical, interactive volume rendering
  technique to weather and uncertainty (with Suresh
  Lodha)
• Transition of tools within VGIS to users and
  applications (e.g., Sarnoff, NIMA, etc.)
• Presentation of mobile emergency response
  application to President Bush and Governor Ridge

4
Urban Visualization Our Ultimate Goal
To make this...
5
Urban Visualization

• To reach this goal requires
• Hierachical structure
• Multiresolution methods
• Geometry-based methods
• Image-based rendering

6
Hierarchical, Multiresolution Methods forHighly
Detailed Urban Scenes
7
Facades Bounding Sphere Hierarchy
Sphere surrounding façade vertex
Divide along longest axis
8
Continuous LOD Simplification
The initial combination step
This works well with the bounding sphere
hierarchy (there are few sliver triangles or too
many triangles associated with a single points).
9
Application of Hierarchical Continuous LOD Method
Factor of 50 reduction in textured triangles
Full Resolution (geometry plus texture)
10
Application of Hierarchical Continuous LOD Method
Factor of 50 reduction in textured triangles
Full Resolution (geometry plus texture)
11
Transition from Façade-based LODto Block-based
LOD
Requires simplification of façade to a few
textured polygons
12
Hierarchical, Multiresolution Methods forHighly
Detailed Urban Scenes (cont.)
View-Dependent LOD
Global quadtree
Bounding box
Eye
Selected LOD
13
Automatic Identification and Placementof Trees,
Shrubs, and Foliage
This can be used with Avideh Zakhors results to
automatically identify, remove, and model foliage.
14
Application to Tree Modeling
Automated identification and modeling of trees
15
Multiresolution, Embedded Vector Features
Road, river, boundary or other vector features
that continuously change LOD as one flies in
Fly-in to Korea
This can be combined with Suresh Lodhas
feature-preserving techniques
16
Multiresolution, Embedded Vector Features
Terrain-following feature
17
Interactive, Hierarchical Rendering of
Non-Uniform Volumes Weather and Uncertainty
Dynamic 3D structure
Volume rendering of reflectivity of severe storm
passing over 2 Doppler radars
Doppler reflectivity over North Georgia
Doppler velocity magnitude
18
Situational Visualization

• Mobile, interactive visualization with real-time
  inputs based on location, orientation, and
  situation
• Awareness based on where you are, where you are
  going, and what you are doing
• Requires mobile computing and new type of
  interface

19
Situational Visualization Applications

• Based on visual computing that you can take with
  you anywhere.
• Nano-Forecaster
• Mobile Surveyor (building personal worldview)
• Traffic Situator
• Situational Awareness Game
• Mobile pathfinder (e.g., entryways for disabled
  people)
• Venue Director
• And many other possible mobile applications

20
Applications Nano-Forecaster
Events severe storm cells, mesocylones, tornado
signatures
x
Tornadic signatures
Mesocyclones over detailed map
x
User position
21
Multimodal Interface Motivation

• For wearable or ubiquitous computing, traditional
  computer interaction often fails
• For wearable or ubiquitous computing, user is
  often employing hands for other tasks
• Ever smaller devices will carry ever larger
  capacity for computing, storage, and networking.
  The interface must be enriched to successfully
  use all this.
• For these reasons...

• We are investigating new interaction techniques
• speech, gesture, and in combination for
  multimodal interaction
• two-handed interaction

22
Implementation

• VGIS,
• a 3D terrain visualization
• environment
• Gesture Pendant, a chest mounted camera and
  gesture recognition software

Gesture pendant (worn on chest)
Infrared lights
Camera with Infrared filter
23
Speech and Gesture

• Speech
• a rich channel for human-to-human communication
• provides rich command and query interfaces
• Gestures
• complement speech with redundancy, emphasis
• provides concise spatial references and
  descriptions
• can be used where speech is inappropriate or in
  use for other actions

24
Multimodal Interfaces

• Multimodal interfaces appear to be well suited
  for spatio-visual interfaces
• May be able to combine the strengths of speech
  and gesture
• Provides a large repertoire of commands, allowing
  users to chose their means of expression
• Mutual disambiguation may allow more robust
  recognition by examining data from both interface
  channels.

25
Implementation

• IBM ViaVoice speech recognition software
• Software to integrate speech and gesture commands

26
Speech Interface

• The speech interface uses IBM ViaVoice
• Recognized speech utterances are time-stamped and
  transferred over the network.
• Limited vocabulary
• Command synonyms

27
Types of 3D Interaction
Selection Navigation Manipulation

• Bowman (Ph.D. Thesis, Georgia Tech, 1999) has
  studied these techniques for 3D interaction in
  virtual environments

28
Navigation

• Complex due to the large range of scales involved
• Methods must work at all scales
• Including scale, seven degrees of freedom must be
  managed
• Constrained navigation in 3 modes (orbital, fly,
  walk)

29
Speech Interface

• Continuous Movement
• Move In, Out, Forwards, Backwards
• Move Left, Right, Up, Down
• Move Higher, Lower
• Discrete Movement
• Jump Forwards, Backwards
• Jump Left, Right, Up, Down
• Jump Higher, Lower

30
Speech Interface

• Speed
• Slower, Faster, Stop
• Direction
• Turn Left, Right
• Pitch Up, Down
• Modes of Navigation
• Orbit, Fly, Walk

31
Gesture Interface

• The Gesture Pendant
• Chest mounted BW video camera
• A series of IR LEDs illuminate the users hand
• A IR pass filter prevents other light sources
  from interfering with segmentation
• Gestures are with respect to the body

32
Gesture Interface

• Gesture recognition software segments the video
  image into blobs, based on preset thresholds
• If the blob conforms to trained height, width,
  and motion parameters, particular gestures are
  recognized.

33
Gesture Interface

• Pan Left/Right
• Zoom In/Out

• Pan Up/Down
• Stop

34
Multimodal Interface

• Users first give a speech command.
• The Gesture Pendant tracks the fingertip,
  allowing gestures to describe the speed of the
  command, i.e. how fast to turn or move.

35
Metrics for Multimodal Interface

• Voice Gesture recognizability and
  responsiveness
• Speed efficient task completion
• Accuracy target proximity
• Ease of learning
• Ease of use
• User comfort

36
Multimodal Task

• Users trained speech recognizer (not necessary in
  latest version)
• Users were shown how to position their hands for
  the Gesture Pendant
• Experimented with multimodal interface

37
Multimodal Task

• The navigation task began in high orbit
• traveled west into the Grand Canyon
• traveled east into Atlanta
• in fly mode, traveled to the Georgia Tech campus
• in walk mode, parked in from of Tech Tower

38
Lessons

• Users could remember both the voice and gesture
  commands and some felt they were easier to learn
  than keystroke commands.
• It is important that commands be mapped to some
  action in every navigation mode. If users try a
  command in the wrong mode and it does nothing,
  users will conclude it does not exist.
• For better results, gesture recognition should be
  improved and lag lessened.
• Alternative gestures should be evaluated.

39
Lessons

• There were some issues with speech recognition
  lag. This has since been improved by restricting
  the vocabulary and grammar and improvements in
  network code.
• Users would sometimes move their hands out of the
  camera field of view. Displaying cursor to
  indicate hand position may address this.

40
Future Work

• Work on faster recognition, especially for
  gesture interface
• Gesture recognition using a neural network
• Larger vocabulary possible
• Developing new version of the Gesture Pendant
• For outdoor use
• Stereo camera pair
• Laser grid and structured light

41
Future Work

• Developing other capabilities for 3D interaction
• Two-handed interface (two twiddlers with
  flywheels and orientation tracking)

42
Results

• We published three papers on this work
• David Krum, William Ribarsky, Chris Shaw, Larry
  Hodges, and Nickolas Faust Situational
  Visualization, pp. 143-150, ACM VRST 2001
  (2001).
• David Krum, Olugbenga Omoteso, William Ribarsky,
  Thad Starner, and Larry Hodges Speech and
  Gesture Multimodal Control of a Whole Earth 3D
  Virtual Environment, pp. 195-200,
  Eurographics-IEEE Visualization Symposium 2002.
  Winner of SAIC Best Student Paper award.
• David Krum, Olugbenga Omoteso, William Ribarsky,
  Thad Starner, and Larry Hodges Evaluation of a
  Multimodal Interface for 3D Terrain
  Visualization, to be published, IEEE
  Visualization 2002. This is a formal user study
  of the multimodal interface.

43
Situational Visualization System
Laptop with wireless Wavelan GPS
Full color, video resolution display
Twiddler
44
Situational Visualization Demonstration
ofEmergency Response to Terrorist Attack
Sarin gas cloud
GPS positions of first responders
Overview and fly-in to attack point on Georgia
Tech campus
Demonstrated to President Bush and Governor Ridge
on March 27, 2002.
45
Next Phase Work

• Scalable multiresolution framework for urban
  landscape geometry and textures (so that one can
  navigate smoothly from a close-up of a building
  façade to an overview of several city blocks).
• Initial work on image-based rendering techniques
  applied to cityscapes (these will be needed for
  overviews when one collects hundreds of city
  blocks with tens of thousands of buildings)
• Use of the mobile geospatial database with
  computer vision techniques to determine position
  and orientation when sensor data are inaccurate
  or missing.
• Further work on uncertainty visualization in the
  VGIS environment

46
Next Phase Work

• Further Situational Visualization applications.
• Continued development and evaluation of
  multimodal interface for 3D interaction.
• Transfer of mobile visualization applications to
  a networked PDA.

47
Additional Publications on this Work

• William Ribarsky, Towards the Visual Earth,
  Workshop on Intersection of Geospatial
  Information and Information Technology, National
  Research Council (October, 2001).
• William Ribarsky, Christopher Shaw, Zachary
  Wartell, and Nickolas Faust, Building the Visual
  Earth, Vol. 4744B, SPIE 16th International
  Conference on Aerospace/Defense Sensing,
  Simulation, and Controls (2002).
• William Ribarsky, Tony Wasilewski, and Nickolas
  Faust, From Urban Terrain Models to Visible
  Cities, to be published, IEEE CGA.
• Justin Jang, William Ribarsky, Chris Shaw, and
  Nickolas Faust, View-Dependent Multiresolution
  Splatting of Non-Uniform Data, pp. 125-132,
  Eurographics-IEEE Visualization Symposium 2002.

48
User Studies of Multimodal Interface (cont.)
Demonstration of use of gesture pendant to
recognize hand gestures
User task objects are located, navigated to, and
identified.

• Results
• Mouse interface has best performance, then speech
  alone, multimodal, and gesture alone.
• When mouse is not available or easy to use, a
  speech interface is a good alternative for
  navigation tasks.
• Better, faster recognition of gestures could
  significantly improve performance of the
  multimodal interface.