Using Tweek to Create Graphical User Interfaces in Virtual Reality

1
Using Tweek to Create Graphical User Interfaces
inVirtual Reality

• Patrick Hartling
• IEEE VR 2003

2
Overview

• Problem description
• Goals for Tweek software
• Design overview
• Implementation overview
• Future work

3
Problem Description

• Interaction in VR is primarily spatial
• Not all interactions map well to this method
• Alphanumeric input
• Fine-grained interaction
• List selection
• Per-application custom interfaces
• Map abilities of specific devices to interaction
• Typically use 3D input devices
• Why abandon effective 2D interaction techniques
  in 3D space?

4
Motivation

• Why have GUIs in virtual reality?
• GUIs are everywhere in the computing world
• Most computer users understand how to use a GUI
• Why not have GUIs in VR?

5
Issues with GUIs inVirtual Reality (1/2)

• Positives
• Typically more flexible and more extensible than
  other input devices
• More input methods means less overloading of
  button presses, gestures, etc.
• More portable between VR systems than hardware
  input devices
• Familiarity from desktop experience

6
Issues with GUIs inVirtual Reality (2/2)

• Negatives
• No direct physical correspondence
• In VR, a user could grab an object with his/her
  hand or use a GUI
• No GUI in real world for moving the object
• Remote users may not see GUI Park00
• Use of the interface may be difficult
• Collision detection
• Placement
• Immersive interfaces have to re-invent GUI
  technology

7
Previous Work (1/3)

• Many areas of previous work
• Widget sets for VR
• PDAs as input devices
• Floating menus and heads-up displays
• Virtual User Interface (VUI)
• Originally developed at Iowa State University
  Heath98
• 3D immersive user interface
• May be an open source release from Fuel Tech

8
Previous Work (2/3)

• Java Interface to Virtual Environment (JAIVE)
• Used Java interface on handheld computer Hill00
• Communicated with C applications
• Interface could be extended from C using JAIVE
  protocol

9
Previous Work (3/3)

• PDA in a CAVE system
• Kent Watsen, Naval Postgraduate School
• Experimented with use of Palm PDA in CAVE
  Watsen99
• 3DI Group
• Doug Bowman, Virginia Tech
• Investigating differences in system interaction
  techniques between VE displays
• Active on 3DUI mailing list

10
Tweek

• Framework for the development of cross-platform,
  cross-language, scalable GUIs
• GUIs communicate with remote applications
• GUIs can be use on the desktop, on a PDA, or in
  an immersive VR environment
• Features
• GUI panels are dynamically loaded components
• Panels use standard 2D GUI widgets
• Dynamic interface update and extension
• Multi-language capabilities

11
Goals

• Allow GUI scalability
• Same GUI runs on the desktop, a PDA, and a
  projection-based VR system
• Must provide an easily extendable framework for
  cross-platform, cross-device GUIs
• Interact with VR applications written in any
  programming language
• Fully separate the GUI from the application
• Reuse existing GUI technology
• We do not want to re-invent scrollbars, menus,
  etc.
• Avoid cluttering immersive space

12
Same GUI, Multiple Uses (1/2)

• PDA in Projection System
• Offers force feedback
• High-resolution display
• Back-lit display is needed
• Widely available
• A person with a PDA can walk into the VR system
  and get the GUI dynamically

13
Same GUI, Multiple Uses (2/2)

• Desktop simulation
• Standard GUI interface used to control VR
  software in simulator mode
• Immersive 3D space
• VRJ VNC incorporates 2D GUI into 3D visualization
• Modeled after 3Dwm (www.3dwm.org) use of VNC and
  other VR VNC viewers

14
Design Overview (1/3)

• Distributed Model/View/Controller

15
Design Overview (2/3)

• Observer design pattern
• Simplified MVC subject, observer
• VR application is the subject
• Maintains user-defined state information
• Per-application customizations
• GUI is the observer
• Provides a view of subjects state
• GUI interactions manipulate application state
• Benefits
• Distinct separation of application and GUI
• Allows multiple GUIs to observe the same
  application

16
Design Overview (3/3)

• Two-way communication between application and GUI
• Application can send data to the GUI
• GUI flexibility
• A GUI can load any component it needs to make
  interaction more effective
• Not limited to direct manipulation of application
  state

17
Implementation Overview

• Language tools
• C VR applications
• Java-based GUI composed of JavaBeans
• Beans can be pushed to the GUI while the user
  is in the virtual space
• Dynamic extension of interface
• CORBA communication between C and Java

18
Example Tweek Interface

• Multi-purpose tool
• Navigation
• Map visualization
• Data visualization
• Information storage to spreadsheet-friendly format

19
VRJ VNC Embedding 2D GUIs in 3D Space

• Embed any 2D GUI in 3D world
• Uses VNC
• Can be added to existing VR Juggler applications

20
VRJ VNC Presentations in VR

• Bring PowerPoint presentation into the 3D
  environment

21
VRJ VNC Embedding Tweek

• Tweek Java GUI is just another window on a
  desktop
• Set VNC desktop size to give the desired embedded
  frame size
• Interaction with the GUI is translated to the
  desktop via VNC

22
Future Work

• Use Java Applets to allow Tweek to be used from a
  web browser
• Make use of a familiar interface
• Easy access to application GUI for remote
  demonstrations

23
Online Information

• 3DUI
• http//www.mic.atr.co.jp/poup/3dui.html
• Tweek
• http//www.vrjuggler.org/tweek/
• Contains publications, guides, and programmer
  references

24
References

• Park00 Park, K. et al. Lessons Learned from
  Employing Multiple Perspectives in a
  Collaborative Virtual Environment for Visualizing
  Scientific Data. In Proceedings of ACM
  Conference on Collaborative Virtual Environments,
  San Francisco, California, September 1012, 2000.
• Heath98 Heath, D. Virtual User Interface
  (VUI) A Windowing System for VR. In Proceedings
  of 2nd Immersive Projection Technology Workshop,
  Ames, Iowa, May 1112, 1998.
• Hill00 Hill, L., Usability of 2D Palmtop
  Interaction Device in Immersive Virtual
  Environment. Masters thesis, Iowa State
  University, Ames, IA, 2000.
• Watsen99 Watsen, K. A Handheld Computer as an
  Interaction Device to a Virtual Environment, In
  Proceedings of 3rd Immersive Projection
  Technology Workshop, Stuttgart, Germany, May
  1011, 1999.