Visualizing Image Datasets with Covise

1
Visualizing Image Datasets with Covise

• Osaka University, Japan
• Ava Pierce
• David Jackson
• PRIME 2007
• Cal-(IT)2
• University of California, San Diego

2
Project Motivation

• Scientists want
• Intuitive viewing of very large images
• Intuitive manipulation of 3D objects
• Tools to better analyze and express these large
  images and objects
• Ways to work collaboratively internationally
  using these data sets
• Developers of Covise want to make the scientists
  life easier

3
Project Motivation

• Developers of Covise have
• An Image plugin to view and manage large images
• Other plugins made to help researchers
• Developers of Covise want
• A user interface to load images and show extra
  information about the images
• An user interface for all Covise plugins

4
Project Motivation

• Solution, File Browser Plugin, which can
• Graphically show all files in a directory
• Navigate to other directories
• Load any Covise file to their respective plugins,
  including images
• Be easy and intuitive to use
• Show extra information about images being loaded

5
Open Scene Graph (OSG)

• Covise uses this open source 3D graphics toolkit
• Uses C and OpenGL to create scene graphs
• A scene graph provides a tree-like object
  oriented framework around OpenGL

6
What comprises a scene graph

• The leafs of the tree are called nodes, which can
  be many things, but most importantly
• Root Node reference to top of the tree
• Geode geometry-node, that represents an object
  of some kind to be placed on the screen
• Matrix Transform A 4x4 matrix node that
  expresses positional, rotational and scaling
  relationships between two nodes
• Group Node Allows special manipulations between
  its child nodes

7
What comprises a scene graph
Root node
geode1s parent node
Group Node
Matrix Transform
Switch between the two
Group nodes first child
geode1
geode2
geode3
8
What comprises a scene graph

• Nodes can
• be removed and added to parent nodes
• Have multiple children and/or multiple parents
• be hidden or visible
• be switched with other children
• This makes them very useful for interactivity and
  for creating a smart user interface

9
The OSG Geode

• Geodes can consist of many different drawables,
  including
• 3D or 2D geometries
• Text
• Textures, including 2D images, or 3D maps
• Different material properties and states
• Ex, color and lighting properties

Not THIS geode!
10
Matrix Transform ExampleCar Turning Right
the road (root)
Car (position/ rotaion)
Car (group)
steering wheel (position/ rotation)
Car body (geode)
steering wheel (geode)
Steering wheels motion determined by
the multiplication of two matrix transforms
11
Intersection Testing though OSG

• OSG has a built-in function called Intersection
  Visitor
• Tests to see if a line passes through a geodes
  geometry
• Modified so line is the mouse or tracker pointing
  through covises 3D space
• Customized intersection tester returns
• Geode that it intersected with
• Point in space it intersected
• Boolean check if it did or did not intersect with
  something

12
Intersection Testing though OSG
Virtual intersection line
Mouse pointer on screen intersects with a
cylinder in 3D virtual space
13
Covise Interactions, OSG Intersections

• Used built-in Covise interactions to determine
• Left mouse click
• Left mouse hold (dragging something)
• Left mouse release
• Used built-in OSG interactions to determine when
  the mouse is over
• A thumbnail image
• The main panel
• The scroller buttons

14
File Browser Plugin SceneGraph
World Pos
attached through COVISE
Scene root
locked
not locked

Image 1
Image n
Panel Pos
Scroller Group
Loader Pos
Histogram Pos
Panel Group
Loader Group
Main Panel
Triangle Buttons
Histogram Group
Thumbnail Pos n

Thumbnail Pos 1
Loading Bar
Invisible Gradient
switch
Histogram
Loader Panel
Thumbnail n
Thumbnail 1
Info Panel Pos
Histogram Gradient
Loader Text
root
matrix transform
Info Panel
group
geode/other
15
File Browser Plugin SceneGraph
Scene root
Loader Pos
Loader Group
Loading Bar
Loader Panel
Loader Text
root
matrix transform
group
geode/other
16
File Browser Plugin SceneGraph
Scene root
Histogram Pos
Histogram Group
Histogram
Histogram Gradient
root
matrix transform
group
geode/other
17
File Browser Plugin SceneGraph
Scene root
Scroller Group
Triangle Buttons
Invisible Gradient
root
matrix transform
group
geode/other
18
File Browser Plugin SceneGraph
Scene root
Panel Pos
Panel Group
Main Panel
Thumbnail Pos n

Thumbnail Pos 1
switch
Thumbnail 1
Thumbnail n
Info Panel Pos
Info Panel
root
matrix transform
group
geode/other
19
File Browser Plugin SceneGraph
World Pos
attached through COVISE
Scene root

Image 1
Image n
root
matrix transform
group
geode/other
20
File Browser Plugin SceneGraph
World Pos
attached through COVISE
Scene root
locked
not locked
Image 1
Image n

Panel Pos
Scroller Group
Histogram Pos
Loader Pos
Panel Group
Loader Group
Triangle Buttons
Main Panel
Histogram Group
Thumbnail Pos n

Thumbnail Pos 1
Loading Bar
Invisible Gradient
switch
Histogram
Loader Panel
Thumbnail 1
Thumbnail n
Info Panel Pos
Histogram Gradient
Info Panel
Loader Text
root
matrix transform
group
geode/other
21
Functionality
Main Menu
Browser Menu
Loaded image (via image plugin)
Histogram of image
Right scroller arrow
Thumbnail (actual thumbnails not shown)
Info panel name, dimensions, size
Main panel
22
Accessing the File Browser Plugin

• File browser plugin is built into covise
  structure, so can be accessed from the main menu
• This will open a sub- panel
• To open the browser, click the Open/Close
  Browser option

23
Covise config.xml file

• Input parameters to initialize the browser plugin
• Dir set current directory to open
• FileHeight, Filewidth set the size of each
  thumbnail
• Spacing set space between thumbnails, and border
• Bufferspace set offset of panel from bottom left
  corner
• ScollerSize set width of scroll buttons
• TextSize set size of text
• HistSize set size of histogram square
• Allows the plugin to be fully customizable

24
File Browser Plugin
scrollerSize
bufferSpace
spacing
fileWidth
Browser is opening Use of these parameters are
throughout the plug-in, not just the spaces noted
25
Loader bar while opening
Scene root

• Opens while placing thumbnails on panel
• Displays percentage in terms of fraction of
  thumbnails loaded in the given directory
• When all files are loaded, removes itself from
  the tree

Loader Pos
Loader Group
Loading Bar
Loader Panel
Loader Text
26
Main Panel
Scene root

• Sizes and positions itself to fit all thumbnails
• Interact with panel by
• Dragging it horizontally
• Highlighting over it to change border color
• Interactions are done through Covises built-in
  functions

Panel Pos
Panel Group
Main Panel
Pointer off panel, border gray
Pointer on panel, border green-gray
27
Dragging the main panel
Scene root

• An open scene graph is in 3D space, and the mouse
  pointer is also in 3D space
• However, only wanted to move the panel in the
  x-direction
• Had to take the Matrix Transforms of the mouse
  and panel, and compress to one dimension
• The result is the panel only moves horizontally
  on the screen
• Also, made it so once the drag interaction has
  started, the user can drag the panel even with
  the mouse off the panel
• The panel will be dropped on mouse release

(This matrix is changing during dragging)
Panel Pos
Panel Group
Main Panel
Dragging the panel to the right, off the screen
28
The scroller
Scene root

• On mouse-over of the side arrows, the panel will
  scroll left or right
• This action will alter the same Panel Pos Matrix
  transform as when you drag the panel
• Gradient speed
• Depends on how far to the left or right you are
• Can set the base speed of the scroll in the side
  panel
• That speed is multiplied with the cursor position
• Will also turn green-gray on panel mouse-over

Speed x1 x2 x4
Scroller Group
Triangle Buttons
Invisible Gradient
29
Histogram

• A histogram is a visualization of rgb pixel
  values in an image
• Can also be grayscale
• Added when user clicks on an image thumbnail in
  the browser, created using ImageMagick
• Size of histogram determined in config file
• Still need to work on interactions with the
  histogram window, so it can be moved around

Scene root
Histogram Pos
Histogram Group
Histogram
Histogram Gradient
A completely dark gray image
A very dark, reddish image
30
Info Panel
Scene root

• Attaches to thumbnail that mouse is over
• Three matrix transforms are required to put the
  panel in the correct spot above the chosen
  thumbnail
• So far, shows three pieces of image information
• Name
• Dimensions (pixel width x height)
• Size of image
• Metadata is found using ImageMagick
• Currently, just checks to see if thumbnails file
  is or isnt an image

Panel Pos
Panel Group
Main Panel
Thumbnail Pos n

Thumbnail Pos 1
switch
Thumbnail 1
Thumbnail n
Info Panel Pos
Info Panel
31
Info Panel
Scene root
Panel Pos
Panel Group
Main Panel
Thumbnail Pos n

Thumbnail Pos 1
switch
Thumbnail 1
Thumbnail n
Mouse-over deer.jpg
Mouse-over a directory
Info Panel Pos
Info Panel
32
Browers Sub-menu options
Open/Close Browser Adds/deletes all nodes
shown, and loads all thumbnails each time
Browser Visible Hide the browser, which means
the browser does not have to be re-loaded
Autoscroll Enable/disable the scroller
function, in case you dont want it to
automatically move Additional Info Toggle
the extra info panel on and off. Turning it off
may make other options run smoother
Histogram Toggle the histogram on and off
Lock Browser Attaches the panel either to the
screen or world position. If it is not locked,
it can be moved around with Covise Set Speed
Dial Sets the basespeed for scrolling If you
have many files, you may want to cycle through
them faster
33
Unlocking the browser

• With more advanced 3D tracking input devices
  (unlike a 2D mouse), you may want to move the
  panel around in 3D
• However, have not yet gone into all of the
  situations in pure 3D

34
Summary of File Browser Plugin
Main Menu
Browser Menu
Loaded image (via image plugin)
Histogram of image
Right scroller arrow
Thumbnail (actual thumbnails not shown)
Info panel name, dimensions, size
Main panel
35
Next Steps

• Improve ease of use of file browser
• Also, more visually appealing
• Improve areas of the browser that cause Covise to
  slow down a little
• Add a directory tree system to visually navigate
  to any directory
• Add functionality to the browser, so any Covise
  file, from any plugin can be loaded
• So far, only Image files are loaded

36
Culture