FT

1
FT

• 21.febrúar 2008

2

• Kari Hardarson
• Masters degree from University of North Carolina
  at Chapel Hill, USA
• I teach at Reykjavik University

3
Where I come in

• A research group at Reykjavik University
  (http//gagnasetur.ru.is/) could use better
  visual displays of quantiative information
• Good visual representations can be invaluable

4
One of the best examples is French
5
..taken from this book
http//www.edwardtufte.com/tufte/
6
(Picasa looks like this)
7
Then theres PhotoSynth
8
PhotoSynth
9
PhotoSynth
10

• Millions of people are taking thousands of
  photographs
• Surely, research into image browsing could be
  time well spent?

11
My (1st) idea

• Allow the user to pick any three dimensions
  from an image database containing thumbnails an
  numerical data about each image
• Display the images in a space defined by these
  dimensions
• This could be thought of as visual OLAP

12

• I extracted
• image data from the pictures themselves
• Metadata provided by the camera (EXIF)
• and placed in a database along with thumbnails

13
Sample of result

• The next picture is a collection of photo-
  graphs, taken at one hour intervals of
  Laugardalur valley in Reykjavik, during the
  course of one year
• The three dimensions are
• Day of year
• Hour of day
• Absolute Brightness

14
Can your browser do this?
15
365 days x 24 hours x absolute brightness
16
Closeup
17
The next picture

• Photos taken in Reykjavik, ordered by red, green
  and blue component

18
red x green x blue
19
Closeup(all the football pictures end up along
the green axis)
20

• What can I do about displaying local descriptors
  for Dr. Bjorn?
• I need to add other ways of visualizing
  organizations in the data
• Textual tags on images
• Hierarchy trees or graphs
• Ranking non euclidian distances
• Support for pivoting, axis stretching, selection
  of ranges

21

• If three dimensions are not useful, the browser
  should still be as useful as Picasa
• Picasa has two dimensions and time ranking only

22

• Then I went to France...

23
So what have you been doing with yourself in
France?

• Running with the local running team
• Bicycling
• Eatings saucisson sec and fromage
• Eating Gallettes completes
• Learning French
• ..and trying to write a prototype at IRISA

24
Galettes
25
Saucisson Sec
26

• A dedicated computer science research institute
  in Bretagne
• 600 people
• Runs the gamut of computer science from robotics
  to peer-to-peer and
• Connected to University of Rennes 1

27
Ideas to take home from IRISA

• Linux used everywhere
• Remote admin, no admin privileges, no viruses, no
  trojans, it just works. Special is OK but you
  have to ask, otherwise everything is uniform.
  Everything compiles the same everywhere. LaTex,
  all utilities are there. /usr/bin is not local.
• The cafeteria has a command line interface
• Everything is paid with wireless card
• Closets are whiteboards
• Library has a good video and cartoon book
  selection
• Welcome program for foreigners works
• Web pages are in English!
• I got my keycard and account the same morning

28
Whiteboards!
29
The mission

• Today I have more than 15 000 digital photos
  increasing by about 1 000 every other month on my
  computer, all in different folders. There's no
  system to help me. And I'm getting frustrated
  with the situation.
• Photo Retrieval Multimedia's Chance to solve a
  real problem for real people by Forouzan
  Golshani, Wright State University

30
Why don't users annotate?

• It's not important enough?
• If annotation was pleasant, then maybe they would
  ?
• Will you invest work in an application which
  won't even let you export your data?
• It's not hard to improve on the situation

31
Possible solutions

• Let the computer annotate them (Not possible, it
  needs smarts)?
• Make it really, really pleasant
• Make it really easy to browse so annotation isn't
  as necessary

32
What did people do with their printed photos
earlier?

• Spread them on a big desk and make piles!
• Pile of christmas pictures
• Pile of friends pictures
• Sort them, sort of
• Then put the piles back in the shoebox order
  was lost!
• Or
• Put them in a photo album, in time order with
  chapter headings (if you were work at home mom)?

33
Photo Piles
34
How do current computer solutions work?

• Picasa is the best of breed.
• Only one axis time
• Only one layout 5 x n field of thumbnails
• If you pay extra you get database capabilities
  select by criteria, but context is lost when you
  view the search results
• Double click on thumbnail and enter text
• Other approaches exist, like...

35
...Photo Mesa
36
My priorities

• Important to maximize information - to - ink
  ratioUse the screen well don't decorate too
  much
• People are good at seeing patterns,make use of
  it.
• Let people browse, show them lots of photos
• Use picture location to annotate, not button
  clicks and property windows

37
Bad info to ink ratio
38
The tyranny of the timeline

• The timeline is very often a good way to look at
  your photos, but it should not be the only one.
• If I want to look at my photos along another
  dimension then the browser should let me.
• Special photo collections such as time-lapse
  photos or car photos may benefit from a more
  precise specification of the available dimensions
  on the screen.

39
Zoomable user interface

• The user should be able to seamlessly zoom in on
  thumbnails until they become the picture itself.
• Zooming out can also make good sense if the
  thumbnails contain large objects that are
  recognizable from a distance.
• The eye is very good at scanning a large number
  of items at a glance.

40
Drag and drop annotation of thumbnails

• It should not be necessary to click on a
  thumbnail and type text on the keyboard to
  annotate it, when the annotation can be done
  using a drag and drop method.
• We are used to classifying things by moving them
  between piles on a desk.

41
Integrated browsing and searching

• When the user searches for images in most
  browsers, the browser returns a flat list of
  retrieved matches. The pictures are thereby
  removed from their context.
• I want to return search results without removing
  the matched images from their surroundings, by
  using the Third Dimension.

42
Is 3D better than 2D ?

• Space games can be very disconcertingThe pilot
  loses orientation when flying upside down.
• Can be used, for example in album lookup in the
  iLife browser on iMac and iPod.
• Must be used judiciously

43
3D

• A Natural way to implement a ZUI
• A way to emphasize some thumbnails if they
  match, move them forward

44
Another example
45
OLAP

• Can I create some sort of visual OLAP?
• Olap is a way to browse information which has
  many dimensions Who sold what to whom where
  and why?

46
An OLAP Cube
47
A prototype?

• First I was going to use C and OpenGL. This was
  a trip up the wrong tree, because both tools are
  too low level for prototyping
• Then I looked at higher level libraries
• Ogre, OpenSceneGraph, Open Mask (Thierry Duval at
  IRISA)?

48
How does one write 3-D?

• Work with raw polygons or use Scene Graphs
• A scene-graph is a general data structure used by
  modern computer games.
• The scene-graph is a structure that arranges the
  logical and spatial representation of a graphical
  scene.
• A game might define a logical relationship
  between a knight and a horse so that the knight
  is considered an extension to the horse. The
  scene graph would have a 'horse' node with a
  'knight' node attached to it.

49
Now I am using Python, MySQL and VTK toolkit
50
Python Slow ?

• Yes but fast enough for a prototype
• Proof of concept is enough, I don't want to
  display 10 000 pictures at the same time.

51
Axes, Properties

• Can be of types
• Magnitude (Think Ruler)?
• Rank (Sorted list)?
• Stack (Think pile)?
• Properties
• String
• Integer
• Floats

52
Magnitude axis
0 10 20 30 40 50 60 70 80 90 Horsepower
53
Magnitude Axis with intervals
10 51 102 155 223
10-49 50-99 100-149 150-199 200-249
54
Rank Axis (order)?
10 11 17 21 55 67
Maximum length optional
55
Stack Axis (no order)?
Maximum length optional
56
Not just for numbers
Stack
Rank
Laugavegur
Lækjargata
Frakkastígur
Fríkirjuvegur
Hringbraut
Suðurgata
Hverfisgata
57
Picasa 8 rank x unlimited rank
58
The underlying data model

• List of image filenames
• List of attribute names
• A list image,attribute,value
• An image may have an arbitrary number of
  attributes.
• An image may have the same attribute twice.

59
Baby steps

• First I was going to write a complex data
  structure myself.
• Then I thought Why not SQL ?
• Added benefit If the data is open, this may
  cause people to want to invest their time in
  maintaining their photo collection. A photo
  collection is a long term project which may
  outlive many browsers

60
Also

• A 3-D cube in memory which remembers which
  voxels are occupied by a thumbnail. It may
  become an SQL table if I store views.

61
A necessary optimization

• If the cloud of images is very big, then display
  only stars if far away, load the thumbnails
  when the stars come closer.
• Use an oct-tree as a data structure to store the
  images.

62
My little language

• dring1751python prada.py
• Prada shoebox - for great shoes - and photos.
• Pradagt help
• Documented commands (type help lttopicgt)
• addfolder axis exit filter info
  listattr shell
• attr axisorder exportview help join
  makeview thumbs
• Pradagt

63
My little language

• Pradagt help axis
• axis xyz
• magnituderankstack
• attribute from the database
• nameattribute if not specified
• interval1.0, denotes increments
  between ticks on the axis
• magnifier1.0, magnification of
  attribute value if any
• Defines one of the three axes
• Pradagt

64
My little language

• Pradagt help attr
• attr result setassignask attribute
• Sets an attribute on each image in a result
  set.
• Examples of use
• attr 4 set hp
• attr 4 ask hp
• attr 4 assign hp 120
• Pradagt

65
My little language

• Pradagt help filter
• filter result attribute
  rangeequalexists args
• Narrows a result set down. Result set 0 is
  the entire database.
• Examples of use
• filter 1 dagur range 2007-12 2007-13
• filter 1 f-stop equal 4
• filter 1 f-stop exists
• Pradagt

66
Sample session

• dring1805python prada.py
• Prada shoebox - for great shoes - and photos.
• Pradagt filter
• filter result attribute
  rangeequalexists args
• Narrows a result set down. Result set 0 is
  the entire database.
• Examples of use
• filter 1 dagur range 2007-12 2007-13
• filter 1 f-stop equal 4
• filter 1 f-stop exists
• Pradagt filter 0 ExifShotDay equal 20050812
• New result is number 1 , it has 0 pictures.
• Pradagt filter 0 ExifShotDate equal 20050812
• New result is number 2 , it has 39 pictures.
• Pradagt thumbs 2
• Generating thumbnail 2.jpg

67
Result thumbnail
68
You open your entire database and no axes are
defined
69
All the images end up as a smear at the origin so
you decide to use x to display time
70
You decide to use the x axis for dates and y axis
for time-of-day
71
Images are still overlapping, this time on the
y-axis, so you switch to a ranking y-axis
72
You zoom out to see the entire year
73
You select Christmas
74
And get a new view
75
You set x and y axes to stack
76
You want to see which pictures are in hi-res so
you use the z-axis to show you the image size
77
Some of the images jumped backwards so you look
at the images from above to see them better.
78
Finally, you click Assign Attribute and label
them with the string High resolution Christmas
2006
79
Things to finish

• Add search ability, use the z-axis or make
  matched thumbnails like Bright stars
• Add new properties to selected images on-the-fly
• Allow editing of the properties by moving the
  images in the x-y plane
• Allow opening of new window based on selected
  items in existing window

80
Future work

• I need to switch to C and Open GL I think.

81

• - Fin -
• (The end)?