Query Interfaces for a 3D Model Search Engine

1
Query Interfaces fora 3D Model Search Engine

• Patrick Min
• Tom Funkhouser, Misha Kazhdan, Joyce Chen, Alex
  Halderman, David Dobkin, David Jacobs
• DLI2 meeting, Edinburgh, 25 June 2002

2
Online Data

• On-line multimedia data is changing the way we
  get and use information

Registered Saddlebred out of Famous Sultan
Supreme line. 100 sound. 16 year old, flashy,
chestnut w/white, loving, high energy horse,
needs experienced rider. Was shown professionally
in early years as gaited saddlebred. Most
recently shown and always placed in training and
first level dressage shows.Currently used as
dressage/pleasure horse, jumps, loves
trailriding.
www.dreamhorse.com
2D Image
Text
Audio
What about 3D Models?
3
3D Models

• are becoming easier to create, use and distribute

4
3D Models

• are becoming easier to create, use and distribute
• Laser range scanners supply

5
3D Models

• are becoming easier to create, use and distribute
• Laser range scanners supply
• Fast graphics cards demand
• World Wide Web distribution

6
3D Models

• are becoming easier to create, use and distribute
• Laser range scanners supply
• Fast graphics cards demand
• World Wide Web distribution
• One day 3D models will be as ubiquitous as
  images are now

Georgia Tech
Protein Data Bank
Cyberware Scanner
Large Model Archive
7
Research Challenges

• Current research
• How do we construct 3D models ?

8
Research Challenges

• Current research
• How do we construct 3D models ?

• Future research
• How do we find 3D models ?
• How do we understand them ?

9
Finding 3D Models
10
Project Goal

• A Search Engine for 3D Models on the Web

car
11
3D Model Search Engine
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
12
3D Model Search Engine
Acquisition
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
13
3D Model Search Engine
Analysis
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
14
3D Model Search Engine
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
Matching
15
3D Model Search Engine
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
Querying
16
3D Model Search Engine
Analysis
World Wide Web
Crawler
Repository of 3D Models
Crawler
Indexer
Crawler
Crawler
Off-line
3D Index
2D Index
Text Index
On-line
Query
QueryInterface
Text Matcher
2D Matcher
3D Matcher
User
Matches
Querying
Matching
17
Main Challenge today's topic

• Which Query Interfaces are Useful
• for a 3D Model Search Engine?

Query
QueryInterface
User
Matches
18
Desired Properties of Query Interfaces

• easy to use
• also for novice users
• possible to quickly specify query
• efficient matching method
• effective
• high number of relevant results

19
Available Data for Queries
Example of a 3D Model available on the Web
http//www.3dspline.com
20
Available Data for Queries
Text
Shape
Appearance
21
Available Data for Queries
Text
Shape
Appearance
22
Text Queries
23
Text Queries

• easy to use
• possible to quickly specify query
• efficient matching method
• /- effective

24
Effective Text Queries
747
dna
caffeine
dog
fork
25
Text Query Problems

• objects are badly annotated
• ferrar03.3ds, ferrair, rahmen,
• keywords are too common
• face, plane

26
Text Query Problems

• objects are badly annotated
• ferrar03.3ds, ferrair, rahmen,
• keywords are too common
• face, plane
• keywords are unknown to user, or annotator
• not possible to anticipate every relevant text
  query
• elf, fairy-tale creature, puppet, person, ...

27
Text Query Problems

• objects are badly annotated
• ferrar03.3ds, ferrair, rahmen,
• keywords are too common
• face, plane
• keywords are unknown to user, or annotator
• not possible to anticipate every relevant text
  query
• elf, fairy-tale creature, puppet, person, ...

• We hypothesize shape can help

28
Outline

• Introduction
• Text Queries
• 3D Shape Queries
• 2D Shape Queries
• Results
• Conclusions
• Future Work

29
3D Shape Queries
use 3D shape to find 3D shape
30
3D Model Queries

• (create ) upload 3D model
• select 3D model on results page
• ("shape iteration")

31
3D Matching Method
32
3D Matching Method
33
Shape Iteration User Study
34
Shape Iteration User Study

• How does shape combine with text in interactive
  searches?

• Test Design
• 18 students in an introductory CS class
• input on website, initial text keyword search,
  then find similar object
• where similar means similar text or similar shape
  (unknown to user)

35
Shape Iteration Results
Similarity

• shape iteration finds target object more often
• in fewer iterations and in less time

36
3D Model Query Problems

• upload 3D model
• such a model may not be available
• creating one with standard tools is hard
• shape iteration
• have to get such a result first

37
3D Sketch Queries

• Sketch 3D model using simple modeling program
• e.g. Teddy, Sketch

Teddy A Sketching Interface for 3D Freeform
Design Igarashi et al., SIGGRAPH 1999
38
3D Sketch Queries
Teddy A Sketching Interface for 3D Freeform
Design Igarashi et al., SIGGRAPH 1999
39
Teddy Sample Results
40
3D Sketch Query Problems

• Teddy
• supports only coarse, blobby shapes of genus 0
• still hard to use for novice users

41
3D Sketch Query Properties

• - easy to use
• - possible to quickly specify query
• efficient matching method
• /- effective

• So we need a simpler query interface

42
2D Sketch Queries
43
2D Sketch Queries

• paintbrush style interface
• simple outline sketches
• easy to use for most users

44
Multiple 2D Sketches
45
2D Sketch Matching Method
5
3
6
4
7
1
2
Query sketch
7 projections per database model
46
2D Sketch Matching Method

• compute 2D shape descriptor in similar fashion
• intersect with circles instead of spheres

47
2D Sketch User Study
48
2D Sketch Interface User Study

• Does shape add to text?
• Test design
• 43 students in introductory CS class
• goal retrieve target objects
• input text keywords and sketches on paper

49
Example Input
50
2D Sketch User Study Results
51
2D Sketch User Study Results

• Class is too large
• text is too inspecific
• sketch works best

52
2D Sketch User Study Results

• text and shape can complement each other
• sketch allows refinement within a class

53
2D Sketch Query Problems

• The drawing is free-form
• hard to draw accurately
• hard to get proportions right

54
2D Sketch Query Problems

• The drawing is free-form
• hard to draw accurately
• hard to get proportions right
• Image matching method
• user and database outline have to look similar

55
Search Engine Usage Results
56
Search Engine Results

• online since early November 2001
• 20,707 models indexed, 2,873 commercial
• one typical week of usage
• 4,522 queries
• 1,346 unique hosts from 55 different countries
• 1,029 models downloaded

57
Search Engine Results
Text vs. Shape

• about 70 text, 30 shape-based

58
Search Engine Results
Shape Query Type

• about 58 shape iteration, 38 2D/text 2D

59
Number of Hosts per Day

• average about 175 hosts and 525 searches/day

60
New Hosts per Day

• about 20 returning hosts per day

61
Searches per User
hosts
7000
6000
5000
4000
3000
2000
1000
1 3 5 7 9 11 13 15 17 19
21 23 25
searches

• about 40 does 3 or more searches per day

62
Conclusions and Future Work
63
Conclusions

• shape-based search can help
• each interface has its strengths and weaknesses
• so useful to combine them

64
Future Work

• 2D Structural Interface
• primitive drawing user study
• evaluation of several interfaces
• oval rectangle
• skeletal graphs
• 3D Structural Interface

65
Future Work II

• 3D Model Database
• fully automated acquisition and model processing
• support for more model formats

66
Acknowledgements

• Funding
• NSF (CCR-0093343, 11S-0121446, CCR-99-88173,
  DGE-9972930)
• Army Research Organization (DAAD19-99-1-0205)
• Alfred P. Sloan Fellowship (for Prof. Tom
  Funkhouser)
• People
• Brian Kernighan and the students of CS109 and
  CS111
• Bernard Chazelle, Adam Finkelstein,
• Szymon Rusinkiewicz, Emil Praun
• Princeton CS department technical staff
• Princeton Shape Analysis Group
• 3D Models
• Viewpoint Datalabs, http//www.viewpoint.com
• De Espona Infografica, http//www.deespona.com

67
http//shape.cs.princeton.edu
68
(No Transcript)
69
3D Model Search Engine
70
Test Database

• 50 models, 10 categories of 5 models each