Improving the search experience with categorized overviews

1
Improving the search experience with categorized
overviews

• Bill Kules
• Takoma Software, Inc.
• University of Maryland, College Park
• wmk_at_acm.org
• February 27, 2006
• NFAIS Annual Conference

2
Outline

• Exploratory search
• Categorized overview examples
• Design principles for exploratory search
  interfaces
• Comparative study
• Conclusion

3
Marchionini, G. (2006), Exploratory search From
finding to understanding, CACM, 49(4).
4
Why do exploratory search

• Limitations in user knowledge
• Limited domain knowledge
• Imprecise or evolving information need
• Exploratory tasks
• Legal precedents involving non-citizens being
  mistreated by law enforcement officials
• System limitations
• Photo search w/o metadata or annotations

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
SWISH Dyna-Cat
11
SWISH Dyna-Cat
12
SWISH Dyna-Cat
13
SWISH Dyna-Cat
14
SWISH Dyna-Cat
15
SWISH Dyna-Cat
16
What makes a good exploratory search interface?

• Initial ideas
• Enhance user control
• Orient searchers
• Help connect
• What they see
• What they know

17
Adding categorized overview

• Enables lightweight interaction
• Cognitive recognition vs. recall
• Physical mouse move/click vs. typing

18
Adding categorized overview

• Exposes meaningful domain structure
• A roadmap
• Increases connections between cognitive
  representations of
• User need
• Search results

19
Changes in search strategies

• Use category knowledge
• Predicting utility of search results
• Individual results
• Overall
• Novelty
• Expected vs. unexpected results
• Patterns
• Stability ? Benefits from learning
• More decisions

20
(No Transcript)
21
Design principles for exploratory search
interfaces

• Provide categorized overviews
• Organize by meaningful, stable categories
• Provide example documents for each category
• Arrange important text (title, snippet, URL) for
  scanning and skimming
• Support multiple kinds of categories
• Support multiple kinds of visual displays

22
Comparative study

• Scenario Journalist asked to generate story
  ideas for an upcoming meeting
• Topics like human smuggling
• Conducted 4 web searches
• 2 with categorized overview
• 2 with baseline

23
Experimental conditions
24
Experimental conditions
25
What changed

• Categorized overviews were used to narrow,
  filter, refine, and explore
• I loved it. I was in love with that. I wish
  Google had that. With 3 clicks you have 5 pieces
  of information...
• Sort of a search within a search. That was very
  cool.
• Fewer, broader queries
• Reduced cognitive effort

26
What changed

• Provoked ideas / questions
• For the art crimes one, I saw Science and
  asked, "What does that have to do with art
  crimes?" and I found out that science can help
  solve art crimes.
• When feeling stuck

27
Subjective ratings
Agreement



Location of pages in list
Significance p0.10
28
Subjective ratings
Agreement



Location of pages in list
Significance p0.10
29
Limitations of study

• Narrow subject pool, scenario, task
• Three kinds of categories
• One form of categorized overview
• Hierarchy presentation
• Alternative Graphical overviews

30
Design principles for exploratory search
interfaces

• Provide categorized overviews
• Organize by meaningful, stable categories
• Provide example documents for each category
• Arrange important text (title, snippet, URL) for
  fast scanning/skimming
• Support multiple kinds of categories
• Support multiple kinds of visual displays

31
Future directions

• Hierarchy structure
• Graphical overviews
• Extend
• Domains
• Categories
• Information seeking
• Stages
• Tasks

32
Conclusions

• Categorized overviews based on meaningful and
  stable categories
• Help organize and assess results
• Enable lightweight interaction and exploration
• Suggest ideas, provoke questions

33
Conclusions

• Categorized overviews based on meaningful and
  stable categories
• Help organize and assess results
• Enable lightweight interaction and exploration
• Suggest ideas, provoke questions
• Improve search experience

34
For more informationwmk_at_acm.orgwww.cs.umd.edu
/hcil/categorizedsearchHuman-Computer
Interaction Laboratorys23rd Annual
SymposiumJune 1-2, 2006University of Maryland,
College ParkThis research was partially
supported by an AOL Fellowship in Human-Computer
Interaction andby the National Science
Foundation (NSF EIA 0129978).
35
Ancillary Material
36
CitiViz
37
SWISH Dyna-Cat
38
GRiDL
39
Antarctica Visual Net
40
Cat-a-Cone
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
Categorized Search Results Research Goals

• What exploratory tasks benefit?
• How does the visual presentation of the overview
  impact exploratory search?
• How do the categories affect the exploratory
  search experience?

45
Research questions

• How do categorized overviews change
• Behavior
• Strategies
• Do searchers think about their search tactics
  differently?

46
(No Transcript)
47
Two early studies

• Domain Government web site search
• Pre-computed results
• Urban sprawl
• Breast cancer
• Alternative energy
• Two-level department/agency hierarchy
• Top Department of Interior
• 2nd National Park Service
• Motivating scenario tasks

48
Study 1
Baseline

• Categorized overviews (government agencies)
• Expandable outliner
• Treemap

49
Study 1

• Categorized overviews (government agencies)
• Expandable outliner
• Treemap

50
Study 2

• Stable categories
• (government agencies)

Automated clustering
51
With categorized overviews

• Users noticed missing results
• Agencies without search results
• What I found informative was what didnt show
  up.
• Easier to use
• More helpful
• Generating ideas
• Identifying resources

52
Search process model
External task
Identify problem
Need satisfied OR satisfice
Domainknowledge
Articulate needs
Evaluate results
Query(re)formulation
Sutcliffe Ennis (1998)
53
Search process model
External task
Identify problem
Need satisfied OR satisfice
Domain categoryknowledge
Articulate needs
Evaluate results
Explore/ navigate results
Query(re)formulation
54
The SERVICE search system

• Platform for building categorized overview
  interfaces
• Architecture to facilitate easy plug-in of web
  search result classifiers
• Working search interface for user study

Query
Generates
Result Set
Categorized-using
Classifier
Generates
Categorizedoverview
55
SERVICE architecture
User Interface

• Data Model
• Search result set- Individual search results-
  Category operations- Caching

• Classifiers
• Thematic ODP- Geographic- US government-
  Domain Name System (DNS)- Document size- Last
  time visited- Music genre

Search Engine Interface - Google- AOL Music
Search
Logging facility - JavaScript events
56
AOL Music Search
57
AOL Music Search
58
Experimental design

• Quantitative and qualitative
• Within-subjects, factorial design, N24
• Independent variables
• Interface type
• Baseline Google-like
• Experimental Adds categorized overview (CO)
• Topic type
• Broad
• Narrow

59
Summary - Categorized overviews

• Eased exploration and assessment
• Improved organization
• Changed search tactics
• Fewer, broader queries
• Reduced cognitive effort
• Provoked
• Ideas
• Questions
• Yielded more satisfying experiences

60
(No Transcript)
61
Scenario and task

• Journalist asked to generate story ideas for an
  upcoming meeting
• Conduct a short web search (12 min)
• Generate ideas
• Collect pages that suggested the idea
• Validation
• Reviewed with journalism professor
• Verified during exit interview

62
Distribution of pages viewed
Percent viewed at location
Location of pages in list
63
(No Transcript)
64
Results

• Viewed pages deeper with categorized overview
• Median 18 vs. 12
• Did not collect pages deeper
• Median 16
• Depth of viewed pages

• Baseline
• Categorized overview

All results are significant at p where noted
65
Results

• Topics were interpreted very individually
• Depth of viewed pages

• Aging workforce
• Human smuggling
• International art crime
• Workplace allergies

Marginally significant, p
66
Results

• Collected proportionally more categorized pages
  with categorized overview
• Modest difference

67
Results
Ease of exploration Helped
assess results
Baseline
Baseline
Categorized overview
Categorized overview
68
(No Transcript)
69
Qualitative Results

• Categorized overviews were used to narrow,
  filter, refine, and explore
• I loved it. I was in love with that. I wish
  Google had that. With 3 clicks you have 5 pieces
  of information...
• Sort of a search within a search. That was very
  cool.
• Fewer, broader queries

70
Qualitative Results

• Provoked ideas / questions
• For the art crimes one, I saw Science and
  asked, "What does that have to do with art
  crimes?" and I found out that science can help
  solve art crimes.
• Reduced cognitive effort
• Maybe it made me a little bit lazy. But I felt
  like I had to do less because it would do more.
• When feeling stuck
• allergies in the workplace It was tougher to
  find varying things so I used the categories more
  when I was kind of stuck.

71
Qualitative Results

• Minor problems with categories
• Why did they put News and Media under Computers?

72
Limitations of study

• Narrow subject pool, scenario, task
• Only considered one form of categorized overview
• Hierarchy presentation
• Graphical approaches
• Unmatched topics (broad vs. narrow)

73
SERVICE Architecture
API/UI
SelectVisualization(1)
SelectClassification(2)
Query (3)
Data Model
Visualization Components
ModelChangeEvent(9)
1
Data Flow
Coordinated Events(e.g. select, brush)(10)
SearchResults(8)
Query
Query(4)
SearchResults(7)
Coordination Components
Generates
Result Set
Search Engine Interface
Classification Components
Categorized-using
Classification
SearchResults(6)
Query(5)
Generates
Visualization
Search Engine
74
Definitions

• Classification A set of categories
• May be hierarchical
• Categorized overview An overview of a set of
  results based on a classification

75
Non-Selection of Categories

• Read categories with selecting them
• Meaningful, but not relevant
• Not meaningful

76
Importance of Text Scanning

• Category labels alone are not sufficient
• Users scanned text extensively
• Titles
• Snippets
• URLs

77
Other Findings

• Expandable outliner vs. treemap
• No significant measured differences
• More preferred expandable outliner

78
User-controlled reorganizationof search results

• Thematic hierarchy
• Government hierarchy
• Lightweight classifications
• DNS domain
• Page size
• Last time seen

79
(No Transcript)
80
(No Transcript)
81
(No Transcript)
82
Audio clips

• Empty categories quote

83
(No Transcript)
84
Topics
85
Hypotheses With COs, Ss will

• Select results more deeply in result list
• Collect pages more deeply in result list
• Collect larger proportion of pages from
  categories
• Issue fewer queries
• More strongly with broad topics

86
Hypotheses With COs, Ss will

• Develop more familiarity with topic
• Find more useful information
• Make more progress toward scenario goal
• Gain a better overview
• Find exploring search results easier

87
Hypotheses Ss will find COs

• More helpful in assessing results and deciding
  next steps
• More satisfying, organized, stimulating, easier
  to use
• More complex
• Harder to use

88
Participants

• Recruited Journalism students
• Ages 18-27, median 20
• Mostly undergraduate
• 3 years search experience
• 22 searched at least once per day
• 2 search 1-2 times per week
• Reported success on most searches

89
Procedure

• Welcome, informed consent, demographic
  questionnaire
• First interface
• Training
• Pre-search questionnaire
• Talk-aloud during search
• Post-search questionnaire
• Repeat for second search
• Second interface
• Exit interview

90
Results

• Why no difference for human subjects topic?
• Baseline CO

Depth of viewed pages
91
Broader queries

• I knew that if I did a broader word it could be
  divided by the categories I didn't necessarily
  have to be so specific.
• Rather than narrow down my search by adding
  additional search words I found myself narrowing
  my search by exploring categories and
  subcategories.

92
Fewer queries

• I didn't use as many queries, which is part of
  the reason why I didn't get as good information.
• Maybe it made me a little bit lazy. But I felt
  like I had to do less because it would do
  more.it didn't take as much from me ... I guess
  I changed by doing less.

93
Category problems

• I wasn't exactly sure what I thought Shopping
  would be but I didnt think it was going to be
  here is where you can buy things like mold
  remover...whatever I thought it wasn't a web site
  where you can go shopping.
• Why did they put News and Media under Computers?
  .Publications under Shopping?
• Problems were considered minor

94
Getting an overview

• I looked to see how the search results were
  divided up, what main categories there were,
  because I thought it would be faster way to see
  what I had in front of me especially for this
  particular task where I'm looking for different
  angles within a larger topic I wanted to see,
  well, theres a social issue and a health issue
  and a business issue, so that lends itself very
  well to that.