Personalized%20Web%20Search%20Uncommon%20Responses%20to%20Common%20Queries

1
Personalized Web SearchUncommon Responses to
Common Queries

• Jaime Teevan, MIT
• with Susan T. Dumais
• and Eric Horvitz, MSR

2
(No Transcript)
3
Personalizing Web Search

• Motivation
• Algorithms
• Results
• Future Work

4
Personalizing Web Search

• Motivation
• Algorithms
• Results
• Future Work

5
Study of Personal Relevancy

• 15 participants
• Microsoft employees
• Managers, support staff, programmers,
• Evaluate 50 results for a query
• Highly relevant
• Relevant
• Irrelevant
• 10 queries per person

6
Study of Personal Relevancy

• Query selection
• Chose from 10 pre-selected queries
• Previously issued query

Pre-selected
cancer Microsoft traffic
bison frise Red Sox airlines
Las Vegas rice McDonalds
Mary
Joe
Total 137
53 pre-selected (2-9/query)
7
Relevant Results Have Low Rank
Highly Relevant
Relevant
Irrelevant
8
Relevant Results Have Low Rank
Highly Relevant
Rater 1
Rater 2
Relevant
Irrelevant
9
Same Results Rated Differently

• Average inter-rater reliability 56
• Different from previous research
• Belkin 94 IRR in TREC
• Eastman 85 IRR on the Web
• Asked for personal relevance judgments
• Some queries more correlated than others

10
Same Query, Different Intent

• Different meanings
• Information about the astronomical/astrological
  sign of cancer
• information about cancer treatments
• Different intents
• is there any new tests for cancer?
• information about cancer treatments

11
Same Intent, Different Evaluation

• Query Microsoft
• information about microsoft, the company
• Things related to the Microsoft corporation
• Information on Microsoft Corp
• 31/50 rated as not irrelevant
• Only 6/31 do more than one agree
• All three agree only for www.microsoft.com
• Inter-rater reliability 56

12
Search Engines are for the Masses
Joe
Mary
13
Much Room for Improvement

• Group ranking
• Best improves on Web by 38
• More people ? Less improvement

14
Much Room for Improvement

• Group ranking
• Best improves on Web by 38
• More people ? Less improvement
• Personal ranking
• Best improves on Web by 55
• Remains constant

15
Personalizing Web Search

• Motivation
• Algorithms
• Results
• Future Work

- Seesaw Search Engine
- See
- Seesaw
16
Personalization Algorithms

• Related to relevance feedback

• Query expansion

• Standard IR

Query
Server
Document
Client
User
17
Personalization Algorithms

• Related to relevance feedback

• Query expansion

• Standard IR

Query
Server
Document
Client
User
v. Result re-ranking
18
Result Re-Ranking

• Ensures privacy
• Good evaluation framework
• Can look at rich user profile
• Look at light weight user models
• Collected on server side
• Sent as query expansion

19
BM25
with Relevance Feedback
Score S tfi wi
N
ni
R
ri
N ni
wi log
20
BM25
with Relevance Feedback
Score S tfi wi
N
ni
R
ri
(ri0.5)(N-ni-Rri0.5) (ni-ri0.5)(R-ri0.5)
wi log
21
User Model as Relevance Feedback
Score S tfi wi
N
R
N NR
ni niri
ri
ni
(ri0.5)(N-ni-Rri0.5) (ni-ri0.5)(R-ri0.5)
wi log
22
User Model as Relevance Feedback
Score S tfi wi
N
R
N NR ni niri
ri
ni
(ri0.5)(N-ni-Rri0.5) (ni- ri0.5)(R-ri0.5)
wi log
23
User Model as Relevance Feedback
World
Score S tfi wi
N
User
R
ri
ni
24
User Model as Relevance Feedback
World
Score S tfi wi
N
User
World related to query
R
ri
ni
ni
N
25
User Model as Relevance Feedback
World
Score S tfi wi
N
User
World related to query
R
ri
ni
R
ni
User related to query
N
ri
Query Focused Matching
26
User Model as Relevance Feedback
World Focused Matching
World
Score S tfi wi
N
User
Web related to query
R
ri
ni
R
ni
User related to query
N
ri
Query Focused Matching
27
Parameters

• Matching
• User representation
• World representation
• Query expansion

28
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
29
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
30
User Representation

• Stuff Ive Seen (SIS) index
• MSR research project Dumais, et al.
• Index of everything a users seen
• Recently indexed documents
• Web documents in SIS index
• Query history
• None

31
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
All SIS Recent SIS Web SIS Query history None
32
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query Focused World Focused
All SIS Recent SIS Web SIS Query History None
33
World Representation

• Document Representation
• Full text
• Title and snippet
• Corpus Representation
• Web
• Result set title and snippet
• Result set full text

34
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
All SIS Recent SIS Web SIS Query history None
Full text Title and snippet
Web Result set full text Result set title and
snippet
35
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
All SIS Recent SIS Web SIS Query history None
Full text Title and snippet
Web Result set full text Result set title and
snippet
36
Query Expansion

• All words in document
• Query focused

The American Cancer Society is dedicated to
eliminating cancer as a major health problem by
preventing cancer, saving lives, and diminishing
suffering through ...
The American Cancer Society is dedicated to
eliminating cancer as a major health problem by
preventing cancer, saving lives, and diminishing
suffering through ...
37
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
All SIS Recent SIS Web SIS Query history None
Full text Title and snippet
Web Result set full text Result set title and
snippet
All words Query focused
38
Parameters

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
All SIS Recent SIS Web SIS Query history None
Full text Title and snippet
Web Result set full text Result set title and
snippet
All words Query focused
39
Personalizing Web Search

• Motivation
• Algorithms
• Results
• Future Work

40
Best Parameter Settings

• Matching
• User representation
• World representation
• Query expansion

Query focused World focused
Query focused World focused
Query focused
All SIS Recent SIS Web SIS Query history None
All SIS
Recent SIS
Web SIS
All SIS Recent SIS Web SIS Query history None
All SIS
Full text Title and snippet
Full text
Title and snippet
Web Result set full text Result set title and
snippet
Result set title and snippet
Web
Result set title and snippet
All words Query focused
All words
Query focused
41
Seesaw Improves Retrieval

• No user model
• Random
• Relevance Feedback
• Seesaw

42
Text Alone Not Enough
43
Incorporate Non-text Features
44
Summary

• Rich user model important for search
  personalization
• Seesaw improves text based retrieval
• Need other features
• to improve Web
• Lots of room
• for improvement

future
45
Personalizing Web Search

• Motivation
• Algorithms
• Results
• Future Work

• Further exploration
• Making Seesaw practical
• User interface issues

46
Further Exploration

• Explore larger parameter space
• Learn parameters
• Based on individual
• Based on query
• Based on results
• Give user control?

47
Making Seesaw Practical

• Learn most about personalization by deploying a
  system
• Best algorithm reasonably efficient
• Merging server and client
• Query expansion
• Get more relevant results in the set to be
  re-ranked
• Design snippets for personalization

48
User Interface Issues

• Make personalization transparent
• Give user control over personalization
• Slider between Web and personalized results
• Allows for background computation
• Creates problem with re-finding
• Results change as user model changes
• Thesis research ReSearch Engine

49
Thank you!
50
Search Engines are for the Masses

• Best common ranking
• DCG(i)
• Sort results by number marked highly relevant,
  then by relevant
• Measure distance with Kendall-Tau
• Web ranking more similar to common
• Individuals ranking distance 0.469
• Common ranking distance 0.445

Gain(i), if i 1 DCG(i1)
Gain(i)/log(i), otherwise