Exercising these ideas

1
Exercising these ideas

• You have a description of each item in a small
  collection. (30 web sites)
• Assume we are looking for information about
  boxers, a type of dog.
• List the items that are relevant to this
  information need. (If it is impossible to tell,
  as with 9, mark it not relevant.)
• Rq
• Assume that an unspecified search system has
  returned a ranked result as follows
• (1, 4, 6, 8, 10, 12, 15, 16, 17, 18, 20, 23,26,
  28, 30)
• Show the ranking for the query as done on page
  76.
• Plot the precision vs. recall curve

2
Exercise, continued

• We do a second query against the same collection.
  This time we are interested in the Boxer rescue
  organizations.
• Rq
• Ranked result (4, 6, 7, 10, 11, 15, 16, 17, 18,
  19, 22, 23, 25, 28, 29)
• Calculate the average precision at recall level 3
  for these two queries.

3
More exercise

• We redo the first query, using a new search
  algorithm. The ranked result is now this
• (4, 5, 6, 7, 8, 10, 11, 15, 16, 17, 18, 20, 23,
  28, 29)
• Produce the average recall vs. precision figures
  for these two algorithms. How would you describe
  the performance?
• We redo the second query using the new search
  algorithm. The ranked result is this
• (7, 10, 11, 15, 17, 18, 19, 21, 23, 24, 26, 28,
  29, 30)

4
Precision Histogram
5

• For each of the four searches, what is the
  R-precision?
• Use a precision histogram to compare these two
  algorithms for the two queries shown.
• Calculate the harmonic mean at document 5 in the
  rankings, using the first query and repeating for
  each algorithm.
• Calculate the E measure for algorithm 1 using
  query 1 and a moderate preference for recall
  rather than precision.

6
Collection
Answer Set A
Relevant docs R
Relevant docs unknown to the User which were
retrieved Ru
Relevant docs known to the User U
Relevant docs known to the User which were
retrieved Rk
7

• Assume that the even number elements in the
  collection are known to the user.
• Calculate the coverage ratio for algorithm 1 on
  search 1
• Calculate the novelty ratio for the same
  algorithm and search