The Comings and Goings of SummarizationOne Perspective

1
The Comings and Goings of SummarizationOne
Perspective

• Judith D. Schlesinger
• IDA/Center for Computing Sciences

2
Where We Started

• User community tasked with processing a huge of
  documents
• Document order determined solely by the search
  engine
• An awareness that good information was being
  missed
• A desperate need for help

3
What has Happened

• Contracted out for a summarizer
• Delivered early mid-90s
• Initially batch mode/added web access
• Added a ticker tape
• Attached to single data base
• Currently have several hundred users
• Single document summaries
• Both generic and query-based

4
What has Happened

• 1999 1 year research effort
• Improve the summarizer
• Generic and query-based still supported
• Now also do multi-doc summaries
• Doing very well at DUC
• Improved on the original system

5
What has Happened

• Technology Transfer
• Using original user interface to minimize change
• Must introduce new multi-document summarization
  capability

6
What Drives Our Research

• USER NEED!
• Must be able to justify work we are doing
• Must discover what will most help the user
• DUC
• As long as consistent with user needs
• Our determination
• As long as we have ideas

7
What We Have Learned

• Evaluation is difficult
• Problems with sentence selection
• Problems with n-gram matching
• Problems with human evaluations
• Problems getting humans to agree
• Need to understand purpose!

8
What We Have Learned

• Document genre is important
• Methods for one dont necessarily work for
  another
• But, dont want to have lots of different methods

9
What We Have Learned

• Summaries are used for prioritization and
  selection
• Its dangerous to label summaries as
  informative
• Users want to use in place of full document
• Impossible to capture all information

10
What We Have Learned

• Readability is important
• Summary must flow
• Cant have boiler plate
• Must do anaphora resolution
• Presentation
• Must be easy to use
• Users want 1 tool/integrated tool

11
Current Issues

• Continue work on current system
• More anaphora work
• More boiler plate removal
• Better understanding of acceptable compression
  rates
• Better content selection
• Better coherency

12
Current Issues

• Better evaluation
• Need to know what really works
• Not matching some humans summary doesnt make it
  bad
• Its difficult to get people to evaluate
• Time consuming
• Tedious
• Better evaluation metrics
• Intrinsic and extrinsic

13
Current Issues

• Multi-document clustering in current evaluations
  is performed by humans
• Real systems must do automatically
• Need an evaluation of automatic clustering
• Clustering is not a solved problem!

14
Current Issues

• Improve entity based summaries
• who is, what is, where is
• Special form of query-based summaries
• Apply to different databases/
• different genres