Speechbased Information Retrieval System with Clarification Dialogue Strategy

1
Speech-based Information Retrieval System with
Clarification Dialogue Strategy

• Teruhisa Misu, Tatsuya Kawahara
• Kyoto University, Japan

2
Background

• Progress in ASR and IR technique
• ?Target of spoken dialogue system is being
  extended to general documents
• Previous works on document retrieval with speech
• NTCIR-3 Web search task
  Speech input sub-task Fujii03
• Speech input newspaper article retrieval system
  Chang2002
• Speech input ODQA system Hori03Schofield03
• Speech Dialogue Navigator Retrieval for
  software support article Misu04

3
Problems in Spoken Language Input in Document
Retrieval Tasks

• 1. Vagueness of query
• Speech interface makes input easier,
  BUT makes query vague or fragmental
  ( Very limited information
  can be used as clues)
• 2. Speech recognition errors
• Important information may be lost
• ?Enormous list of possible relevant document is
  usually obtained

4
Query Clarification in
Conventional Tasks

• Document retrieval task with
  typed text input (TREC QA Track, etc.)
• Queries are (supposed to be) definite specific
• ?One question and one answer
• Dialogue strategy in DB query
  spoken dialogue systems
  (ATIS, DARPA
  communicator, etc.)
• Make questions based on pre-defined keywords and
  semantic slots
• ? NOT applicable to general document sets

5
Proposed Method to Clarify Users Query in
Document Retrieval Tasks

• Optimal clarification question is dynamically
  selected based on Information Gain (IG)
• ?How many retrieved documents can be eliminated
• Pool of questions are prepared based on bottom-up
  knowledge sources together with
  top-down ones
• Dependency structure analysis
• Metadata of knowledge base Human knowledge
• Update users query sentence

6
System Overview
User
System
Automatic speech recognition (ASR)
Speech input
Knowledge base
Matching with Knowledge base (KB)
Candidate questions with Information gain
Retrieval result
Any question with large IG
Question pool
No
Yes
Question
Update query sentence and retrieval result
Select question with Largest IG for clarification
Reply
7
Target Document Set(Knowledge Base KB)

• Software support knowledge base provided by
  Microsoft Corp.
• Written in natural language (plain text)
• 40K entries in total
• Example of support article
• HOWTO Use Speech Recognition in Windows XP
• The information in this article applies to
• Microsoft Windows XP Home Edition
• Summary
• This article describes how to use speech
  recognition in Windows XP. If you installed
  speech recognition with Microsoft Office XP, or
  if you

8
Back-end Retrieval System

• Dialog Navigator (developed at Univ. of Tokyo
  Kiyota et al., 2002)
• Retrieves Microsoft support article
• Accepts a typed-text input from users
• Interprets input sentences in consideration of
  syntactic dependency and synonymous expressions
• Outputs a dozen of retrieved documents as in
  Web search engine

This work Develop speech interface by adopting
Dialog Navigator as back-end system
9
System Overview
User
System
Automatic speech recognition (ASR)
Speech input
Knowledge base
Matching with Knowledge base (KB)
Candidate questions with Information gain
Retrieval result
Any question with large IG
Question pool
No
Yes
Question
Update query sentence and retrieval result
Select question with Largest IG for clarification
Reply
10
Definition of Information Gain (IG)

• Criterion for selecting optimal clarification
  question
• Represents reduction of entropy
• ?How many retrieved documents can be eliminated
  by incorporating additional information (reply to
  question)
• Define IG H(S) for a candidate question S

Dk k-th retrieved document by query
sentence CM(D) Matching score of document D Ci
of documents classified into category i
by candidate question S (weighted
by matching score) Documents related to NO
category belong to category 0
11
System Overview
User
System
Automatic speech recognition (ASR)
Speech input
Knowledge base
Matching with Knowledge base (KB)
Candidate questions with Information gain
Retrieval result
Any question with large IG
Question pool
No
Yes
Question
Update query sentence and retrieval result
Select question with Largest IG for clarification
Reply
12
Question Set for Clarification

• 1. Candidate questions based on
  dependency structure analysis (Method 1)
• 2. Questions based on Metadata
  included in KB (Method 2)
• 3. Questions based on human knowledge (Method 3)

13
Candidate Questions based on Dependency
Structure Analysis (Method 1)

• Clarify missing object
• ex. delete ? delete application program
• delete address
  book
• How to make candidate questions
• Automatically extract potentially ambiguous words
  from knowledge base
• Make question sentence via template
  What did you verb?
• Example questions (40 candidate questions in
  total)
• - What did you install? , What did you
  delete?

14
Selection by Modifier-head Pair Information (in
Method 1)

• Determine verbs that take various objects

INSTALL Applications ?20 Service
pack ?10 Drivers ?10 Devices ?8 Client
program ?6
SHUTDOWN Windows ? 40 Computer ?
50 Server ? 5
Large entropy
Small entropy
Selected 40 words with large value of entropy (
potentially ambiguous)
15
Candidate Questions based on Metadata in KB
(Method 2)

• Use metadata attached to target knowledge base
• Example of metadata in KB
• Example of question (16 candidate questions in
  total)
• - What is the version of your Windows?
• - What is the version of your Excel?

HOWTO Use Speech Recognition in Windows XP
The information in this article applies to
- Microsoft Windows XP Professional -
Microsoft Windows XP Home Edition Summary
This article describes how to use speech
recognition in Windows
XP. If you installed speech recognition with
Microsoft Office XP, you can use speech
recognition
16
Candidate Questions based on Human Knowledge
(Method 3)

• Use human knowledge accumulated at call centers
• Candidate questions are hand-crafted
• Example
• Candidate question When did the symptom occur?
  tries to identify phrase which appears after
  when, after, etc.
• When you install a application, after an
  unattached setup
• List of questions (3 candidate questions in
  total)
• - When did the symptom occur?
• - Tell me the error message
• - Specifically, what do you want to do?

17
Example Dialogue (1/2)

• S1 What is your problem?
• U1 I cannot print out
• Retrieval result
• 1. Print out photo taken by digital cameras.
• 2. How to print out titles of help.
• 3. Error in printing out Word documents.
• Calculate IG
• - Candidate question 1 What do you want to
  print out? IG 5.27
• - Candidate question 2 What is version of your
  Windows? IG 1.43
• - Candidate question 3 When did the symptom
  occur? IG 2.47
• -
• S2 (Make question with largest IG)
  
  What do you want to print out?
• U2 Image file.

18
Example Dialogue (2/2)

• S3 (Update query sentence) Retrieving with I
  cannot print out image file.
• Retrieval result
• 1. You cannot print a background graphic on an
  Excel worksheet.
• 2. You cannot print graphics on Web page.
• 3. Color Images print black and white from
  frontpage editor
• Calculate IG
• - Candidate question 1 What is your application
  program? IG 7.32
• - Candidate question 2 What is version of your
  Windows? IG 1.23
• - Candidate question 3 When did the symptom
  occur? IG 1.69
• -
• S4 What is your application program?
• U3 Internet Explorer 5.5.
• S5 Retrieving with I cannot print out image
  file on
• Internet Explorer 5.5.

19
System Implementation

• Operates on Web browser Internet Explorer
• ASR decoder Julius for SAPI
• Clarification questions are presented by
  synthesized voice
• User replies
  the question
  by speech

20
Experimental Condition

• 14 subjects who had not used the system
• 14 tasks for each subject
• - Task example
  You want to send photograph taken
  using digital camera by mail. As mailer soft, you
  want to use Outlook 2002 as usual. However, you
  dont know how to send it, since you have never
  send them.
• - Example of users query
• I want to send a photo!
• Subjects were allowed to rephrase their
  utterances up to once per task if relevant result
  (they thought) was not obtained.
• 239 utterance for 196 tasks in total
• Task was judged successful when retrieval result
  contained correct one

21
Experimental Results (Success Rate and Rank of
Correct Doc.)

• - Proposed method (log data) System generated
  question, and the users replied to them as they
  thought appropriate
• - Proposed method (simulation) System generated
  questions, and appropriate answers were given
  manually

Significant improvement achieved in success rate
and rank of correct document
22
Comparison of Question Methods

• Method 1 obtained a relatively high improvement
  rate per question
• Most significant improvement by using three
  methods together

23
Discussion about Difference in Success Rate

• Success rate in simulation? 83.3
• Success rate in log data ? 78.4
• ASR errors in users uttered reply
• tekisuto fairu (text file) ? kisonnno fairu
  (existing file)
• sushiki (numerals) ? tsuuchi
  (announcement)
• Improper reply to the question
• When did the symptom occur? ? Just now!
• What is the APPLICATION program? ? Windows XP!

24
Summary

• Dialogue strategy to clarify users vague query
  in document retrieval task
• Select optimal clarification question
  based on information gain
  (IG)
• Questions based on dependency structure analysis
• Questions based on metadata human knowledge
• Proposed method achieved significant improvement
  in success rate
• All categories of prepared questions contributed
  to improvement

25
Overview of Question Generation
Enormous list of retrieval result
Vague or ill-formed Query
Loss of important information by ASR errors
Knowledge base
Retrieval results
?I cannot start-up?
While , I cannot print it
Candidate questions
User
Candidate II Tell me the error message
Candidate I What is the version of your Windows?
Candidate N ??
Make question with largest IG
Retrieval results
Retrieval result
What is the Version of your Windows?
System
Largest IG
Small IG
26
Overview of Speech
Dialogue Navigator
User
System
Automatic speech recognition (ASR)
Speech input
Confirmation for robust retrieval
Confirmation
Matching with Knowledge base (KB)
Knowledge Base (KB)
Reply
Dialogue to narrow down retrieved documents
Question
Reply
Final Result