Telethon Charity Service

1

• Telethon Charity Service
• Nancie Davidson (CCIR)

2
Background

• Italian language service for donating to charity
• Run in conjunction with TV appeals - Telethons
• Previous versions touch-tone only
• 30 of callers - recorded messages
• Transcribed offline - slow, expensive
• New speech enabled version
• Real time capture of donation information

3
Service Design

• Italian language (Nuance v6.2.4)
• Three main dialogue stages
• Credit card number
• Donation amount
• Confirmation of these details
• (Correction dialogue)
• Low security application

4
Developing the Service

• English version 1st - prompts and grammars
• Cultural as well as literal translation e.g.
• X Im sorry, there seems to be a problem ?
• ? Non è possibile completare l'operazione
• (literally the operation cannot be completed)
• Established phrase, no use of the term
  problem
  

5
Service Design

• Speech dominant design
• Brief, simple top level prompts
• Touch-tone options in the error recovery
• Use knowledge of valid credit card formats
  to aid recognition
  
• Amounts restricted to reasonable values
• 5000L 99 000 000L
• Below 100 000L, multiples of 500L only
• Above 100 000L, multiples of 1000L only

6
Trial Design

• Each participant
• One telephone call
• Unique credit card number
• Freedom to choose amount
• Likert usability questionnaire
• Performance data

7
Trial Design

• 58 participants, balanced for gender
• CSELT staff, Turin
• Prize draw for a bottle of Malt Whisky
• Experiment run during Dec 00 / Jan 01
• Instructions and questionnaire on website, hosted
  at CCIR
  

8
Priming
9
Questionnaire
10
Usability
Usability
7
6
5
4
negative neutral positive
3
Mean 5.81
2
1
Polite
Reliable
Friendly
Efficient
Flustered
Use again
Confusing
Frustrating
Needs improvement
Liked voice
Ease of use
Complicated
Voice clarity
Under stress
Prefer human
Concentration
Enjoyed using
Speed of service
Knew what to do
Degree of control
11
Attitude by Gender
7
6
5
4
negative neutral positive
3
Female N32, Mean5.93
Male N26, Mean5.68
2
1
Polite
Reliable
Friendly
Efficient
Flustered
Use again
Confusing
Frustrating
Needs improvement
Liked voice
Ease of use
Complicated
Voice clarity
Under stress
Prefer human
Concentration
Enjoyed using
Speed of service
Knew what to do
Degree of control
12
Task Completion

• 86.9 completed task successfully
• 93.1 completed call
• 4 participants (6.9) confirmed incorrect
  donation
  
• 1 incorrect credit card number
• 3 incorrect amount
• 2 participants (3.4) broke out
• Amounts involving units of less than 100L
• 2 participants (3.4) hung up
• 1 at confirm stage despite correct recognition
• 1 at credit card stage following 2 false rejects

13
Performance Data

• Callers overwhelmingly chose speech
• Only two incidences of DTMF
• Average call length 66 seconds
• Fast, efficient processing
• Silent responses
• None at credit card number / amount stage
• 12.7 at confirmation stage

14
Credit Card Recognition(Valid CCNo Constraints)
4.5
False Accept
OOG (7.6)
3.1
Correct Reject
6.2
False Reject
IG (92.4)
0.0
Mis-rec
86.2
Correct Recog
0
20
40
60
80
100
utts
15
Credit Card Recognition(Unconstrained)
4.5 (4.5)
False Accept
OOG (7.6)
3.1 (3.1)
Correct Reject
1.5 (6.2)
False Reject
IG (92.4)
Mis-rec
12.3 (0.0)
78.6 (86.2)
Correct Recog
0
20
40
60
80
100
utts
16
Recognition Performance(Amount)
9.1
False Accept
OOG (12.1)
3.0
Correct Reject
1.5
False Reject
IG (87.9)
4.6
Mis-rec
81.8
Correct Recog
0
20
40
60
80
100
utts
17
Recogniser Performance(Confirmation)
3.0
False Accept
OOG (6.0)
Correct Reject
3.0
1.5
False Reject
IG (94.0)
Mis-rec
0.0
92.5
Correct Recog
0
20
40
60
80
100
utts
18
Donations
16
14
12
10
No of participants
8
6
4
2
0
1000
5000
50 000
70 000
90 000
100 000
150 000
200 000
238 420
250 000
300 000
500 000
573 249
1 000 000
1 200 000
1 500 000
1 590 000
3 000 000
5 000 000
10 000 000
20 000 000
Lire
19
Summary

• Excellent task completion rate (86.9)
• Very high attitude rating (5.81)
• Good recognition rates throughout
• Ready for deployment

20
http//spotlight.ccir.ed.ac.uk
The University of Edinburgh