THE USE OF PROSODIC FEATURES TO HELP USERS

1
THE USE OF PROSODIC FEATURES TO HELP USERS
EXTRACT INFORMATION FROM STRUCTURED ELEMENTS
IN SPOKEN DIALOGUE SYSTEMS
Jaakko Hakulinen, Markku Turunen and Kari-Jouko
Räihä Human-Computer Interaction Group Department
of Computer Science University of Tampere, Finland
2
INDEX 1.Introduction (2 slides) 2.Experiment (2
slides) 3.Results (3 slides) 4.Discussion (4
slides) 5.Conclusions (1 slide)
3
INTRODUCTION 1 / 2
Synthetic speech may be very difficult to
understand when complex structural elements like
lists, addresses and tables are spoken.
Added prosodic elements (pitch, volume and rate
variations and pauses) may bring both
intelligibility and pleasantness to synthetic
speech.
Humans use prosody very effectively - we could
borrow a lot from professional human speakers.
Introduction - Experiment - Results - Discussion
- Conclusions
4
INTRODUCTION 2 / 2
We are developing a system for reading e-mail
with a telephone. There are a lot of
structural elements like lists and e-mail
addresses in the system.
Introduction - Experiment - Results - Discussion
- Conclusions
5
EXPERIMENT 1 / 2

• We recorded 15 utterances from our telephony
  based e-mail client using three male readers and
  a synthesizer
• non-professional speaker
• lecturer
• radio voice
• speech synthesizer (InfoVox 230)

Utterances included a lot of structural elements
like lists, addresses and tables.
We gave precise instructions to readers how to
read utterances.
Introduction - Experiment - Results - Discussion
- Conclusions
6
EXPERIMENT 2 / 2
In order to measure intelligibility and
pleasantness of spoken utterances sixteen
subjects listened to one reader each and answered
two questions per utterance.
One question was presented before hearing the
utterance and another one after hearing the
utterance.
We also collected perceived speech profiles by
asking questions about the use of prosodic
elements. The question form included 10 scales
and open questions.
Introduction - Experiment - Results - Discussion
- Conclusions
7
RESULTS 1 / 3
18 answers from the total of 29 intelligibility
questions could be analyzed using numerical
scale. The figure below compares the scores of
each speaker to the average score.
In questions presented after hearing the
utterances answers varied greatly between the
readers.
Introduction - Experiment - Results - Discussion
- Conclusions
8
RESULTS 2 / 3
When listeners evaluated the readers use of
voice (perceived speech profiles) there were
great differences as can be seen in the figure
below, in which the problems are illustrated. The
radio persons voice was ranked clearly the best,
only his use of pauses was not liked.
Listeners ranked the lecturers voice the most
problematic - mainly due to his monotonous and
slow speaking style.
Introduction - Experiment - Results - Discussion
- Conclusions
9
RESULTS 3 / 3
We analyzed the exact ways in which human readers
used prosodic elements by selecting 16 samples
(4 from each reader) for careful analysis word
boundaries were marked to analyze the use of
pauses, and pitch and energy were extracted from
the utterances.
The lecturer used pauses more than the others.
The radio person used a lot of pitch and energy
variations, especially for de-emphasizing
unimportant sections.
Introduction - Experiment - Results - Discussion
- Conclusions
10
DISCUSSION 1 / 4
Perceived speaker profiles and the results of
sample analysis matched very well the lecturer
used pauses a lot (too much according to some
listeners), the radio person used only a few
pauses and the listeners complained about that,
but they also liked his way to use pitch and
energy.
There were also variations in how different
emphasizing methods were used - for example, the
lecturer divided sentences into meaningful
sections.
Introduction - Experiment - Results - Discussion
- Conclusions
11
DISCUSSION 2 / 4
When comparing the results, it seems clear that
pauses are the most important factor for
intelligibility and other features add
pleasantness.
This is especially true when listener
comprehension and memorizability are needed
(segmental intelligibility vs. comprehension).
Overuse of pauses could lead to problems in
addition to affecting pleasantness they are also
ambiguous in human-computer interaction.
Introduction - Experiment - Results - Discussion
- Conclusions
12
DISCUSSION 3 / 4
We did not analyze the uses of speaking rates -
this factor could also be important.
There are other elements in speech than those
considered here and they can affect both the
intelligibility and the pleasantness of a voice.
We will continue our research by instrumenting
the synthesizer so that it uses prosody in the
same manner as human speakers and repeat these
tests.
Introduction - Experiment - Results - Discussion
- Conclusions
13
DISCUSSION 4 / 4
We use software modules called presentation
agents (persons in the paper) to convert
conceptual messages to text that is read to a
user. These agents also add prosodic elements to
messages.
We want to produce different kind of agents that
could use a rich and varying set of prosodic
elements in their speech.
Finally, we shall arrange experiments in real
interactive situations.
Introduction - Experiment - Results - Discussion
- Conclusions
14
CONCLUSIONS 1 / 1
By adding prosodic elements to speech it is
possible to greatly improve both intelligibility
and pleasantness of speech.
The use of pauses seems to improve speech
intelligibility considerably and the use of pitch
and energy had a positive effect on pleasantness.
By analyzing the exact ways in which human
readers use prosodic elements we can use those
elements in our presentation agents.
Introduction - Experiment - Results - Discussion
- Conclusions
15
http//www.cs.uta.fi/research/hci/SUI/
Kari-Jouko Räihä (kjr_at_cs.uta.fi) Markku Turunen
(mturunen_at_cs.uta.fi) Jaakko Hakulinen
(jh_at_cs.uta.fi)
speech_at_cs.uta.fi