CS 224S LINGUIST 181 Speech Recognition and Synthesis

1
CS 224S / LINGUIST 181Speech Recognition and
Synthesis

• Dan Jurafsky

Lecture 7 Intro to ASRHMMs History Forward
and Viterbi
IP Notice
2
Outline for Today

• Speech Recognition Architectural Overview
• Hidden Markov Models in general
• Forward
• Viterbi Decoding
• HMMs for speech structure
• How this fits into the ASR component of course
• 2/2 Baum-Welch (EM) training of HMMs
• 2/7 Acoustic Model estimation Gaussians,
  triphones, etc
• 2/9 Advanced Issues in Acoustic Mod. Guest
  Lecture
• 2/14 Language Modeling
• 2/16 Advanced Issues in Decoding Search

3
LVCSR

• Large Vocabulary Continuous Speech Recognition
• 20,000-64,000 words
• Speaker independent (vs. speaker-dependent)
• Continuous speech (vs isolated-word)

4
Current error rates
5
Why is conversational speech harder?

• Praat example

6
HSR versus ASR

• Conclusions
• Machines about 5 times worse than humans
• Gap increases with noisy speech
• These numbers are rough, take with grain of salt

7
LVCSR Design Intuition

• Build a statistical model of the speech-to-words
  process
• Collect lots and lots of speech, and transcribe
  all the words.
• Train the model on the labeled speech
• Paradigm Supervised Machine Learning Search

8
Speech Recognition Architecture
9
The Noisy Channel Model

• Search through space of all possible sentences.
• Pick the one that is most probable given the
  waveform.

10
The Noisy Channel Model (II)

• What is the most likely sentence out of all
  sentences in the language L given some acoustic
  input O?
• Treat acoustic input O as sequence of individual
  observations
• O o1,o2,o3,,ot
• Define a sentence as a sequence of words
• W w1,w2,w3,,wn

11
Noisy Channel Model (III)

• Probabilistic implication Pick the highest prob
  S
• We can use Bayes rule to rewrite this
• Since denominator is the same for each candidate
  sentence W, we can ignore it for the argmax

12
Noisy channel model
likelihood
prior
13
The noisy channel model

• Ignoring the denominator leaves us with two
  factors P(Source) and P(SignalSource)

14
Speech Architecture meets Noisy Channel
15
HMMs for speech
16
Phones are not homogeneous!
17
Each phone has 3 subphones
18
Resulting HMM word model for six
19
Architecture Five easy pieces (only 2 for today)

• Feature extraction
• Acoustic Modeling
• HMMs, Lexicons, and Pronunciation
• Decoding
• Language Modeling

20
Feature Extraction

• Digitize Speech
• Extract Frames

21
Quick reminder digitizing Speech
22
Reminder digitizing Speech (A-D)

• Sampling
• measuring amplitude of signal at time t
• 16,000 Hz (samples/sec) Microphone (Wideband)
• 8,000 Hz (samples/sec) Telephone
• Why?
• Need at least 2 samples per cycle
• max measurable frequency is half sampling rate
• Human speech lt 10,000 Hz, so need max 20K
• Telephone filtered at 4K, so 8K is enough

23
Digitizing Speech (II)

• Quantization
• Representing real value of each amplitude as
  integer
• 8-bit (-128 to 127) or 16-bit (-32768 to 32767)
• Formats
• 16 bit PCM
• 8 bit mu-law log compression
• LSB (Intel) vs. MSB (Sun, Apple)
• Headers
• Raw (no header)
• Microsoft wav
• Sun .au

40 byte header
24
Quick sketch of Frame Extraction

• A frame (25 ms wide) extracted every 10 ms

25 ms
. . .
10ms
a1 a2 a3
Figure from Simon Arnfield
25
MFCC (Mel Frequency Cepstral Coefficients)

• Do FFT to get spectral information
• Like the spectrogram/spectrum we saw earlier
• Apply Mel scaling
• Linear below 1kHz, log above, equal samples above
  and below 1kHz
• Models human ear more sensitivity in lower freqs
• Plus Discrete Cosine Transformation
• More on this later in course

26
Final Feature Vector

• 39 Features per 10 ms frame
• 12 MFCC features
• 12 Delta MFCC features
• 12 Delta-Delta MFCC features
• 1 (log) frame energy
• 1 Delta (log) frame energy
• 1 Delta-Delta (log frame energy)
• So each frame represented by a 39D vector

27
Where we are

• Given a sequence of acoustic feature vectors,
  one every 10 ms
• Goal output a string of words
• Well spend 6 lectures on how to do this
• Rest of today
• Markov Models
• Hidden Markov Models in the abstract
• Forward Algorithm
• Viterbi Algorithm
• Start of HMMs for speech

28
Reminder First-order observable Markov Model

• a set of states
• Q q1, q2qN the state at time t is qt
• Current state only depends on previous state
• Transition probability matrix A
• Special initial probability vector ?
• Constraints

29
Reminder Markov model for Dow Jones
Figure from Huang et al, via
30
Reminder Markov Model for Dow Jones

• What is the probability of 5 consecutive up days?
• Sequence is up-up-up-up-up
• I.e., state sequence is 1-1-1-1-1
• P(1,1,1,1,1)
• ?1a11a11a11a11 0.5 x (0.6)4 0.0648

31
Reminder Hidden Markov Models

• a set of states
• Q q1, q2qN the state at time t is qt
• Transition probability matrix A aij
• Output probability matrix Bbi(k)
• Special initial probability vector ?
• Constraints

32
Reminder Assumptions

• Markov assumption
• Output-independence assumption

33
Reminder HMM for Dow Jones
From Huang et al.
34
HMMs for weather and ice-cream

• Jason Eisners cute HMM in Excel, showing Viterbi
  and EM
• http//www.cs.jhu.edu/jason/papers/ - tnlp02
• Idea
• You are climatologists in 3004
• Want to know about Baltimore weather in 2004
• Only data you have is Jason Eisners diary
• Which records how much ice cream he ate each day
• Observation
• Number of ice creams
• Hidden State Simplify to only 2 states
• Weather is Hot or Cold that day.

35
The Three Basic Problems for HMMs

• (From the classic formulation by Larry Rabiner
  after Jack Ferguson)
• L. R. Rabiner. 1989. A tutorial on Hidden Markov
  Models and Selected Applications in Speech
  Recognition. Proc IEEE 77(2), 257-286. Also in
  Waibel and Lee volume.

36
The Three Basic Problems for HMMs

• Problem 1 (Evaluation) Given the observation
  sequence O(o1o2oT), and an HMM model ?
  (A,B,?), how do we efficiently compute P(O ?),
  the probability of the observation sequence,
  given the model
• Problem 2 (Decoding) Given the observation
  sequence O(o1o2oT), and an HMM model ?
  (A,B,?), how do we choose a corresponding state
  sequence Q(q1q2qT) that is optimal in some
  sense (i.e., best explains the observations)
• Problem 3 (Learning) How do we adjust the model
  parameters ? (A,B,?) to maximize P(O ? )?

From Rabiner
37
The Evaluation Problem for speech

• The observation sequence O is a series of MFCC
  vectors
• The hidden states W are the phones and words
• For a given phone/word string W, our job is to
  evaluate P(OW)
• Intuition how likely is the input to have been
  generated by just that word string W

38
The Evaluation Problem

• Given observation sequence O and HMM ?, compute
  P(O ?)
• Why is this hard? Sum over all possible sequences
  of states!

P(o1o2o3q0q0q0) P(o1o2o3q0q0q1) P(o1o2o3q0q
1q2) P(o1o2o3q0q1q0)
q2
q1
q0
q2
q2
q1
q1
q0
q0
q2
q0
q1
q0
o1
o2
o3
o4
oT
39
Evaluation for speech Summing over all different
paths!

• f ay ay ay ay v v v v
• f f ay ay ay ay v v v
• f f f f ay ay ay ay v
• f f ay ay ay ay ay ay v
• f f ay ay ay ay ay ay ay ay v
• f f ay v v v v v v v
• etc

40
Computing observation likelihood P(O?)

• Why cant we do an explicit sum over all paths?
• Because its intractable. O(NT)
• What we do instead
• The Forward Algorithm. O(N2T)

41
The Forward Algorithm
42
The inductive step, from Rabiner and Juang

• Computation of ?t(j) by summing all previous
  values ?t-1(i) for all i

?t-1(i)
?t(j)
43
The forward lattice for five
44
The Forward trellis computation for five
45
Forward trellis for Dow Jones
46
The Decoding Problem

• Given observations O(o1o2oT), and HMM
  ?(A,B,?), how do we choose best state sequence
  Q(q1,q2qt)?
• The forward algorithm computes P(OW)
• Could find best W by running forward algorithm
  for each W in L, picking W maximizing P(OW)
• But we cant do this, since number of sentences
  is O(WT). Instead
• Viterbi Decoding dynamic programming, slight
  modification of the forward algorithm
• A Decoding search the space of all possible
  sentences using the forward algorithm as a
  subroutine.

47
The Viterbi Algorithm
48
The Viterbi Algorithm
49
Viterbi for Dow Jones
50
The Viterbi Algorithm
51
The Viterbi Lattice
52
The Viterbi Lattice
53
Why Dynamic Programming

• I spent the Fall quarter (of 1950) at RAND. My
  first task was to find a name for multistage
  decision processes. An interesting question is,
  Where did the name, dynamic programming, come
  from? The 1950s were not good years for
  mathematical research. We had a very interesting
  gentleman in Washington named Wilson. He was
  Secretary of Defense, and he actually had a
  pathological fear and hatred of the word,
  research. Im not using the term lightly Im
  using it precisely. His face would suffuse, he
  would turn red, and he would get violent if
  people used the term, research, in his presence.
  You can imagine how he felt, then, about the
  term, mathematical. The RAND Corporation was
  employed by the Air Force, and the Air Force had
  Wilson as its boss, essentially. Hence, I felt I
  had to do something to shield Wilson and the Air
  Force from the fact that I was really doing
  mathematics inside the RAND Corporation. What
  title, what name, could I choose? In the first
  place I was interested in planning, in decision
  making, in thinking. But planning, is not a good
  word for various reasons. I decided therefore to
  use the word, programming I wanted to get
  across the idea that this was dynamic, this was
  multistage, this was time-varying I thought, lets
  kill two birds with one stone. Lets take a word
  that has an absolutely precise meaning, namely
  dynamic, in the classical physical sense. It also
  has a very interesting property as an adjective,
  and that is its impossible to use the word,
  dynamic, in a pejorative sense. Try thinking of
  some combination that will possibly give it a
  pejorative meaning. Its impossible. Thus, I
  thought dynamic programming was a good name. It
  was something not even a Congressman could object
  to. So I used it as an umbrella for my
  activities. Richard Bellman, Eye of the
  Hurrican an autobiography 1984.

Thanks to Chen, Picheny, Eide, Nock
54
HMMs for Speech

• We havent shown how to learn the A and B
  matrices for HMMs.
• This is done with an algorithm called EM
• We wont cover this today youll get this in the
  machine learning or ASR class.
• Instead lets turn now to applying the HMM to
  speech

55
Why Dynamic Programming

• I spent the Fall quarter (of 1950) at RAND. My
  first task was to find a name for multistage
  decision processes. An interesting question is,
  Where did the name, dynamic programming, come
  from? The 1950s were not good years for
  mathematical research. We had a very interesting
  gentleman in Washington named Wilson. He was
  Secretary of Defense, and he actually had a
  pathological fear and hatred of the word,
  research. Im not using the term lightly Im
  using it precisely. His face would suffuse, he
  would turn red, and he would get violent if
  people used the term, research, in his presence.
  You can imagine how he felt, then, about the
  term, mathematical. The RAND Corporation was
  employed by the Air Force, and the Air Force had
  Wilson as its boss, essentially. Hence, I felt I
  had to do something to shield Wilson and the Air
  Force from the fact that I was really doing
  mathematics inside the RAND Corporation. What
  title, what name, could I choose? In the first
  place I was interested in planning, in decision
  making, in thinking. But planning, is not a good
  word for various reasons. I decided therefore to
  use the word, programming I wanted to get
  across the idea that this was dynamic, this was
  multistage, this was time-varying I thought, lets
  kill two birds with one stone. Lets take a word
  that has an absolutely precise meaning, namely
  dynamic, in the classical physical sense. It also
  has a very interesting property as an adjective,
  and that is its impossible to use the word,
  dynamic, in a pejorative sense. Try thinking of
  some combination that will possibly give it a
  pejorative meaning. Its impossible. Thus, I
  thought dynamic programming was a good name. It
  was something not even a Congressman could object
  to. So I used it as an umbrella for my
  activities. Richard Bellman, Eye of the
  Hurrican an autobiogrpahy 1984.

Thanks to Chen, Picheny, Eide, Nock
56
HMMs for Speech

• We havent yet shown how to learn the A and B
  matrices for HMMs well do that on Thursday
• But lets return to think about speech

57
Reminder a word looks like this
58
Search space with bigrams
59
(No Transcript)
60
Viterbi trellis with 2 words and uniform LM
61
Viterbi backtrace
62
Learning

• Setting all the parameters in an ASR system
• Given
• training set wavefiles word transcripts for
  each sentence
• Hand-built HMM lexicon
• Initial acoustic models stolen from another
  recognizer
• train a LM on the word transcripts other data
• For each sentence, create one big HMM by
  combining all the HMM-words together
• Use the Viterbi algorithm to align HMM against
  the data, resulting in phone labeling of speech
• train new Gaussian acoustic models
• iterate (go to 2).

63
(No Transcript)
64
Evaluation

• How do evaluate the word string output by a
  speech recognizer?

65
Word Error Rate

• Word Error Rate
• 100 (InsertionsSubstitutions Deletions)
• ------------------------------
• Total Word in Correct Transcript
• Aligment example
• REF portable PHONE UPSTAIRS last
  night so
• HYP portable FORM OF STORES last
  night so
• Eval I S S
• WER 100 (120)/6 50

66
NIST sctk-1.3 scoring softareComputing WER with
sclite

• http//www.nist.gov/speech/tools/
• Sclite aligns a hypothesized text (HYP) (from the
  recognizer) with a correct or reference text
  (REF) (human transcribed)
• id (2347-b-013)
• Scores (C S D I) 9 3 1 2
• REF was an engineer SO I i was always with
  MEN UM and they
• HYP was an engineer AND i was always with
  THEM THEY ALL THAT and they
• Eval D S I
  I S S

67
Sclite output for error analysis

• CONFUSION PAIRS Total
  (972)
• With gt 1
  occurances (972)
• 1 6 -gt (hesitation) gt on
• 2 6 -gt the gt that
• 3 5 -gt but gt that
• 4 4 -gt a gt the
• 5 4 -gt four gt for
• 6 4 -gt in gt and
• 7 4 -gt there gt that
• 8 3 -gt (hesitation) gt and
• 9 3 -gt (hesitation) gt the
• 10 3 -gt (a-) gt i
• 11 3 -gt and gt i
• 12 3 -gt and gt in
• 13 3 -gt are gt there
• 14 3 -gt as gt is
• 15 3 -gt have gt that
• 16 3 -gt is gt this

68
Sclite output for error analysis

• 17 3 -gt it gt that
• 18 3 -gt mouse gt most
• 19 3 -gt was gt is
• 20 3 -gt was gt this
• 21 3 -gt you gt we
• 22 2 -gt (hesitation) gt it
• 23 2 -gt (hesitation) gt that
• 24 2 -gt (hesitation) gt to
• 25 2 -gt (hesitation) gt yeah
• 26 2 -gt a gt all
• 27 2 -gt a gt know
• 28 2 -gt a gt you
• 29 2 -gt along gt well
• 30 2 -gt and gt it
• 31 2 -gt and gt we
• 32 2 -gt and gt you
• 33 2 -gt are gt i
• 34 2 -gt are gt were

69
Better metrics than WER?

• WER has been useful
• But should we be more concerned with meaning
  (semantic error rate)?
• Good idea, but hard to agree on
• Has been applied in dialogue systems, where
  desired semantic output is more clear

70
Summary ASR Architecture

• Five easy pieces ASR Noisy Channel architecture
• Feature Extraction
• 39 MFCC features
• Acoustic Model
• Gaussians for computing p(oq)
• Lexicon/Pronunciation Model
• HMM what phones can follow each other
• Language Model
• N-grams for computing p(wiwi-1)
• Decoder
• Viterbi algorithm dynamic programming for
  combining all these to get word sequence from
  speech!

71
ASR Lexicon Markov Models for pronunciation
72
Summary

• Speech Recognition Architectural Overview
• Hidden Markov Models in general
• Forward
• Viterbi Decoding
• Hidden Markov models for Speech
• Next time Learning and EM