CS 904: Natural Language Processing ngrams in Speech Recognition

1
CS 904 Natural Language Processingn-grams in
Speech Recognition

• L. Venkata Subramaniam
• January 22, 2002

2
Linguistics and the Noisy Channel Model

• In linguistics we cant control the encoding
  phase. We want to decode the output to give the
  most likely input.

I
O
Noisy Channel p(oI)
decoder
3
The Noisy Channel Model for Speech Recognition

• i is the word sequence, o is the speech signal.
  So given an observed speech signal we want to get
  to the word sequence.
• p(i) is the language model and is the
  Acoustic model .

4
Language Models

• Provide constraints on sequence of words.
• Reduce search space
• Linguistic knowledge
• Domain knowledge

5
Statistical Speech Recognition
L EH T Z G OW DH ER
let's go there (valid) there go let's (not
valid) Lets go hair (not valid)
let's go there
6
n-grams

• p(go/let's) is high but p(lets'/go) is low
• These language characteristics are captured in
  the n-gram based LM.
• p(w1N) Pp(wi/wi-1) (bigram) or Pp(wi/wi-1,
  wi-2) (trigram)

7
Measuring Model Quality

• The ultimate measure of the quality of an LM is
  its impact on the application. In speech reco the
  impact can be measured in terms of the
  recognition accuracy. But that is indirect.
• A common alternative is to judge the LM by how
  well it predicts unseen text.

8
Cross Entropy

• We need to predict how well our LM M predicts
  unseen text L.
• A measure of this is Cross Entropy
  H(L,M)-limn-gtinf SxPT(x).logPM(x)/n l -
  logPM(x)/n
• A measure of How surprised is our model M on
  seeing L?

9
Perplexity

• Often, the perplexity of the text T with regard
  to the model M is reported.
• Perplexity (L,M)2 H(L,M)PM(x)-1/n
• A language with perplexity X has roughly the same
  difficulty as another language in which every
  word can be followed by X different words with
  equal probability.