Improving Chinese handwriting Recognition by Fusing speech recognition

1
Improving Chinese handwriting Recognition by
Fusing speech recognition

• Zhang Xi-Wen
• CSE, CUHK and HCI Lab., ISCAS
• 2005.4.12

2
Outline

• 1 Chinese handwriting recognition
• 2 Chinese speech recognition
• 3 Information fusion
• 4 Experimental results

3
Handwriting Recognition

• Handwriting segmentation
• Character recognition

4
1.1 Handwriting segmentation

• It is more difficult for Chinese handwriting
  segmentation

5
Character extraction using histogram

• A histogram of between-stroke gaps.
• The dimidiate threshold of the histogram is to
  extract lines of strokes.
• The dimidiate threshold of the histogram of a
  line of strokes is to extract characters.

6
Figure 1. Handwriting segmentation
7
Problems remained

• A Chinese character may be mis-segmented into
  many characters.
• Many Chinese characters may be mis-grouped as a
  character.
• The segmentation error will inevitably result in
  handwriting recognition errors.

8
1.2 Character recognition

• Isolated character recognizer from HW
• Many candidates

9
Handwriting.
Text recognized from the handwriting.
The ground-truth text.
Figure 2. Handwriting recognition
10
2 Speech recognition

• Chinese speech.
• On-line, microphone.
• Continuous speech recognizer from MS.

11
Text recognized from the speech corresponding to
the handwriting.
The ground-truth text.
Figure 3. Speech recognition
12
3 Text fusion

• An optimization problem
• Dynamic Programming

13
3.1 Principles

• The fused text should contain more semantic
  information.
• Construct a text with the least characters and
  the most semantic information.

14
3.2 Four ways
Text recognized from the handwriting.
Text recognized from the speech corresponding to
the handwriting.
Figure 4. Texts to be fused
15
3.3 Dynamic Programming

• A directed graph.
• Optimal paths.

16
Figure 5. A directed graph with N levels.
17
(a) Text recognized from the handwriting.
(b) Text recognized from the speech corresponding
to the handwriting.
(c) The optimal fused text corresponding to the
optimal path.
(d) The ground-truth text.
Figure 6. Text fusion using DP.
18
3.4 A language model

• Lexicon
• Syntax
• Semantic

19
Lexicon
20
(No Transcript)
21
4 Experimental results
22
(No Transcript)
23
(No Transcript)
24

• Thank you very much for
• your criticism, comments and suggestions!
• Email xwzhang_at_cse.cuhk.edu.hk
• Tel [phone]