Introduction to Natural Language Processing

1
Introduction to Natural Language Processing

• Martin Volk
• Stockholm University
• volk_at_ling.su.se

2
The slides are taken from http//www.umiacs.umd.e
du/bonnie/courses/cmsc723-03 CMSC 723 / LING
645 Intro to Computational Linguistics
January 28, 2004 Lecture 1 (Dorr) Overview,
History, Goals, Problems, Techniques Prof. Bonnie
J. Dorr (University of Maryland)Dr. Nizar Habash
3
CL vs NLP

• Why Computational Linguistics (CL) rather than
  Natural Language Processing (NLP)?
•  
• Computational Linguistics
• Computers dealing with language
• Modeling what people do
• Natural Language Processing
• Applications on the computer side

4
Relation of CL to Other Disciplines
Electrical Engineering (EE) (Optical Character
Recognition)
 
Artificial Intelligence (AI) (notions of rep,
search, etc.)
Linguistics (Syntax, Semantics, etc.)
Machine Learning (particularly, probabilistic or
statistic ML techniques)
Psychology
CL
Philosophy of Language, Formal Logic
Human Computer Interaction (HCI)
Information Retrieval
Theory of Computation
5
A Sampling of Other Disciplines

• Linguistics formal grammars, abstract
  characterization of what is to be learned.
• Computer Science algorithms for efficient
  learning or online deployment of these systems in
  automata.
• Engineering stochastic techniques for
  characterizing regular patterns for learning and
  ambiguity resolution.
• Psychology Insights into what linguistic
  constructions are easy or difficult for people to
  learn or to use

6
History 1940-1950s

• Development of formal language theory (Chomsky,
  Kleene, Backus).
• Formal characterization of classes of grammar
  (context-free, regular)
• Association with relevant automata
• Probability theory language understanding as
  decoding through noisy channel (Shannon)
• Use of information theoretic concepts like
  entropy to measure success of language models.

7
1957-1983 Symbolic vs. Stochastic

• Symbolic
• Use of formal grammars as basis for natural
  language processing and learning systems.
  (Chomsky, Harris)
• Use of logic and logic based programming for
  characterizing syntactic or semantic inference
  (Kaplan, Kay, Pereira)
• First toy natural language understanding and
  generation systems (Woods, Minsky, Schank,
  Winograd, Colmerauer)
• Discourse Processing Role of Intention, Focus
  (Grosz, Sidner, Hobbs)
• Stochastic Modeling
• Probabilistic methods for early speech
  recognition, OCR (Bledsoe and Browning, Jelinek,
  Black, Mercer)

8
1983-1993 Return of Empiricism

• Use of stochastic techniques for part of speech
  tagging, parsing, word sense disambiguation, etc.
• Comparison of stochastic, symbolic, more or less
  powerful models for language understanding and
  learning tasks.

9
1993-Present

• Advances in software and hardware create NLP
  needs for information retrieval (web), machine
  translation, spelling and grammar checking,
  speech recognition and synthesis.
• Stochastic and symbolic methods combine for real
  world applications.

10
Language and Intelligence Turing Test

• Turing test
• machine, human, and human judge
• Judge asks questions of computer and human.
• Machines job is to act like a human, humans job
  is to convince judge that hes not the machine.
• Machine judged intelligent if it can fool
  judge.
• Judgement of intelligence linked to appropriate
  answers to questions from the system.

11
ELIZA

• Remarkably simple Rogerian Psychologist
• Uses Pattern Matching to carry on limited form of
  conversation.
• Seems to Pass the Turing Test! (McCorduck,
  1979, pp. 225-226)
• Eliza Demo

http//www.lpa.co.uk/pws_dem4.htm
12
Whats involved in an intelligent Answer?
Analysis Decomposition of the signal (spoken
or written) eventually into meaningful units.
This involves
13
Speech/Character Recognition

• Decomposition into words, segmentation of words
  into appropriate phones or letters
• Requires knowledge of phonological patterns
• Im enormously proud.
• I mean to make you proud.

14
Morphological Analysis

• Inflectional
• duck s N duck plural s
• duck s V duck 3rd person s
• Derivational
• kind, kindness
• Spelling changes
• drop, dropping
• hide, hiding

15
Syntactic Analysis

• Associate constituent structure with string
• Prepare for semantic interpretation

16
Semantics

• A way of representing meaning
• Abstracts away from syntactic structure
• Example
• First-Order Logic watch(I,terrapin)
• Can be I watched the terrapin or The terrapin
  was watched by me
• Real language is complex
• Who did I watch?

17
Lexical Semantics
The Terrapin, is who I watched. Watch the
Terrapin is what I do best. Terrapin is what I
watched the I experiencer Watch the Terrapin
predicate The Terrapin patient
18
Compositional Semantics

• Association of parts of a proposition with
  semantic roles
• Scoping

19
Word-Governed Semantics

• Any verb can add able to form an adjective.
• I taught the class . The class is teachable
• I rejected the idea. The idea is rejectable.
• Association of particular words with specific
  semantic forms.
• John (masculine)
• The boys ( masculine, plural, human)

20
Pragmatics

• Real world knowledge, speaker intention, goal of
  utterance.
• Related to sociology.
• Example 1
• Could you turn in your assignments now (command)
• Could you finish the homework? (question,
  command)
• Example 2
• I couldnt decide how to catch the crook. Then I
  decided to spy on the crook with binoculars.
• To my surprise, I found out he had them too.
  Then I knew to just follow the crook with
  binoculars.
• the crook with binoculars
• the crook with binoculars

21
Discourse Analysis

• Discourse How propositions fit together in a
  conversationmulti-sentence processing.
• Pronoun reference The professor told the
  student to finish the assignment. He was pretty
  aggravated at how long it was taking to pass it
  in.
• Multiple reference to same entityGeorge W.
  Bush, president of the U.S.
• Relation between sentencesJohn hit the man. He
  had stolen his bicycle

22
NLP Pipeline
speech
text
Phonetic Analysis
OCR/Tokenization
Morphological analysis
Syntactic analysis
Semantic Interpretation
Discourse Processing
23
Relation to Machine Translation
input
analysis
generation
output
Morphological analysis
Morphological synthesis
Syntactic analysis
Syntactic realization
Semantic Interpretation
Lexical selection
Interlingua
24
Ambiguity

• I made her duck.
• I made duckling for her
• I made the duckling belonging to her
• I created the duck she owns
• I forced her to lower her head
• By magic, I changed her into a duck

25
Syntactic Disambiguation
S
S NP
VP NP
VP I V NP VP
I V NP
made her V
made det N
duck
her duck

• Structural ambiguity

26
Part of Speech Tagging and Word Sense
Disambiguation

• verb Duck !
• noun Duck is delicious for dinner
• I went to the bank to deposit my check.
• I went to the bank to look out at the river.
• I went to the bank of windows and chose the
  one dealing with last names beginning with d.

27
Resources forNLP Systems

• Dictionary
• Morphology and Spelling Rules
• Grammar Rules
• Semantic Interpretation Rules
• Discourse Interpretation
• Natural Language processing involves (1) learning
  or fashioning the rules for each component, (2)
  embedding the rules in the relevant automaton,
  (3) and using the automaton to efficiently
  process the input .

28
Some NLP Applications

• Machine TranslationBabelfish (Alta Vista)
• Question AnsweringAsk Jeeves (Ask Jeeves)
• Language SummarizationMEAD (U. Michigan)
• Spoken Language Recognition EduSpeak (SRI)
• Automatic Essay evaluationE-Rater (ETS)
• Information Retrieval and ExtractionNetOwl
  (SRA)

http//babelfish.altavista.com/translate.dyn
http//www.ask.com/
http//www.summarization.com/mead
http//www.eduspeak.com/
http//www.ets.org/research/erater.html
http//www.netowl.com/extractor_summary.html