Natural Language Processing

1
Natural Language Processing

• Lecture Notes 1

2
Today

• Administration and Syllabus
• course web page
• Introduction

3
Natural Language Processing

• What is it?
• What goes into getting computers to perform
  useful and interesting tasks involving human
  languages.
• Secondarily insights that such computational
  work give us into human languages and human
  processing of language.

4
Natural Language Processing

• Foundations are in computer science (AI, theory,
  algorithms,) linguistics mathematics logic
  and statistics and psychology

5
Why Should You Care?

• Two trends
• An enormous amount of knowledge is now available
  in machine readable form as natural language text
• Conversational agents are becoming an important
  form of human-computer communication

6
Knowledge of Language

• Words (words and their composition)
• Syntax (structure of sentences)
• Semantics (explicit meaning of sentence)
• Discourse and pragmatics (implicit and contextual
  meaning)

7
Small Applications

• Line breakers
• Hyphenators
• Spelling correctors
• Optical Character Recognition software
• Grammar and style checkers

8
Big Applications

• Question answering
• Conversational agents
• Text summarization
• Machine translation

9
Note

• NLP, as in many areas of AI
• Were often dealing with ill-defined problems
• We dont often come up with perfect
  solutions/algorithms
• We cant let either of those facts get in our way

10
Course Material

• Well be intermingling discussions of
• Linguistic topics
• Syntax and meaning representations
• Computational techniques
• Context-free grammars
• Applications
• Translation and QA systems

11
Chapter 1

• Knowledge of language
• Ambiguity
• Models and algorithms
• History

12
Knowledge of Language

• Phonetics and phonology speech sounds, their
  production, and the rule systems that govern
  their use
• Morphology words and their composition from more
  basic units
• Cat, cats (inflectional morphology)
• Child, children
• Friend, friendly (derivational morphology)

13
Knowledge of Language

• Syntax the structuring of words into legal
  larger phrases and sentences

14
Semantics

• The meaning of words and phrases
• Lexical semantics the study of the meanings of
  words
• Compositional semantics how to combine word
  meanings
• Word-sense disambiguation
• River bank vs. financial bank

15
Pragmatics

• Indirect speech acts
• Do you have a stapler?
• Presupposition
• Have you stopped beating your wife?
• Deixis and point of view
• Zoe was angry at Joe. Where was he?
• Implicature
• -Yes, there are 3 flights to Boston. In fact,
  there are 4.
• The general was assassinated. In fact, he
  isnt dead.

16
Discourse

• Utterance interpretation in the context of the
  text or dialog
• Sue took the trip to New York. She had a great
  time there.
• Sue/she
• New York/there
• took/had (time)

17
Ambiguity

• Almost all of the non-trivial tasks performed by
  NLP systems are ambiguity resolution tasks
• There is ambiguity at all levels of language

18
Ambiguity

• I saw the woman with the telescope
• Syntactically ambiguous
• I saw (NP the woman with the telescope)
• I saw (NP the woman) (PP with the telescope)

19
I made her duck

• I cooked waterfowl for her
• I cooked waterfowl belonging to her
• I create the duck she owns
• I caused her to lower her head quickly
• Part of speech tagging is duck a noun or verb?
• Parsing syntactic structure is her part of
  the duck NP?
• Word-sense disambiguation (lexical semantics)
  does make mean create, lower head, or cook?

20
Dealing with Ambiguity

• Two approaches
• Tightly coupled interaction among processing
  levels knowledge from other levels can help
  decide among choices at ambiguous levels.
• Pipeline processing
• Most NLP systems are probabilistic they make
  the most likely choices

21
Models and Algorithms

• Models (as we are using the term here)
• Formalisms to represent linguistic knowledge
• Algorithms
• Used to manipulate the representations and
  produce the desired behavior
• choosing among possibilities and combining pieces

22
Models

• State Machines finite state automata, finite
  state transducers
• Formal rule systems context free grammars
• Logical formalisms first-order predicate
  calculus higher-order logics
• Models of uncertainty Bayesian probability theory

23
Algorithms

• Many of the algorithms that well study will turn
  out to be transducers algorithms that take one
  kind of structure as input and output another.

24
Algorithms

• In particular..
• State-space search
• To manage the problem of making choices during
  processing when we lack the information needed to
  make the right choice
• Dynamic programming
• To avoid having to redo work during the course of
  a state-space search
• Machine Learning (classifiers, EM, etc)

25
State Space Search

• States represent pairings of partially processed
  inputs with partially constructed answers
• E.g. sentence partial parse tree
• Goal is to arrive at the right/best structure
  after having processed all the input.
• E.g. the best parse tree spanning the sentence
• As with most interesting AI problems the spaces
  are too large and the criteria for bestness are
  difficult to encode (so heuristics, probabilities)

26
Dynamic Programming

• Dont do the same work over and over.
• Avoid this by building and making use of
  solutions to sub-problems that must be invariant
  across all parts of the space.