CPSC 503 Computational Linguistics

1
CPSC 503Computational Linguistics

• Natural Language Generation
• Lecture 14
• Giuseppe Carenini

2
Knowledge-Formalisms Map for Generation
Intended meaning
Pragmatics Discourse and Dialogue
AI planners

• Logical formalisms
• (First-Order Logics)

Semantics
Rule systems (features and unification)
Syntax
Morphology
State Machines
Discourse (English)
3
NLG Systems (see handout)
NLG System

• Communicative Goals
• Domain Knowledge
• Context Knowledge

Text

• Examples
• FOG Input numerical data about future. Output
  textual wheatear forecasts
• IDAS Input KB describing a machinery (e.g.,
  bike), users level of expertise Output
  hypertext help messages
• ModelExplainer Input OO model. Output textual
  description of information on aspects of the
  model
• STOP Input user history and attitudes toward
  smoking Output personalize smoking cessation
  letter

4
GEA the Generator of Evaluative Arguments
5
Four Basic Types of Persuasive Text
-Arguments (main claim)

• Factual Argument (e.g., Canada is the only
  country outside of Asia to record SARS-related
  deaths.)
• Causal Argument (e.g., Travelers from Honk Kong
  brought SARS to Toronto.)
• Recommendation (e.g., You should not go to China
  in the next few weeks..)
• Evaluative Argument (e.g., Some Asian governments
  were inefficient in stopping the SARS outbreak)

6
Sample Textual Evaluative Arguments
7
Evaluative Arguments Importance

• Natural Language Generation Theory model of
  argument type which is pervasive in natural human
  communication.

• Ability to generate evaluative arguments is
  crucial in large classes of systems
• Personal assistants (e.g., travel advisor)
• Recommender systems (e.g., movie, book)
• Tutoring Systems

8
Limitations of Previous Research

• Ardissono and Goy 99 Chu-Carroll and Carberry
  98
• Elhadad 95 Kolln 95 Klein 94 Morik 89
• Focus on specific aspects of generation
• Selection of content
• Realization of content into language
• Lack of systematic evaluation
• proof-of-concept system
• analyzed on a few examples

9
Methodology

• Develop generator of evaluative arguments
• complete
• integrate and extend previous work
• Develop evaluation framework
• Perform experiment within framework to test
  generator

10
Outline

• Generator of Evaluative Arguments (GEA)
• Evaluation Framework
• Experiment
• More recent results from others

11
Text Generator Architecture
Content Selection and Organization
Content Realization
12
GEA User Model

• Argumentation Theory tells us Miller 96,
  Mayberry 96
• Supporting (opposing) evidence depends on values
  and preferences of audience
• Evidence arranged according to importance (i.e.,
  strength of support or opposition)
• Concise only important evidence included

• User Model must

and can be elicited in practice ...
13
Model of Users Preferences

• Additive Multi-attribute Value Function (AMVF)
• Decision Theory and Psychology (Consumers
  Behavior)
• Can be elicited in practice Edwards and Barron
  1994

14
AMVF application
OBJECTIVES
COMPONENT VALUE FUNCTIONS
Neighborhood
0.4
House-A
Location
0.7
Westend
0.6
House Value
Park-Distance
0.5 km
0.3
0.64
0.8
Amenities
20 m2
Deck-Size
36 m2
0.2
Porch-Size
15
Supporting and Opposing Evidence
0.4
Neighborhood
Location
0.6
0.78
0.7
0.6
House Value
Park-Distance
0.9
0.64
0.3
0.8
Amenities
Deck-Size
0.32
0.25
0.2
Porch-Size
0.6
16
Measure of Importance Klein 94
0.4
Neighborhood
Location
0.6
0.78
0.7
House-A
0.6
House Value
Park-Distance
n2
0.9
0.64
0.3
0.5 km
0.8
Amenities
Deck-Size
20 m2
0.32
0.25
36 m2
0.2
Porch-Size
0.6
17
Why AMVF? - summary

• An AMVF
• Represents users values and preferences
• Enables identification of supporting and opposing
  evidence
• Provides measure of evidence importance
• Evidence arranged according to importance
• Concise arguments can be generated
• Can be elicited in practice

18
GEA Architecture
Content Selection and Organization
Knowledge Sources - User Model - Domain Model
Text Planner
AMVF
Communicative Strategies
Text Plan
Content Realization
Text Micro-planner
Linguistic Knowledge Sources - Lexicon - Grammar
Sentence Generator
English
19
Argumentative Strategy
Based on guidelines from argumentation theory
Miller 96, Mayberry 96

• Selection include only important evidence
• (i.e., above threshold on measure of importance)
• Organization
• (1) Main Claim (e.g., This house is
  interesting)
• (2) Opposing evidence
• (3) Most important supporting evidence
• (4) Further supporting evidence -- ordered by
  importance with strongest last
• Strategy applied recursively on supporting
  evidence

20
Sample GEA Text Plan
EVALUATIVE ARGUMENT
SUPPORTING EVIDENCE
MAIN-CLAIM
(VALUE (House-A) 0.72)
SUB-CLAIM
OPPOSING EVIDENCE
SUPPORTING EVIDENCE
(VALUE (Location) 0.7)
(VALUE (distance-from-park 1.8m) 0.3)
(VALUE (distance-from-rap-trans 0.5 mi) 0.75)
(VALUE (distance-from-work 1mi) 0.75)
decomposition
ordering
rhetorical relations
21
GEA Architecture
Content Selection and Organization
Knowledge Sources - User Model - Domain Model
Text Planner
AMVF
Argumentative Strategy
Communicative Strategies
Text Plan
Content Realization
Text Micro-planner
Linguistic Knowledge Sources - Lexicon - Grammar
Sentence Generator
English
22
Text Micro-Planner

• Aggregation combining multiple propositions in
  one single sentence Shaw 98
• Lexicalization
• Scalar Adjectives (e.g., nice, far, convenient)
  Elhadad 93
• Discourse cues (e.g., although, because, in fact)
  Knott 96 Di Eugenio, Moore and Paolucci 97
• Pronominalization deciding whether to use a
  pronoun to refer to an entity (centering
  Grosz,Joshi and Weinstein 95)

23
Aggregation (Logical Forms)

• Conjunction via shared participants
• House B-11 is far from a shopping area
• House B-11 is far from public transportation
• House B-11 is far from a shopping area and
  public transportation.

• Syntactic embedding
• House B-11 offers a nice view
• House B-11 offers a view on the river
• House B-11 offers a nice view on the river.

24
Scalar Adjectives Selection
Value gt 0.8
The house has an excellent location
Value gt 0.8
The house has an excellent location

a convenient
a convenient
0.65 lt Value lt 0.8
0.65 lt Value lt 0.8
HOUSE-LOCATION
HOUSE-LOCATION
0.5 lt Value lt 0.65

a reasonable
0.5 lt Value lt 0.65

a reasonable
HAS_PARK_DISTANCE
0.35 lt Value lt 0.5

an average
HAS_PARK_DISTANCE
0.35 lt Value lt 0.5
an average

a bad
a bad
0.2 lt Value lt 0.35
0.2 lt Value lt 0.35
HAS_COMMUTING_DISTANCE
HAS_COMMUTING_DISTANCE
Value lt 0.2

a terrible
Value lt 0.2
a terrible
HAS_SHOPPING_DISTANCE
HAS_SHOPPING_DISTANCE
HOUSE-AMENITIES
HOUSE-AMENITIES
.
.
.
25
Discourse Cues Selection
Type-of- nesting
Rel-type
Typed-ordering
Discourse cue
Although (placed on contributor)
("CORE" "CONCESSION" "EVIDENCE")
CONCESSION
ROOT
However (placed on core)
("CONCESSION"CORE" "EVIDENCE") )
EVIDENCE
Even though (placed on contributor)
("CORE" "CONCESSION" "EVIDENCE")
EVIDENCE
SEQUENCE
26
Simple Pronominalization Strategy inspired by
Centering Theory

• Centering tells us entity providing link
  preferentially realized as pronoun (within a
  discourse segment)

• Our Strategy
• Within a discourse segment successive references
  always pronoun
• First reference in segment definite description
  unless
• Segment boundary explicitly marked by discourse
  cue and
• No pronoun was used in previous sentence

• House B-11 is an interesting house. In fact, it
  has a reasonable.

27
Output of MicroPlanning

• Sequence of Lexicalized Functional Descriptions
  (LFDs)
• Example
• House B-11 is close to shops and reasonably
  close to work
• ((CAT CLAUSE)
• (PROCESS
• ((TYPE ASCRIPTIVE) (MODE ATTRIBUTIVE)((POLARITY
  POSITIVE(EPISTEMIC-MODALITY NONE)))
• (PARTICIPANTS
• ((CARRIER
• ((CAT NP)(COMPLEX APPOSITION) (RESTRICTIVE YES)
• (DISTINCT
• ((AND ((CAT COMMON)(DENOTATION
  ZERO-ARTICLE-THING)(HEAD ((LEX "house"))))
• ((CAT PROPER) (LEX
  "B-11")))(CDR NONE))))
• (ATTRIBUTE
• (AND((CAT AP)(HEAD ((CAT ADJ)(LEX "close")))
• (QUALIFIER
• ((CAT PP)
• (PREP ((CAT PREP) (LEX
  "to")))
• (NP((CAT COMMON) (NUMBER
  PLURAL)(DEFINITE NO)

28
Last Step Sentence Generator

• Unify LFDs with large grammar of English
  (FUF/SURGE Elhadad 93, Robin 94)
• fill in syntactic constraints (e.g., agreement,
  ordering)
• choose closed class words (e.g., prepositions,
  articles)
• Apply morphology
• Linearize as English sentences

29
GEA Highlights

• GEA implements a computational model of
  generating evaluative arguments
• All aspects covered in a principled way
• argumentation theory (argumentative strategy and
  requirements on user model)
• decision theory (user model and elicitation
  method)
• computational linguistics (architecture,
  micro-planning techniques and sentence generator)

30
Outline

• Generator of Evaluative Arguments (GEA)
• Evaluation Framework
• Experiment
• More recent results from others

31
Evaluation Framework Task Efficacy
32
Selection Task in Real-Estate

• Why Real-Estate?
• No sophisticated background or expertise
• But still presents challenging decision task
• Instructions
• Move to new town
• Buy house
• Use system for data exploration

33
Data Exploration System
2-13
34
Argument is presented
2-13
35
Measures of Effectiveness

• Behavior and Attitude change
• Record of user actions
• Whether or not adopts new instance
• Position in Hot List
• Final Questionnaire
• How much likes new instance
• How much likes the instances in Hot-List
• Others (Final questionnaire)
• Decision Confidence
• Decision Rationale

36
Outline

• Generator of Evaluative Arguments (GEA)
• Evaluation Framework
• Experiment
• More recent results from others

37
Two Empirical Questions

• Argument content, structure and phrasing tailored
  to user-specific AMVF, but . . .
• Does this tailoring actually contribute to
  argument effectiveness?

• Arguments should be concise.
• Conciseness can be varied, but.
• What is the optimal level of conciseness?

38
Experimental Conditions

• Tailored-Concise ( 50 of objectives)
• Tailored-Verbose ( 80 of objectives)
• Non-Tailored-Concise ( 50 of objectives)
• No-Argument

39
Experimental Hypotheses
Tailored-Verbose
Tailored-Concise
Non-Tailored-Concise
No-Argument
40
Experimental Procedure
40 subjects (10 for each condition)
PHASE1 Online questionnaire to acquire
preferences (AMVF - 19 objectives, 3 layers)
Edwards and Barron 1994

• PHASE2
• - randomly assigned to condition
• interacts with evaluation framework
• - fill-out questionnaire

41
AMVF used in the experiment
42
Experiment Results

• Satisfaction Z-score
• Decision Confidence
• Decision Rationale

43
Results Satisfaction Z-score
Dennett test
Tailored-Verbose
p0.02
gt
Non-Tailored-Concise
Tailored-Concise
gt
p0.08
gt
p0.08
No-Argument
44
Summary
Generator of Evaluative Argument (GEA) generates
concise arguments tailored to a model of the
users preferences (AMVF)

• Evaluation Framework
• Basic decision tasks
• Evaluate wide range of generation techniques

• Experiment
• Differences in conciseness influence
  effectiveness
• Tailoring to AMVF seems to be effective

45
Future Work (in 2001!)

• Extend Argument Generator
• More Complex Textual Arguments
• Speech
• Other domains
• Other languages
• Arguments combining text and graphics

• More Experiments to test
• Whether tailoring to AMVF is actually effective
• Extensions

46
Multimodal Access to City Help (MATCH)
(ATT Johnston, Ehlen, Bangalore, Walker, Stent,
Maloor and Whittaker 2002)

• Multimodal interface
• Portable Fujitsu tablet
• Input Pen for deictic gestures and Speech input
• Output Text, Speech and graphics

47

• MATCH Example

User Show me Italian restaurants in the West
Village
User Recommend

• Comparison evaluative argument comparing at most
  five alternatives (reasons for choosing each of
  them)
• Recommendation evaluative argument about the
  best alternative

MATCH generates responses using techniques
inspired by GEA
48
MATCH Evaluation
CogSci 2004

• 16 subjects overheard 4x2 dialogues each about
  selecting a restaurant
• In each dialogue 6 arguments are generated (3
  tailored and 3 non-tailored)
• Subjects rate each argument information quality
  on 0-5 scale ..is easy to understand and it
  provides exactly the info I am interested in when
  choosing a restaurant

• 768 judgments (vs. 36 in our experiment)

Result tailored preferred plt.05
49
Conclusions

• Computational framework for generating and
  testing user-tailored evaluative arguments
• Argumentation theory
• Decision Theory
• Computational Linguistics
• Interactive Data Exploration
• Social Psychology
• Independent experiments indicate that proposed
  tailoring influences users behavior/attitudes

50
Next Time

• Project proposal deadline (bring your write-up to
  class)
• Project proposal Presentation
• 10 min presentation 3 min for questions
• For content, follow instructions at course
  project web page
• Bring 6 handouts
• If you want to use PowerPoint send me your
  presentation by Tue _at_12pm

51
Combining Measures of Behavior and Attitude

• Record of user interaction
• Whether or not adopts new instance
• Position in Hot List

- Ranking (no equality)
- Ranking with equality - Measure of difference

• Self-reports
• How much likes new instance
• How much likes (other)
• instances in Hot List

a) How would you judge the houses in your Hot
List? The more you like the house the closer you
should put a cross to good choice 1st
housebad choice ___ ___ ___ ___ __
___ ___ ___ ___ good choice 2nd housebad
choice ___ ___ ___ ___ __ ___ ___
___ ___ good choice 3rd house bad choice
___ ___ ___ ___ __ ___ ___ ___
___ good choice 4th housebad choice ___
___ ___ ___ __ ___ ___ ___ ___
good choice
Combine self-reports in a single, precise
measure satisfaction z-score for new instance
52
Satisfaction z-score

• For each subject compute

- sni is the measure of satisfaction with the new
instance - SHL is the set of measures of
satisfaction with the instances in the Hot-List
and the new instance
53
AMVF used in the experiment
54
Sample GEA Text Plan
EVALUATIVE ARGUMENT
SUPPORTING EVIDENCE
MAIN-CLAIM
(VALUE (House-A) 0.72)
SUB-CLAIM
OPPOSING EVIDENCE
SUPPORTING EVIDENCE
(VALUE (Location) 0.7)
(VALUE (distance-from-park 1.8m) 0.3)
(VALUE (distance-from-rap-trans 0.5 mi) 0.75)
(VALUE (distance-from-work 1mi) 0.75)
decomposition
ordering
rhetorical relations
55
Decision-Support Research Space
(Preferences)
Explain, Justify and Present
Representation and Inference
Elicitation
High-stakes
GEA
MATCH
Low-stakes
? (CF)
Individual
Group
56
Natural Language Generation

• Goal
• (computational model implemented as a) computer
  software which produces understandable texts in
  English or other human languages
• Input
• some underlying non-linguistic representation of
  information
• Output
• documents, reports, explanations, help messages,
  and other kinds of texts
• Knowledge sources required
• knowledge of language and of the domain

57
NLG Systems
NLG System

• Communicative Goals
• Domain Knowledge
• Context Knowledge

Text

• Examples
• FOG Input numerical data about future. Output
  textual wheatear forecasts
• IDAS Input KB describing a machinery (e.g.,
  bike), users level of expertise Output
  hypertext help messages
• ModelExplainer Input OO model. Output textual
  description of information on aspects of the
  model
• STOP Input user history and attitudes toward
  smoking Output personalize smoking cessation
  letter

58
Example z-score
a) How would you judge the houses in your Hot
List?The more you like the house the closer you
should put a cross to good choice 1st
housebad choice ___ ___ ___ ___ __
___ ___ ___ _ good choice 2nd
housebad choice ___ ___ ___ ___ __
___ ___ ___ good choice 3rd house
(NEW HOUSE) bad choice ___ ___ ___ ___
__ ___ ___ ___ good
choice 4th housebad choice ___ ___ ___
___ __ _ ___ ___ ___ good
choice