David Chen

1
CS 388Natural Language Processing Grounded
Language Acquisition

• David Chen
• Guest Lecture
• October 27, 2008

2
Semantics of Language

• The meaning of words, phrases, etc
• Crucial in communications

3
Semantics of Language

• The meaning of words, phrases, etc
• Crucial in communications
• Example
• Spanish goalkeeper Iker Casillas blocks the
  ball
• Merriam-Webster (transitive verb) to interfere
  usually legitimately with (as an opponent) in
  various games or sports
• WordNet (v) parry, deflect

4
Language Grounding

• Problem We are circularly defining the meanings
  of words in terms of other words.
• The meanings of many words are grounded in our
  perception of the physical world red, ball, cup,
  run, hit, fall, etc.
• Symbol Grounding Harnad (1990)
• Even many abstract words and meanings are
  metaphorical abstractions of terms grounded in
  the physical world up, down, over, in, etc.
• Lakoff and Johnsons Metaphors We Live By
• Its difficult to put my ideas into words.
• Interest in competitions is up.

5
Grounding Language

• Casillas blocks the ball

6
Grounding Language

• Casillas blocks the ball

Block(Casillas)
7
Grounding Language

• Casillas blocks the ball

Block(Casillas)
8
Natural Language and Meaning Representation

• Casillas blocks the ball

Block(Casillas)
9
Natural Language and Meaning Representation
Natural Language (NL)
Block(Casillas)

• Casillas blocks the ball

NL A language that has evolved naturally, such
as English, German, French, Chinese, etc
10
Natural Language and Meaning Representation
Meaning Representation Language (MRL)
Natural Language (NL)
Block(Casillas)

• Casillas blocks the ball

NL A language that has evolved naturally, such
as English, German, French, Chinese, etc MRL
Formal languages such as logic or any
computer-executable code
11
Semantic Parsing and Tactical Generation
NL
MRL
Block(Casillas)

• Casillas blocks the ball

Semantic Parsing (NL ? MRL)
Semantic Parsing maps a natural-language
sentence to a complete, detailed semantic
representation
12
Semantic Parsing and Tactical Generation
NL
MRL
Tactical Generation (NL ? MRL)
Block(Casillas)

• Casillas blocks the ball

Semantic Parsing (NL ? MRL)
Semantic Parsing maps a natural-language
sentence to a complete, detailed semantic
representation Tactical Generation Generates a
natural-language sentence from a meaning
representation.
13
Learning Approach
Semantic Parser Learner
Manually Annotated Training Corpora (NL/MRL
pairs)
Semantic Parser
MRL
NL
14
Learning Approach
Tactical Generator Learner
Manually Annotated Training Corpora (NL/MRL
pairs)
Tactical Generator
MRL
NL
15
Example of Annotated Training Corpus
Meaning Representation Language (MRL)
Natural Language (NL)

• Alice passes the ball to Bob
• Bob turns the ball over to John
• John passes to Fred
• Fred shoots for the goal
• Paul blocks the ball
• Paul kicks off to Nancy

Pass(Alice, Bob) Turnover(Bob, John) Pass(John,
Fred) Kick(Fred) Block(Paul) Pass(Paul, Nancy)
16
Example of Annotated Training Corpus
Meaning Representation Language (MRL)
Natural Language (NL)

• Alice passes the ball to Bob
• Bob turns the ball over to John
• John passes to Fred
• Fred shoots for the goal
• Paul blocks the ball
• Paul kicks off to Nancy

P1(C1, C2) P2(C2, C3) P1(C3, C4) P3(C4) P4(C5) P5(
C5, C6)
17
Learning Language from Perceptual Context

• Constructing annotated corpora for language
  learning is difficult
• Children acquire language through exposure to
  linguistic input in the context of a rich,
  relevant, perceptual environment
• Ideally, a computer system can learn language in
  the same manner

18
Goals

• Learn to ground the semantics of language

• Casillas blocks the ball

• Learn language through correlated linguistic and
  visual inputs

19
Challenge
20
Challenge
21
Challenge
??????????
22
Challenge
A linguistic input may correspond to many
possible events
??????????
?
?
?
23
Challenge
A linguistic input may correspond to many
possible events
??????????
?
Pass(GermanyPlayer1, GermanyPlayer2)
?
?
Block(SpanishGoalie)
Kick(GermanyPlayer2)
24
Overview

• Sportscasting task
• Related works
• Tactical generation
• Strategic generation
• Human evaluation

25
Learning to Sportscast

• Robocup Simulation League games
• No speech recognition
• Record commentaries in text form
• No computer vision
• Ruled-based system to automatically extract game
  events in symbolic form
• Concentrate on linguistic issues

26
Robocup Simulation League
27
Robocup Simulation League
Purple goalie blocked the ball
28
Learning to Sportscast

• Learn to sportscast by observing sample human
  sportscasts
• Build a function that maps between natural
  language (NL) and meaning representation (MR)
• NL Textual commentaries about the game
• MR Predicate logic formulas that represent
  events in the game

29
Robocup Sportscaster Trace
Natural Language Commentary
Meaning Representation
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
30
Robocup Sportscaster Trace
Natural Language Commentary
Meaning Representation
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
31
Robocup Sportscaster Trace
Natural Language Commentary
Meaning Representation
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
32
Robocup Sportscaster Trace
Natural Language Commentary
Meaning Representation
P6 ( C1, C19 )
P5 ( C1, C19 )
Purple goalie turns the ball over to Pink8
P1( C19 )
P2 ( C19, C22 )
Purple team is very sloppy today
P1 ( C22 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
P1 ( C22 )
P0
P1 ( C22 )
Pink11 makes a long pass to Pink8
P2 ( C22, C19 )
P1 ( C19 )
P2 ( C19, C22 )
Pink8 passes back to Pink11
33
Robocup Data

• Collected human textual commentary for the 4
  Robocup championship games from 2001-2004.
• Avg events/game 2,613
• Avg sentences/game 509
• Each sentence matched to all events within
  previous 5 seconds.
• Avg MRs/sentence 2.5 (min 1, max 12)
• Manually annotated with correct matchings of
  sentences to MRs (for evaluation purposes only).

34
Overview

• Sportscasting task
• Related works
• Tactical generation
• Strategic generation
• Human evaluation

35
Semantic Parser Learners

• Learn a function from NL to MR

NL Purple3 passes the ball to Purple5
Semantic Parsing (NL ? MR)
Tactical Generation (MR ? NL)
MR Pass ( Purple3, Purple5 )

• We experiment with two semantic parser learners
• WASP (Wong Mooney, 2006 2007)
• KRISP (Kate Mooney, 2006)

36
WASP Word Alignment-based Semantic Parsing

• Uses statistical machine translation techniques
• Synchronous context-free grammars (SCFG) Wu,
  1997 Melamed, 2004 Chiang, 2005
• Word alignments Brown et al., 1993 Och Ney,
  2003
• Capable of both semantic parsing and tactical
  generation

37
KRISP Kernel-based Robust Interpretation by
Semantic Parsing

• Productions of MR language are treated like
  semantic concepts
• SVM classifier is trained for each production
  with string subsequence kernel
• These classifiers are used to compositionally
  build MRs of the sentences
• More resistant to noisy supervision but incapable
  of tactical generation

38
KRISPER KRISP with EM-like Retraining

• Extension of KRISP that learns from ambiguous
  supervision Kate Mooney, 2007
• Uses an iterative EM-like method to gradually
  converge on a correct meaning for each sentence.

39
KRISPER
1. Assume every possible meaning for a sentence
is correct
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
40
KRISPER
1. Assume every possible meaning for a sentence
is correct
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
41
KRISPER
2. Resulting NL-MR pairs are weighted and given
to semantic parser learner
badPass ( Purple1, Pink8 )
1/2
turnover ( Purple1, Pink8 )
1/2
Purple goalie turns the ball over to Pink8
kick ( Pink8)
1/3
1/3
pass ( Pink8, Pink11 )
1/3
Purple team is very sloppy today
1/3
kick ( Pink11 )
1/3
1/3
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
1/3
1/3
kick ( Pink11 )
1/3
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
1/2
pass ( Pink8, Pink11 )
1/2
Pink8 passes back to Pink11
42
KRISPER
3. Estimate the confidence of each NL-MR pair
using the resulting trained semantic parser
badPass ( Purple1, Pink8 )
0.65
turnover ( Purple1, Pink8 )
0.87
Purple goalie turns the ball over to Pink8
kick ( Pink8)
0.22
0.35
pass ( Pink8, Pink11 )
0.13
Purple team is very sloppy today
0.85
kick ( Pink11 )
0.81
Pink8 passes the ball to Pink11
0.37
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
0.76
0.49
kick ( Pink11 )
0.76
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
0.67
pass ( Pink8, Pink11 )
0.86
Pink8 passes back to Pink11
43
KRISPER
4. Use maximum weighted matching on a bipartite
graph to find the best NL-MR pairs Munkres,
1957
badPass ( Purple1, Pink8 )
0.65
turnover ( Purple1, Pink8 )
0.87
Purple goalie turns the ball over to Pink8
kick ( Pink8)
0.22
0.35
pass ( Pink8, Pink11 )
0.13
Purple team is very sloppy today
0.85
kick ( Pink11 )
0.81
Pink8 passes the ball to Pink11
0.37
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
0.76
0.49
kick ( Pink11 )
0.76
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
0.67
pass ( Pink8, Pink11 )
0.86
Pink8 passes back to Pink11
44
KRISPER
4. Use maximum weighted matching on a bipartite
graph to find the best NL-MR pairs Munkres,
1957
badPass ( Purple1, Pink8 )
0.65
turnover ( Purple1, Pink8 )
0.87
Purple goalie turns the ball over to Pink8
kick ( Pink8)
0.22
0.35
pass ( Pink8, Pink11 )
0.13
Purple team is very sloppy today
0.85
kick ( Pink11 )
0.81
Pink8 passes the ball to Pink11
0.37
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
0.76
0.49
kick ( Pink11 )
0.76
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
0.67
pass ( Pink8, Pink11 )
0.86
Pink8 passes back to Pink11
45
KRISPER
5. Give the best pairs to the semantic parser
learner in the next iteration, and repeat until
convergence
badPass ( Purple1, Pink8 )
turnover ( Purple1, Pink8 )
Purple goalie turns the ball over to Pink8
kick ( Pink8)
pass ( Pink8, Pink11 )
Purple team is very sloppy today
kick ( Pink11 )
Pink8 passes the ball to Pink11
Pink11 looks around for a teammate
kick ( Pink11 )
ballstopped
kick ( Pink11 )
Pink11 makes a long pass to Pink8
pass ( Pink11, Pink8 )
kick ( Pink8 )
pass ( Pink8, Pink11 )
Pink8 passes back to Pink11
46
Overview

• Sportscasting task
• Related works
• Tactical generation
• Strategic generation
• Human evaluation

47
Tactical Generation

• Learn how to generate NL from MR
• Example
• Two steps
• Disambiguate the training data
• Learn a language generator

Pass(Pink2, Pink3) ? Pink2 kicks the ball to
Pink3
48
WASPER

• WASP with EM-like retraining to handle ambiguous
  training data.
• Same augmentation as added to KRISP to create
  KRISPER.

49
KRISPER-WASP

• First train KRISPER to disambiguate the data
• Then train WASP on the resulting unambiguously
  supervised data.

50
WASPER-GEN

• Determines the best matching based on generation
  (MR?NL).
• Score each potential NL/MR pair by using the
  currently trained WASP-1 generator.
• Compute NIST MT score NIST report, 2002 between
  the generated sentence and the potential matching
  sentence.

51
NIST scores
Target Purple2 quickly passes to Purple3
Candidate Purple2 passes to Purple3
1-grams Purple2, passes, to, Purple3 2-grams
Purple2 passes, passes to, to Purple3 3-grams
Purple2 passes to, passes to Purple3 4-gram
Purple2 passes to Purple3
52
NIST scores
Target Purple2 quickly passes to Purple3
Candidate Purple2 passes to Purple3
1-grams Purple2, passes, to, Purple3 2-grams
Purple2 passes, passes to, to Purple3 3-grams
Purple2 passes to, passes to Purple3 4-gram
Purple2 passes to Purple3
4/4
53
NIST scores
Target Purple2 quickly passes to Purple3
Candidate Purple2 passes to Purple3
1-grams Purple2, passes, to, Purple3 2-grams
Purple2 passes, passes to, to Purple3 3-grams
Purple2 passes to, passes to Purple3 4-gram
Purple2 passes to Purple3
4/4 2/3
54
NIST scores
Target Purple2 quickly passes to Purple3
Candidate Purple2 passes to Purple3
1-grams Purple2, passes, to, Purple3 2-grams
Purple2 passes, passes to, to Purple3 3-grams
Purple2 passes to, passes to Purple3 4-gram
Purple2 passes to Purple3
4/4 2/3 1/2
55
NIST scores
Target Purple2 quickly passes to Purple3
Candidate Purple2 passes to Purple3
1-grams Purple2, passes, to, Purple3 2-grams
Purple2 passes, passes to, to Purple3 3-grams
Purple2 passes to, passes to Purple3 4-gram
Purple2 passes to Purple3
4/4 2/3 1/2 0/1
56
System Overview
Sportscaster
Robocup Simulator
Pass ( Purple5, Purple7 )
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
57
System Overview
Sportscaster
Robocup Simulator
Pass ( Purple5, Purple7 )
Initial Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
58
System Overview
Purple7 loses the ball to Pink2
Kick ( pink2 )
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Initial Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
59
System Overview
Purple7 loses the ball to Pink2
Kick ( pink2 )
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
60
System Overview
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
61
System Overview
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
62
System Overview
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Pass ( pink5 , pink8)
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Ambiguous Training Data
63
KRISPER and WASPER
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Semantic Parser
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Semantic Parser Learner (KRISP/WASP)
Ambiguous Training Data
64
WASPER-GEN
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Sportscaster
Robocup Simulator
Pink2 kicks the ball to Pink5
Pass ( pink2 , pink5 )
Pink5 makes a long pass to Pink8
Kick ( pink5 )
Pink8 shoots the ball
Kick ( pink8 )
Unambiguous Training Data
Pass ( purple5, purple7 )
Tactical Generator
Turnover ( purple7 , pink2 )
Purple7 loses the ball to Pink2
Kick ( pink2 )
Pass ( pink2 , pink5 )
Pink2 kicks the ball to Pink5
Kick ( pink5 )
Pass ( pink5 , pink8)
Pink5 makes a long pass to Pink8
Ballstopped
Kick ( pink8 )
Pink8 shoots the ball
Tactical Generator Learner (WASP)
Ambiguous Training Data
65
Systems
66
Systems
Lower baseline
Upper baseline
67
Systems
Matching
Lower baseline
Upper baseline
68
Matching

• 4 Robocup championship games from 2001-2004.
• Avg events/game 2,613
• Avg sentences/game 509
• Leave-one-game-out cross-validation
• Metric
• Precision of systems annotations that are
  correct
• Recall of gold-standard annotations produced
• F-measure Harmonic mean of precision and recall

69
Matching Results
70
Systems
Lower baseline
Upper baseline
71
Systems
Semantic Parsing
Lower baseline
Upper baseline
72
Semantic Parsing

• 4 Robocup championship games from 2001-2004.
• Avg events/game 2,613
• Avg sentences/game 509
• Leave-one-game-out cross-validation
• Metric
• F-measure Harmonic mean of precision and recall
• Only count parses that are identical to the gold
  standard as correct

73
Semantic Parsing Results
74
Systems
Lower baseline
Upper baseline
75
Systems
Tactical Generation
Lower baseline
Upper baseline
76
Tactical Generation

• 4 Robocup championship games from 2001-2004.
• Avg events/game 2,613
• Avg sentences/game 509
• Leave-one-game-out cross-validation
• NIST score NIST report, 2002
• Evaluate the quality of machine translations
  based on matching n-grams

77
Tactical Generation Results
78
Overview

• Sportscasting task
• Related works
• Tactical generation
• Strategic generation
• Human evaluation

79
Strategic Generation

• Generation requires not only knowing how to say
  something (tactical generation) but also what to
  say (strategic generation).
• For automated sportscasting, one must be able to
  effectively choose which events to describe.

80
Example of Strategic Generation
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
81
Example of Strategic Generation
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
82
Strategic Generation

• For each event type (e.g. pass, kick) estimate
  the probability that it is described by the
  sportscaster.
• Requires correct NL/MR matching
• Use estimated matching from tactical generation
  (KRISPER, WASPER, WASPER-GEN, etc)
• Iterative Generation Strategy Learning

83
Iterative Generation Strategy Learning (IGSL)

• Directly estimates the likelihood of an event
  being commented on
• Self-training iterations to improve estimates
• Uses events not associated with any NL as
  negative evidence

84
EM for Strategic Generation
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
1/4
1/4
1/4
1/4
1/4
1/4
1/5
1/5
1/5
1/5
1/5
85
EM for Strategic Generation
Estimate Generation Probs
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
1/4
1/4
1/4
1/4
1/4
1/4
1/5
1/5
1/5
1/5
1/5
P(pass)(11/41/41/5)/30.57
86
EM for Strategic Generation
Estimate Generation Probs
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
1/4
1/4
1/4
1/4
1/4
1/4
1/5
1/5
1/5
1/5
1/5
P(pass)0.57
P(ballstopped)(1/4)/20.13
87
EM for Strategic Generation
Estimate Generation Probs
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
1/4
1/4
1/4
1/4
1/4
1/4
1/5
1/5
1/5
1/5
1/5
P(pass)0.57
P(ballstopped)0.13
P(kick)(1/41/41/51/5)/30.3
88
EM for Strategic Generation
Estimate Generation Probs
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
1/4
1/4
1/4
1/4
1/4
1/4
1/5
1/5
1/5
1/5
1/5
P(pass)0.57
P(ballstopped)0.13
P(kick)0.3
P(badpass)0.2
P(turnover)0.2
89
EM for Strategic Generation
Reassign link weights
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
0.57
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
0.57
0.13
0.3
0.57
0.3
0.57
0.3
0.57
0.3
0.2
0.2
P(pass)0.57
P(ballstopped)0.13
P(kick)0.3
P(badpass)0.2
P(turnover)0.2
90
EM for Strategic Generation
Normalize link weights
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1.0
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
0.36
0.08
0.2
0.36
0.35
0.65
0.19
0.36
0.19
0.13
0.13
P(pass)0.57
P(ballstopped)0.13
P(kick)0.3
P(badpass)0.2
P(turnover)0.2
91
EM for Strategic Generation
Recalculate Generation Probs and Repeat Until
Convergence
pass ( purple7 , purple6 ) ballstopped kick (
purple6 ) pass ( purple6 , purple2 )
ballstopped kick ( purple2 ) pass ( purple2 ,
purple3 ) kick ( purple3 ) badPass ( purple3 ,
pink9 ) turnover ( purple3 , pink9 )
1.0
purple7 passes the ball out to purple6
purple6 passes to purple2 purple2 makes
a short pass to purple3 purple3 loses the
ball to pink9
0.36
0.08
0.2
0.36
0.35
0.65
0.19
0.36
0.19
0.13
0.13
92
Strategic Generation Performance

• Evaluate how well the system can predict which
  events a human comments on
• Metric
• Precision of systems annotations that are
  correct
• Recall of gold-standard annotations correctly
  produced
• F-measure Harmonic mean of precision and recall

93
Strategic Generation Results
94
Overview

• Sportscasting task
• Related works
• Tactical generation
• Strategic generation
• Human evaluation

95
Human Evaluation (Quasi Turing Test)

• 4 fluent English speakers as judges
• 8 commented game clips
• 2 minute clips randomly selected from each of the
  4 games
• Each clip commented once by a human, and once by
  the machine
• Presented in random counter-balanced order
• Judges were not told which ones were human or
  machine generated

96
Demo Clip

• Game clip commentated using WASPER-GEN with IGSL,
  since this gave the best results for generation.
• FreeTTS was used to synthesize speech from
  textual output.

97
Human Evaluation
98
Human Evaluation
99
Future Work

• Expand MRs to beyond simple logic formulas
• Apply approach to learning situated language in a
  computer video-game environment (Gorniak Roy,
  2005)
• Apply approach to captioned images or video using
  computer vision to extract objects, relations,
  and events from real perceptual data (Fleischman
  Roy, 2007)

100
Instruction Generation

• GIVE Challenge
• http//www.give-challenge.org/
• Inaugural competition in generating instructions
  to help a human player navigate a maze and solve
  a puzzle
• Theory-neutral, end-to-end evaluation of NLG
  systems

101
Conclusion

• Current language learning work uses expensive,
  unrealistic training data.
• We have developed a language learning system that
  can learn from language paired with an ambiguous
  perceptual environment.
• We have evaluated it on the task of learning to
  sportscast simulated Robocup games.
• The system learns to sportscast almost as well as
  humans.