SUBTLE, Implicit Meaning

1
SUBTLE, Implicit Meaning Plan Recognition

• CIS 530 Introduction to NLP
• FINAL LECTURE!!
• Spring 2007

2
Implicit Meaning

• Exchange between firefighters
• (Worcester Cold Storage and Warehouse Fire 1999)
• L-1/P2
• first answers question
• then appears to give command
• L-1/P1s implicit question How can we find you?
• Linguists call this the Question Under Discussion
  (QUD)
• L-1/P2 answers the QUD

3
Scripts Understanding Prototypical Situations

• Also Useful for Planning and Plan Recognition

4
Scripts for Buy Purchase-Ticket
(this following from James Allen, Natural
Language Understanding, 1995)
5
A (Simplistic) Script for Travel-by-Train
6
Triggering a Script in Text

• Describe an action as a goal
• Sam wanted to take the train to Rochester
• Describe a state as a goal
• Jack needed to be in Rochester
• Describe an action in execution
• Jack walked up to the ticket booth
• How might this work?

7
Using Scripts to Analyze Text FragmentsA
Simplistic First Account

• Sam wanted to take a train to Rochester.
• He purchased a ticket at the station.
• (1) triggers a script
• (2) describes a step in the decomposition
• After (1)
• (2) confirms PurchaseTicket after appropriate
  matching

8
A Plan to Travel by Train

• Recognized using Plan Inference
• Method Backchaining from Goal using Action
  Library

9
Plan Recognition Discourse

• Sue wanted to take a train to Rochester
• She walked up to the ticket clerk

10
Scripts for SUBTLE The PAR Representation

• Allows NL to be used to interact with, and
  control smart avatars and individualized agents.
• Parameterized Action Representation
• Participants
• Conditions (preparatory, termination,
  applicability)
• Execution steps
• Core semantics (motion, force, path, manner, )
• Actionary
• Links natural language and actions.
• Holds persistent, hierarchical definitions
  (database) of actions as PARs.
• Contains persistent, hierarchical definitions of
  objects including agents.

(this following from Prof. Norm Badler
11
Parameterized Action Representation (PAR) Part
of an Embodied Agent Architecture
Virtual agent
Synthesized Movements (animation)
Internal State
Language (instructions)
PAR
Language Generation (text)
Observed Movements (computer vision)
Actionary
August 10, 2007
11
MURI Kick-Off Meeting
12
Carrying Out Instructions

• Effective instructions allow agents to carry out
  instructions correctly as intended.
• Information in effective instructions
• Core action semantics (e.g. remove)
• Participants (agent, objects)
• Path, manner, purpose information (context)
• Initiation conditions (applicability
  preconditions)
• Termination conditions (success or failure
  cases)

August 10, 2007
12
MURI Kick-Off Meeting
13
Action Conceptual Complexity

• Action conceptualization affects description
• State-Space black box (atomic/instantaneous)
  action
• Unplug the power cord
• Kinematic motion over time
• Climb the stairs
• Dynamic forces and motion over time
• Smash in the door
• Constraint
• Dont damage the wall

August 10, 2007
13
MURI Kick-Off Meeting
14
Modifying/Extending Actions using Parameters

• Action type reflected in verb can be changed
• state-space (e.g., with an adverb)
• Open the door completely.
• motion (e.g., with a path prepositional phrase)
• Open the door towards you.
• force (e.g., with a manner adverb)
• Open the door carefully.
• constraint (e.g., with a conditional expression)
• Open the door just enough to see whats on the
  other side.

August 10, 2007
14
MURI Kick-Off Meeting
15
Modelling Actions with PAR

• duration timing, iteration, or extent e.g.,
  for 6 seconds, between
  5 and 6 times
• sub-steps actions to perform to accomplish
  action (include parallel constructs)
• next-step next action to be performed
• super-step parent action

• core semantics motion, force, state-change
• participants agent, objects
• direction/path direction or path of motion
• purpose state to achieve, action to generate,
  etc.
• manner how to perform action (e.g. carefully)

August 10, 2007
15
MURI Kick-Off Meeting
16
Lets Look at a PragBot Transcript
17
Picking a suit

• Player 1, 37281 Hey
• Player 2, 66164 Hi
• Player 1, 102642 Ok, so let's start moving
  around and each pick up a few cards, then we can
  see which suit we have more of and go for those
• Explicit Plan (a) Move around pick up cards,
  (b) Pick maximized suit
• Player 1, 176511 I have 3 heartsPlayer 1,
  186262 Can you find three hearts?
• Implies that Hearts has been chosen as the suit
  that we are looking for
• Player 1, 247185 I found a heart in the upper
  left corner if you're looking
• Implicit I want you to come and pick this card
  up
• Player 2, 272275 I have three spades if you can
  find any
• (Oops!)
• Player 1, 279092 I have three heartsPlayer 1,
  289002 Didn't you read what I said above?
• Looking for a simple yes/no?
• Player 1, 301865 Are you in the upper left
  corner?
• I assume that you have adopted the suit that I
  chose and are now following my plan in 5
• Player 2, 361591 Yes I have 3 hearts
• Overanswer

18
Plan recognition

• Player 1, 505858 i have 7, A, 3Player 1,
  507211 you?
• Ellided input
• Player 2, 521982 5,2, and 6
• More ellision
• Player 1, 529151 ok 5 , 6 ,7
• Lots of ellision
• We have three Hearts in a row (5,6,7), so let's
  build off of those

19
Building off of 5 6 7

• Player 1, 548123 ill keep looking, if you find
  one that fits let me know and vice versa
• Explicit plan, again
• Player 2, 555958 Drop your A and see if you can
  find a 4
• Whats P2s plan? Is it optimal? Does P1 adopt
  it?
• Player 1, 574113 I have a 8Player 1, 579651
  I'm going to drop A for 8Player 1, 604200 so
  now we have 3 _ 5 6 7 8

20
Building off of 3 _ 5 6 7 8

• Player 1, 617743 ok I ound 9
• Player 2, 619988 And a 2
• Again, whats P2s plan? Optimal?
• Player 1, 624067 can you come and pick it up?
• Pick up which?
• Player 1, 630494 and I'll look for 4 tooPlayer
  1, 635382 9 is in the bottom left corner
• Player 2, 638936 Where are you?
• Player 1, 653502 bottom left corner
• Player 2, 658764 Got it.
• Player 1, 709541 Nice

21
Finishing Up

• Player 1, 713152 ok so we need 4?
• Player 2, 725533 Yes.Player 2, 753811 Just to
  verify... what do you have? I have 5, 6, and 9.
• Player 1, 779144 we have 3, 5, 6, 7, 8, 9Player
  1, 781794 just need the 4Player 1, 794187
  although do we konw that the 4 is out there?
• Player 2, 819505 Becuase I just found it.
• Player 1, 832098 Good job!
• Theyre done!

22
One last exchange

• Player 1, 834964 Where???
• Player 2, 918441 Ok... From the top left corner,
  go down about 4 spaces and then go as far right
  as you can. Next go all the way up as far as you
  can. Then go 4 spaces to the right.
• Player 1, 959266 I just wanted an idea because I
  hadn't found it