Bailey Model

1
Bailey Model

• Showed how simple hand action verbs may be
  acquired based on motor control schemas and
  parameterization.
• Used Model Merging which allowed for
• One-shot learning (Maps to recruitment learning)
• Could label and perform actions (given a command,
  world state pair)
• Uses parameters over motor-control schemas as
  inductive bias
• Limitations
• Inference
• Connections between events
• Abstract uses
• Event Structure
• Frames
• Metaphor

2
Event Structure-1

• Srini Narayanan
• CS182/CogSci110/Ling109
• Spring 2006
• snarayan_at_icsi.berkeley.edu

3
Active representations

• Many inferences about actions derive from what we
  know about executing them
• Representation based on stochastic Petri nets
  captures dynamic, parameterized nature of actions

Walking bound to a specific walker with a
direction or goal consumes resources (e.g.,
energy) may have termination condition(e.g.,
walker at goal) ongoing, iterative action
4
X-Schema Extensions to Petri Nets

• Parameterization
• x-schemas take parameter values (speed, force)
• Walk(speed slow, dest store1)
• Dynamic Binding
• X-schemas allow run-time binding to different
  objects/entities
• Grasp(cup1), push(cart1)
• Hierarchical control and durative transitions
• Walk is composed of steps which are composed of
  stance and swing phases
• Stochasticity and Inhibition
• Uncertainties in world evolution and in action
  selection

5
Event Structure in Language

• Commonplace discourse fragments/blurbs
• Low inflation is starting to pull France out of
  recession.
• E3 continue to push Iran to uphold IAEA
  obligations.
• US Economy on the verge of falling back into
  recession after moving forward on an anemic
  recovery.
• Indian Government stumbling in implementing
  Liberalization plan.
• Moving forward on all fronts, we are going to be
  ongoing and relentless as we tighten the net of
  justice.
• The Government is taking bold new steps. We are
  loosening the stranglehold on business, slashing
  tariffs and removing obstacles to international
  trade.

6
Event Structure in Language

• Fine-grained
• Rich Notion of Contingency Relationships.
• Phenomena Aspect, Tense, Force-dynamics, Modals,
  Counterfactuals
• Event Structure Metaphor
• Phenomena Abstract Actions are conceptualized in
  Motion and Manipulation terms.
• Schematic Inferences are preserved.

7
Aspect

• Aspect is the name given to the ways languages
  describe the structure of events using a variety
  of lexical and grammatical devices.
• Viewpoints
• is walking, walk
• Phases of events
• Starting to walk, walking, finish walking
• Inherent Aspect
• run vs cough vs. rub
• Composition with
• Temporal modifiers, tense..
• Noun Phrases (count vs. mass) etc..

8
Grammatical Aspect

• Languages have grammatical constructions that
  indicate the type of situation described.
• Progressive She was running home.
• Perfect Ive had a wonderful evening.
• Inceptive She started knitting.
• Prospective Shes about to leave.
• Resumptive Peace talks resume.
• Iterative They ran twice around the track.

9
Phases, Viewpoints, and Aspects

• John is walking to the store.
• John is about to walk to the store.
• John walked to the store.
• John started walking to the store.
• John is starting to walk to the store.
• John has walked to the store.
• John has started to walk to the store.
• John is about to start walking to the store.
• John resumed walking to the store.
• John has been walking to the store.
• John has finished walking to the store.
• John almost walked to the store.

10
A Walk X-schema
11
A Climb X-schema
12
Common Patterns
START
FINISH
Posture Up Energy Available Ready Dest top(obj) Loop BEGIN Execute (subschema) END At Dest Done
Posture Up Ground ok Ready Loop BEGIN Execute(subschema) END At Dest Done
13
Pre-motor Versus Motor Cortex Whenever we
perform a complex motor movement, such as picking
up a glass and taking a drink, at least two
distinct parts of the brain are activated The
motor cortex, where there are neural ensembles
that control motor synergies relatively
simple actions like opening or closing the hand,
flexing or extending the elbow, turning the
wrist, and so on. Complex motor schemas,
however, are carried out by neural circuitry in
the pre-motor cortex, circuitry connected via
neural bindings to the appropriate synergies in
the motor cortex. In picking up a glass and
taking a drink, both pre-motor cortex and motor
cortex are activated, as are binding connections
between them.
14
The Controller X-Schema In modeling complex
premotor action schemas, we make the following
hypothesis All complex premotor schemas are
compositions of a single type of structure. The
same neural computational structure, when
disengaged from the motor cortex, can
characterize aspect (that is, event structure) in
the worlds languages. When dynamically active,
this structure can compute the logic of aspect.
We call this structure the Controller
X-schema.
15

• The Structure of the Controller X-Schema
• Initial State
• Starting Phase Transition
• Precentral State
• Central Phase Transition (either instantaneous,
• prolonged, or ongoing)
• Postcentral State
• Ending Phase Transition
• Final State
• Postcentral Options
• A check to see if a goal state has been
  achieved
• An option to stop/resume
• An option to iterate or continue the main
  process
• -Narayanan, 1997

16
(No Transcript)
17
A Schema Controller
iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend

• An active controller that sends signals to the
  embedded schema and transitions based on signals
  from the embedded schema.
• Useful for higher level monitoring and
  coordination of actions.

18
A Generic Process Schema
iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend

• Part of Conceptual Structure.
• Generalizes over actions and events. Has
  internal state and models evolution of processes.

19
Aspects of (Climb)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
20
About to (Climb) (Prospective)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
21
Cancel (Climb)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
22
Start (Climb)-ING
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
23
Be (Climb)-ING (Progressive)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
24
Suspend (Climb)-ING
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
25
Resumed (Climb)-ING (Resumptive)
26
Finish (End) (Climb)-ING
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
27
Have (Climb)-ed (Perfect)
Iterate
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
Cancel
BINDINGS
Energy Ready Standing
On top
28
Embedding Has Started (to X)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt

X-Schema for X with bindings
29
Phasal Aspect Maps to the Controller
Iterative (repeat)
Inceptive (start, begin)
Iterate
Ready
Done
Start
Process
Finish
interrupt
resume
Cancel
Suspend
Completive (finish, end)
Resumptive(resume)
30
Embedding About to start (X)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
R
D
S
P
F
r
i
S
C
X-Schema for X with bindings
31
Embedding Has Started (to X)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt

X-Schema for X with bindings
32
Begins and Ends

• This is not the end. It is not even the
  beginning of the end. But it is, perhaps, the end
  of the beginning."
• Speech given at the Lord Mayor's Luncheon,
  Mansion House, London, November 10, 1942.
  Winston Churchill

33
Embedding Its not this (the end)
Ongoing
Finish
Done

X-Schema for X with bindings
34
Embedding Its not this (beginning of the end)
Ongoing
Finish
Done
S
R
D
P
F
r
i
S
C

X-Schema for X with bindings
35
Embedding But this (The end of the beginning)
Ready
Done
Start
Process
Finish
resume
Suspend
interrupt
R
D
S
P
F
r
i
S
C

X-Schema for X with bindings
36
Inherent Aspect (Aksionsart)

• Vendler-Dowty-Taylor (VDT) classification
• Events and States
• Events can be
• Punctual or Durative
• Atelic or Telic
• States satisfy the downward entailment property
• If a state holds in some interval, it holds in
  all sub-intervals of that interval.

37
Inherent Aspect

• Much richer than traditional Linguistic
  Characterizations (VDT (durative/atomic,
  telic/atelic))
• Action patterns
• one-shot, repeated, periodic, punctual
• decomposition concurrent, alternatives,
  sequential
• Goal based schema enabling/disabling
• Generic control features
• interruption, suspension, resumption
• Resource usage

38
(No Transcript)
39
Basic Event X-schemas

• State
• Event Transition
• Simple Event
• Simple Action
• Complex Event/Process
• Complex State

40
Aspectual Types
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
Other Transitions in the Controller may be coded

• Lexical items may code interrupts
• Stumble is an interrupt to an ongoing walk
• A combination of grammatical and aktionsart may
  code of the controller phases
• Ready to walk Prospective
• Resuming his run Resumptive
• Has been running Embedded progressive
• About to Finish the painting Embedded
  Completive.
• Canceling the meeting vs. Aborting the meeting.

46
Interaction of Aspect with Tense

• Reichenbachs system uses three pointers
• Speech Time (S)
• Reference Time (R)
• Event Time (E)
• Tense is a partial ordering relation between the
  pointers
• Simple Past E lt R, E lt S
• Perfect E lt R lt S

47
(No Transcript)
48
The Present Tense

• The Present tense is a state selector.
• It therefore selects a rest from an input event,
  resulting in various coercion effects
• Habitual and generic readings of iterated-event
  sentences, e.g., She smokes, Oil floats on water
• Progressive-style readings of event sentences
  in languages other than English, e.g., French Eh
  bien, à present, je me sens mieux. Le morale
  revient. Now Im feeling better. My morale is
  coming back. (Binet, Bidochon 8 42)
• Perfect-style readings of state-phase sentences
  in languages other than English, e.g., Ca fait
  dix minutes quelle nous parle de la moquette!
  Shes been telling us about the carpet for 10
  minutes. (Binet, Bidochon 1017)

49
The Present Triumvirate
JAN RUNS
JAN IS RUNNING
R
S
P
F
r
i
S
C
R
S
P
F
r
i
S
C
JAN HAS BEEN RUNNING
50
Other Present Tense Affordances

• Of course, we can extend through embedding ANY of
  the available states in the CONTROLLER.
• John is starting his run.
• John starts his run (every morning).
• John stops his run after 3 miles. (He never
  achieves his goal of running 5).
• John has been canceling his run.
• John cancels his run (twice a week).
• We have been restarting this Harley for the last
  5 mins.
• The meeting is about to resume.
• My morale is returning (Michaelis 02).
• Question Do (which) languages have constructions
  for these states?

51
Viewpoint Aspect (Perfective/Imperfective)
52
Perfective/Imperfective
Perfective
Imperfective
53
Simulation and Reference Interval
Perfective
Imperfective
54
Levels of Granularity

• Events can be construed at different levels of
  granularity based on various contextual factors.
• In 1991, McEnroe injured his knee while playing
  tennis.
• This morning, I injured my knee while playing
  tennis.
• He is coughing (normal time scale vs. slow-motion
  film time scale).

55
Summary of Aspect Results

• Controller mediates between linguistic markings
  and individual event/verbal x-schemas (Cogsci99,
  Coling2002)
• Captures regular event structure inspired by
  biological control theory
• Flexible specific events may require only a
  subset of controller interaction of underlying
  x-schemas, linguistic markers and hierarchical
  abstraction/ decomposition of controller accounts
  for wide range of aspectual phenomena.
• Important aspectual distinctions, both
  traditional and novel, can be precisely specified
  in terms of the interaction of x-schemas with the
  controller (CogSci97, CogSci98, AAAI99, IJCAI99,
  CogSci04, CogLing2005)
• stative/dynamic, durative/punctual natural in
  x-schemas
• telic processes depletion of resources
• continuous processes consumption of resources
• temporary/effortful states habituals
• dynamic interactions with tense, nominals,
  temporal modifiers
• incorporation of world knowledge, pragmatics
• Ongoing Work Simulation Semantics and
  Tense-Aspect (with Laura Michaelis)

56
Simulation hypothesis

• We understand utterances by mentally simulating
  their content.
• Simulation exploits some of the same neural
  structures activated during performance,
  perception, imagining, memory
• Linguistic structure parameterizes the
  simulation.
• Language gives us enough information to simulate

57
Simulation Semantics

• BASIC ASSUMPTION SAME REPRESENTATION FOR
  PLANNING AND SIMULATIVE INFERENCE
• Evidence for common mechanisms for recognition
  and action (mirror neurons) in the F5 area
  (Rizzolatti et al (1996), Gallese 96, Buccino
  2002, Tettamanti 2004) and from motor imagery
  (Jeannerod 1996)
• IMPLEMENTATION
• x-schemas affect each other by enabling,
  disabling or modifying execution trajectories.
  Whenever the CONTROLLER schema makes a transition
  it may set, get, or modify state leading to
  triggering or modification of other x-schemas.
  State is completely distributed (a graph marking)
  over the network.
• RESULT INTERPRETATION IS IMAGINATIVE SIMULATION!

58
(No Transcript)
59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
A Precise Notion of Contingency Relations
Activation Executing one schema causes the
enabling, start or continued execution of another
schema. Concurrent and sequential
activation. Inhibition Inhibitory links prevent
execution of the inhibited x-schema by activating
an inhibitory arc. The model distinguishes
between concurrent and sequential inhibition,
mutual inhibition and aperiodicity. Modification
The modifying x-schema results in control
transition of the modified xschema. The execution
of the modifying x-schema could result in
the interruption, termination, resumption of the
modified x-schema.
65
General and Domain Knowledge

• Conceptual Knowledge and Inference
• Embodied
• Language and Domain Independent
• Powerful General Inferences
• Ubiquitous in Language
• Domain Specific Frames and Ontologies
• FrameNet, OWL ontologies
• Metaphor links domain specific to general
• E.g., France slipped into recession.

66
Frames

• Frames are conceptual structures that may be
  culture specific
• Words evoke frames
• The word talk evokes the Communication frame
• The word buy (sell, pay) evoke the Commercial
  Transaction (CT) frame.
• The words journey, set out, schedule, reach etc.
  evoke the Journey frame.
• Frames have roles and constraints like schemas.
• CT has roles vendor, goods, money, customer.
• Words bind to frames by specifying binding
  patterns
• Buyer binds to Customer, Vendor binds to Seller.

67
Event Frames

• Event frames have temporal structure that
  comprises of the controller event structure and
  generally have constraints on what precedes them,
  what happens during them, and what state the
  world is in once the event has been completed.

68
Sample Event FrameCommercial Transaction

• Initial stateVendor has Goods, wants
  MoneyCustomer wants Goods, has Money
• TransitionVendor transmits Goods to
  CustomerCustomer transmits Money to Vendor
• Final stateVendor has Money
• Customer has Goods

69
Sample Event FrameCommercial Transaction

• Initial stateVendor has Goods, wants
  MoneyCustomer wants Goods, has Money
• TransitionVendor transmits Goods to
  CustomerCustomer transmits Money to Vendor
• Final stateVendor has Money
• Customer has Goods

(Its a bit more complicated than that.)
70
Partial Wordlist for Commercial Transactions

• Verbs pay, spend, cost, buy, sell, charge
• Nouns cost, price, payment
• Adjectives expensive, cheap

71
Meaning and Syntax

• The various words that evoke this frame introduce
  the elements of the frame in different ways.
• The identities of the buyer, seller, goods and
  money
• Information expressed in sentences containing
  these words occurs in different places in the
  sentence depending on the word.

72
Language understanding analysis simulation
construction WALKED form selff.phon ?
wakt meaning Walk-Action constraints
selfm.time before Context.speech-time
selfm..aspect ? encapsulated
Harry walked into the cafe.
Utterance
Constructions Lexicon
Analysis Process
General Knowledge
Semantic Specification
Belief State
Simulation