Temporal Relations in Visual Semantics of Verbs

1
Temporal Relations inVisual Semantics of Verbs

• Minhua Eunice Ma and Paul Mc Kevitt
• School of Computing and Intelligent Systems
• Faculty of Engineering
• University of Ulster, Magee
• Derry/Londonderry, N. Ireland

2
Background CONFUCIUS (intelligent storytelling
system)
story in natural language
Storywriter /playwright
speech (dialogue)
User /story listener
movie/drama script
CONFUCIUS
3D animation
non-speech audio
tailored menu for script input

• To interpret natural language stories and to
  extract conceptual semantics from natural
  language
• To generate 3D animation and virtual worlds
  automatically from natural language
• To integrate 3D animation with speech and
  non-speech audio for presenting multimodal stories

3
Previous research

• Temporal relations
• Allens interval relations
• Application in story-based interactive systems
• Temporal relations in technical orders domain
  (Badler et al., 1997)

• Related research in NLP
• Sentence level temporal analysis
• Lexical vs. post-lexical temporal relations
• Lexical semantics

4
Allens interval relations
5
NLP in CONFUCIUS
6
Verb entailments
Verb entailment fixed truth relation between
verbs with entailment given by part of lexical
meaning, i.e. one verb entails another The
implication logic relationship if p then q (p?q)
7
Troponym

• Elaborates manner of base verb (Fellbaum, 1998)
• Examples trot-walk (fast), gulp-eat
  (quickly)

EVENT

go (move)
cause
other action predicates

run
walk
climb
jump
manner-of-motion verbs

limp
stride
swagger
trot
8
Temporal relations in verb entailment
p,m,o,s,f-1, may also represent temporal
relation between pair of cognate verbs and state
of corresponding adjectives e.g. shorten-short,
beautify-beautiful, clarify-clear
9
Representing procedural events

• Arguments of EVENT
• EVENT
• agent
• theme
• space/time
• manner
• instrument
• precondition
• subactivities
• result
• Relationship between definiendum verb and its
  subactivities
• act()-
• subact1(),
• subacti(), .
• subacti R act, i?N, R?d,s,f,?

10
Comparison with Badlers temporal constraints

• Badlers temporal constraints
• (technical orders domain)
• Sequential
• Parallel
• Jointly parallel
• Independently parallel
• While parallel

• Interval relations
• p,m
• s,s-1,?
• (act1 s,s-1,? act2) p,m act3
• f,f-1,?
• act_domt s-1,f-1,? act_indomt

• compositional (e.g. jointly parallel) all 5
  constraints are disjunctions of several interval
  relations
• consider other factors such as dominancy of
  action (e.g. while parallel)
• domain-specific

11
Comparison with Badlers temporal constraints

• Badlers temporal constraints
• (technical orders domain)
• Sequential
• Parallel
• Jointly parallel
• Independently parallel
• While parallel

• Interval relations
• p,m
• s,s-1,?
• (act1 s,s-1,? act2) p,m act3
• f,f-1,?
• act_domt s-1,f-1,? act_indomt

• compositional (e.g. jointly parallel) all 5
  constraints are disjunctions of several interval
  relations
• consider other factors such as dominancy of
  action (e.g. while parallel)
• domain-specific

12
Comparison with Badlers temporal constraints

• Badlers temporal constraints
• (technical orders domain)
• Sequential
• Parallel
• Jointly parallel
• Independently parallel
• While parallel

• Interval relations
• p,m
• s,s-1,?
• (act1 s,s-1,? act2) p,m act3
• f,f-1,?
• act_domt s-1,f-1,? act_indomt

• compositional (e.g. jointly parallel) all 5
  constraints are disjunctions of several interval
  relations
• consider other factors such as dominancy of
  action (e.g. while parallel)
• domain-specific

13
Comparison with Badlers temporal constraints

• Badlers temporal constraints
• (technical orders domain)
• Sequential
• Parallel
• Jointly parallel
• Independently parallel
• While parallel

• Interval relations
• p,m
• s,s-1,?
• (act1 s,s-1,? act2) p,m act3
• f,f-1,?
• act_domt s-1,f-1,? act_indomt

• compositional (e.g. jointly parallel) all 5
  constraints are disjunctions of several interval
  relations
• consider other factors such as dominancy of
  action (e.g. while parallel)
• domain-specific

14
Comparison with Badlers temporal constraints

• Badlers temporal constraints
• (technical orders domain)
• Sequential
• Parallel
• Jointly parallel
• Independently parallel
• While parallel

• Interval relations
• p,m
• s,s-1,?
• (act1 s,s-1,? act2) p,m act3
• f,f-1,?
• act_domt s-1,f-1,? act_indomt

• compositional (e.g. jointly parallel) all 5
  constraints are disjunctions of several interval
  relations
• consider other factors such as dominancy of
  action (e.g. while parallel)
• domain-specific

15
Achievement vs. accomplishment events

• Achievement events (Vendler, 1967) e.g. find,
  arrive, die
• punctual events occuring at single moment
• definite time instants
• never hold over intervals
• Why use interval relations instead of point-based
  relations?
• Pragmatic reasons (Verkuyl, 1993)
• Ontological reasons (Pinon, 1997)
• Practical reason for language visualisation
• achievement events depend on existence of context
  
• context visual definitions ? intervals

find()- search(), eyesFixedOn().
arrive()- go(), stopAtDestination().
16
Temporal relations of lexical causatives

• Visual definitions of causative verbs (e.g.
  kill) must subsume result states (stative
  verbs) (e.g. die)
• Represent distinction between
• launching causatives causation of inception of
  motion
• entraining causatives continuous causation of
  motion

disjunction set of interval relations between
cause and effect adequate to define difference
s,p,m,o (launching) ,f-1 (entraining)
17
Lexical and post-lexical repetition

• Post-lexical level repetition
• e.g. Roses come into bloom once a year.
• I visit the school every day.
• or marked by again", "continues to", "a second
  time
• Lexical level repetition
• Represent periodical repetition of subactivities
• walk()- step()R.
• hammer()- hit()R.
• Morphological prefix "re-"

18
Categories of action verb

• 2.2.1. Action verbs
• 2.2.1.1. Movement or partial movement
• 2.2.1.1.1. Biped kinematics, e.g. go, walk,
  jump, swim, climb
• 2.2.1.1.2. Face expressions, e.g. laugh, angry
• 2.2.1.1.3. Lip movement, e.g. speak, say, sing,
  tell
• 2.2.1.2. Lexical causatives
• 2.2.1.2.1. Concerning single object, e.g. push,
  kick, bring, open
• 2.2.1.2.2. Concerning multiple objects
• 2.2.1.2.2.1. Bitransitive verbs, e.g. give,
  sell, show
• 2.2.1.2.2.2. Transitive verbs with object
  implicit instrument/goal/theme,
• e.g. cut, write, butter, pocket
• 2.2.1.3. Verbs without distinct visualization
  when out of context
• 2.2.1.3.1. trying verbs try, attempt, succeed,
  manage
• 2.2.1.3.2. helping verbs help, assist
• 2.2.1.3.3. letting verbs allow, let, permit
• 2.2.1.3.4. create/destroy verbs build, create,
  assemble, construct, break, destroy
• 2.2.1.3.5. verbs whose visualization depends on
  their objects,
• e.g. play (harmonica/football), make (the
  bed/trouble/a phone call), fix (a drink/a lock)
• 2.2.1.4. High level behaviours (routine events)

19
Lexical Visual Semantic Representation

• Lexical Visual Semantic Representation (LVSR)
• necessary semantic representation between
• 3D model and language syntax
• LVSR based on Jackendoffs LCS adapted to task of
  language visualization (enhancement with Schanks
  scripts)
• Interval relations represent temporal
  relationship between subactivities of complex
  actions in LVSR
• e.g.
• The waiter approached me Can I help you?
  Sir.
• 3D animation
• John walked towards the house.
• 3D animation
• Nancy ran across the field.
• 3D animation

20
Conclusion

• Temporal relation is a crucial issue in
  modelling action verbs, their procedures,
  contexts, presupposed and result states
• Temporal relation within verb semantics (lexical
  level)
• Semantic representation of verbs with temporal
  information based on Allens interval logic

21
Future work

• Quantitative factor
• Action composition for simultaneous activities
• Verbs concerning multiple characters
  synchronization coordination
• Character can start a task when another signals
  pre-conditions are ready
• Two or more characters cooperate in shared task