TERQAS: Time and Event Recognition for Question Answering Systems

1
TERQASTime and Event Recognition for Question
Answering Systems

• TERQAS Group
• Final Review
• ARDA Workshop
• NRRC/MITRE
• July 22, 2002

2
TERQAS2002 Workshop Schedule

• January 30-31 Kick-off Meeting Setting Agenda
• March 11-15 Corpus Selection, Query Studies,
  TimeML
• April 22-26 TimeML Specification, Corpus Work
• May 8-15 Annotation Fest
• June 10-20 Algorithm Specification, Annotation
• July 15-22 Wrap-up and Evaluation
• Aug-Sept Prepare Final Report

time2002.org
3
Relevance to Question Answering Systems

• Is Gates currently CEO of Microsoft?
• Were there any meetings between the terrorist
  hijackers and Iraq before the WTC event?
• Did the Enron merger with Dynegy take place?
• How long did the hostage situation in Beirut last?

• When did the war between Iran and Iraq end?
• When did John Sununu travel to a fundraiser for
  John Ashcroft?
• How many Tutsis were killed by Hutus in Rwanda
  in 1994?
• Who was Secretary of Defense during the Gulf
  War?
• What was the largest U.S. military operation
  since Vietnam?
• When did the astronauts return from the space
  station?

4
Workshop Goals

• TimeML Define and Design a Metadata Standard for
  Markup of events, their temporal anchoring, and
  how they are related to each other in News
  articles.
• TIMEBANK Given the specification of TimeML,
  create a gold standard corpus of 300 articles
  marked up for temporal expressions, events, and
  basic temporal relations.

5
Working Groups

• TimeML Definition and Specification
• Algorithm Review and Development
• Article Corpus Collection Development
• Query Corpus Development and Classification
• TIMEBANK Annotation
• TimeML and Algorithm Evaluation

6
TERQAS Participants

• James Pustejovsky, PI
• Rob Gaizauskas
• Graham Katz
• Bob Ingria
• José Castaño
• Inderjeet Mani
• Antonio Sanfilippo
• Dragomir Radev
• Patrick Hanks
• Marc Verhagen
• Beth Sundheim
• Andrea Setzer

• Jerry Hobbs
• Bran Boguraev
• Andy Latto
• John Frank
• Lisa Ferro
• Marcia Lazo
• Roser Saurí
• Anna Rumshisky
• David Day
• Luc Belanger
• Harry Wu
• Andrew See

Supported by
7
Presentation Outline

• TimeML 1.0 Specification
• T3PO Algorithm Development
• Tool Development Effort
• TIMEBANK Annotation Status
• Query Corpus Development and Classification
• TIMEBANK Annotation
• Future Projects

8
TimeML 1.0

• Adopts the core of Setzers annotation framework
  (Sheffield Temporal Annotation Guidelines, STAG)
• Remains compliant (as much as possible) with
  TIDES TIMEX2 annotation.
• Introduces a TLINK tag an object that links
  events/times to events/times.
• Introduces an ALINK tag an object that
  associates aspectual phases to events.
• Introduces an SLINK tag an object that
  subordinates events within modality, negation, or
  another event.
• Enrich temporal relations adds i-after,
  i-before, and aspectual relations.
• Introduces event identity.
• Introduces Temporal functions for doing temporal
  math without evaluation.
• Introduces STATE as a possible event class.

9
How TimeML Differs from Previous Markups

• Extends TIMEX2 annotation
• Temporal Functions three years ago
• Anchors to events and other temporal expressions
  
• Identifies signals determining interpretation of
  temporal expressions
• Temporal Prepositions for, during, on, at
• Temporal Connectives before, after, while.
• Identifies event expressions
• tensed verbs has left, was captured, will
  resign
• stative adjectives sunken, stalled, on board
• event nominals merger, Military Operation, Gulf
  War
• Creates dependencies between events and times
• Anchoring John left on Monday.
• Orderings The party happened after midnight.
• Embedding John said Mary left.

10

• Annotation in an Extension of STAG
• FAMILIES SUE OVER AREOFLOT CRASH DEATHS
• The Russian airline Aeroflot has been
• ltEVENT eid1 relatedToTime1 timeRelTypeBEFORE
  tensePRESENT aspectPERFECTIVE classOCCURRENCEgt
• hit
• lt/EVENTgt
• with a writ for loss and damages,
• ltEVENT eid2 tenseNONE aspectPERFECTIVE
  relatedToEvent1 eventRelTypeBEFORE
  classOCCURRENCEgt
• filed
• lt/EVENTgt
• in Hong Kong by the families of seven passengers
• ltEVENT eid3 tenseNONE aspectPERFECTIVE
  relatedToEvent2 eventRelTypeBEFORE
  classOCCURRENCE relatedToEvent24
  eventRel2TypeIS_INCLUDED signal21gt
• killed
• lt/EVENTgt
• ltSIGNAL sid1gt
• In lt/SIGNALgt
• an air
• ltEVENT eid4 classOCCURRENCEgt

11

• STAG Annotation, cont.
• All 75 people
• on board
• the Aeroflot Airbus
• ltEVENT eid5 tensePAST aspectPERFECTIVE
  relatedToEvent6 eventRelTypeIAFTER signal2gt
• died
• lt/EVENTgt
• ltSIGNAL sid2gt
• when lt/SIGNALgt
• it
• ltEVENT eid6 tensePAST aspectPERFECTIVE
  relatedToTime2 timeRelTypeIS_INCLUDED
  relatedToEvent4 eventRelTypeIDgt
• ploughed
• lt/EVENTgt
• into a Siberian mountain
• ltSIGNAL sid3gt
• in
• lt/SIGNALgt
• ltTIMEX tid2 typeDATE calDate031994gt

12
Drawbacks of Event-Internal Relations in STAG

• Triple attribute structure in EVENT
• (signalID relatedToEvent eventRelType)
• (signalID relatedToTime timeRelType)
• Same attribute structure appears in TIMEX
• (eid signalID relType)
• These three attributes are logically linked,
  allowing eventRelType, eventRelType,and
  eventRelType to be collapsed into single
  attribute.

13
EVENT
attributes eid class tense aspect eid
ID eid EventID EventID eltintegergt class
'OCCURRENCE' 'PERCEPTION' 'REPORTING'
'ASPECTUAL' 'STATE' 'I_STATE'
'I_ACTION' 'MODAL' tense 'PAST'
'PRESENT' 'FUTURE' 'NONE' aspect
'PROGRESSIVE' 'PERFECTIVE' 'PERFECTIVE_PROGRES
SIVE' 'NONE'
14
TimeML Event Classes

• Occurrence
• die, crash, build, merge, sell, take advantage
  of, ..
• State
• Be on board, kidnapped, recovering, love, ..
• Reporting
• Say, report, announce,
• I-Action
• Attempt, try,promise, offer
• I-State
• Believe, intend, want,
• Aspectual
• begin, start, finish, stop, continue.
• Perception
• See, hear, watch, feel.

15
The young industry's rapid growth also is
attracting regulators eager to police its many
facets. The young industry's rapid ltEVENT
eid"e1" class"OCCURRENCE"gt growth
lt/EVENTgt also is ltEVENT eid"e2"
class"OCCURRENCE"gt attracting
lt/EVENTgt regulators ltEVENT eid"e4"
class"I_STATE"gt eager lt/EVENTgt to ltEVENT
eid"e5" class"OCCURRENCE"gt police
lt/EVENTgt its many facets.
16
Israel will ask the United States to delay a
military strike against Iraq until the Jewish
state is fully prepared for a possible Iraqi
attack. Israel will ltEVENT eid"e1"
class"I_ACTION"gt ask lt/EVENTgt the United States
to ltEVENT eid"e2" class"I_ACTION"gt
delay lt/EVENTgt a military ltEVENT eid"e3"
class"OCCURRENCE"gt strike lt/EVENTgt against
Iraq until the Jewish state is fully ltEVENT
eid"e4" class"I_STATE"gt prepared lt/EVENTgt for a
possible Iraqi ltEVENT eid"e5" class"OCCURRENCE"gt
attack lt/EVENTgt
17

TIMEX2 Tag Attributes
18
Temporal Functions

• Temporal expressions where the calendar date is
  not referred to directly, but via an expression
  that acts as a temporal function over a TIMEX3
  expression.
• Examples
• last week
• last Thursday
• the week before last
• next week

19
Pre-theoretic TreatmentDCTDocCreationTime

• last week (predecessor (week DCT))
• That is, we start with a temporal anchor, in
  this case, the DCT,
• coerce it to a week, than find the week
  preceding it.
• last Thursday (thursday (predecessor (week
  DCT))
• Similar to the preceding expression, except that
  we pick out the day named 'thursday' in the
  predecessor week.
• the week before last (predecessor (predecessor
  (week DCT)))
• Also similar to the first expression, except
  that we go back two
• weeks.
• next week (successor (week DCT))
• The dual of the first expression we start with
  the same coercion, but go forward instead of
  back.

20
TIMEX2 Annotation

• Sen. Alton Waldon, who served briefly in Congress
  
• ltTIMEX2 VAL"199 MOD"BEFORE"gt
• more than a decade agolt/TIMEX2gt,
• is
• ltTIMEX2 VAL"PRESENT_REF"gt
• Now
• lt/TIMEX2gt
• retired.

21
TimeML Treatment of Temporal Functions

• Sen. Alton Waldon, who served briefly in Congress
  more than a decade ago, is now retired.
• Sen. Alton Waldon, who
• ltEVENT eid"e1" class"OCCURRENCE" tense"PAST"
  aspect"NONE"gt
• served
• lt/EVENTgt
• ltMAKEINSTANCE eiid"ei1" eventID"e1"/gt
• briefly in Congress
• ltTIMEX3 tid"t1" typeDATE" value199"
  modBEFORE temporalfunctionTRUEgt
• more than a decade ago
• lt/TIMEX3gt
• is
• ltTIMEX3 tid"t2" type"DATE" value"PRESENT_REF"gt
  
• now
• lt/TIMEX3gt
• ltEVENT eid"e2" class"STATE" tense"NONE"
  aspect"NONE"gt
• retired
• lt/EVENTgt
• ltMAKEINSTANCE eiid"ei2" eventID"e2"/gt.

22
Temporal Functions Alternative Analysis

• Sen. Alton Waldon, who
• ltEVENT eid"e1" class"OCCURRENCE" tense"PAST"
  aspect"NONE"gt
• served
• lt/EVENTgt
• ltMAKEINSTANCE eiid"ei1" eventID"e1"/gt
• briefly in Congress
• ltTIMEX3 tid"t1" type"DURATION" value"P1E"
  mod"MORE_THAN"gt
• more than a decade
• lt/TIMEX3gt
• ltSIGNAL sid"s1"gt
• ago
• lt/SIGNALgt,
• is
• ltTIMEX3 tid"t2" type"DATE" value"PRESENT_REF"gt
  
• now
• lt/TIMEX3gt
• ltEVENT eid"e2" class"STATE" tense"NONE"
  aspect"NONE"gt
• retired
• lt/EVENTgt

23
TLINK

• TLINK or Temporal Link represents the temporal
  relationship holding between events or between an
  event and a time, and establishes a link between
  the involved entities, making explicit if they
  are
• Simultaneous (happening at the same time)
• Identical (referring to the same event)
• John drove to Boston. During his drive he ate a
  donut.
• 3. One before the other
• The police looked into the slayings of 14
  women. In six of the cases suspects have already
  been arrested.
• 4. One after the other
• 5. One immediately before the other
• All passengers died when the plane crashed into
  the mountain
• 6. One immediately after than the other
• 7. One including the other
• John arrived in Boston last Thursday.
• 8. One being included in the other
• 9. One holding during the duration of the other
• 10. One being the beginning of the other
• John was in the gym between 600 p.m. and 700
  p.m.
• 11. One being begun by the other
• 12. One being the ending of the other
• John was in the gym between 600 p.m. and 700
  p.m..

24
SLINK
SLINK or Subordination Link is used for contexts
introducing relations between two events, or an
event and a signal, of the following sort 1.
Modal Relation introduced mostly by modal verbs
(should, could, would, etc.) and events that
introduce a reference to a possible world
--mainly I_STATEs John should have bought some
wine. Mary wanted John to buy some wine. 2.
Factive Certain verbs introduce an entailment
(or presupposition) of the argument's veracity.
They include forget in the tensed complement,
regret, manage John forgot that he was in
Boston last year. Mary regrets that she didn't
marry John. John managed to leave the party 3.
Counterfactive The event introduces a
presupposition about the non-veracity of its
argument forget (to), unable to (in past tense),
prevent, cancel, avoid, decline, etc. John
forgot to buy some wine. Mary was unable to
marry John. John prevented the divorce. 4.
Evidential Evidential relations are introduced
by REPORTING or PERCEPTION John said he bought
some wine. Mary saw John carrying only beer.
5. Negative evidential Introduced by REPORTING
(and PERCEPTION?) events conveying negative
polarity John denied he bought only beer. 6.
Negative Introduced only by negative particles
(not, nor, neither, etc.), which will be marked
as SIGNALs, with respect to the events they are
modifying John didn't forgot to buy some wine.
John did not wanted to marry Mary.
25
ALINK
ALINK or Aspectual Link represent the
relationship between an aspectual event, which
will be annotated as a SIGNAL (section 2.3), and
its argument event. Examples of the possible
aspectual relations we will encode are 1.
Initiation John started to read 2.
Culmination John finished assembling the
table. 3. Termination John stopped talking. 4.
Continuation John kept talking.
26
Causation 1
(1) The rains caused the flooding. The ltEVENT
eid"e1" class"OCCURRENCE" tense"NONE"
aspect"NONE"gt rains lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt ltEVENT eid"e2"
class"OCCURRENCE" tense"PAST"
aspect"NONE"gt caused lt/EVENTgt ltMAKEINSTANCE
eiid"ei2" eventID"e2"/gt the ltEVENT eid"e3"
class"OCCURRENCE" tense"NONE"
aspect"NONE"gt flooding lt/EVENTgt ltMAKEINSTANCE
eiid"ei3" eventID"e3"/gt ltTLINK
eventInstanceID"ei1" relatedToEvent"ei2"
relType"IDENTITY"/gt ltTLINK eventInstanceID"ei2"
relatedToEvent"ei3" relType"BEFORE"/gt
27
Causation 2
(2') Kissinger secured the peace at great
cost. Kissinger ltEVENT eid"e1"
class"OCCURRENCE" tense"PAST"
aspect"NONE"gt secured lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt the ltEVENT eid"e2"
class"OCCURRENCE" tense"NONE"
aspect"NONE"gt peace lt/EVENTgt ltMAKEINSTANCE
eiid"ei2" eventID"e2"/gt at great cost. ltTLINK
eventInstanceID"ei1" relatedToEvent"ei2"
relType"BEFORE"/gt
28
Causation 3
(3) He kicked the ball, and it rose into the
air. He ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt kicked lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"/gt the ball, and
it ltEVENT eid"e2" class"OCCURRENCE"
tense"NONE" aspect"NONE"gt rose lt/EVENTgt ltMAKEINS
TANCE eiid"ei2" eventID"e2"/gt into the
air. ltTLINK eventInstanceID"ei1"
relatedToEvent"ei2" relType"BEFORE"/gt
29
TLINK 1
(4) John taught 20 minutes every Monday.
John ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt taught lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1" signalID"s1"
cardinality"EVERY"/gt ltTIMEX3 tid"t1"
type"DURATION" value"PT20M"gt 20
minutes lt/TIMEX3gt ltSIGNAL sid"s1"gt every lt/SIGNAL
gt ltTIMEX3 tid"t2" type"DATE" value"XXXX-WXX-1"gt
Monday lt/TIMEX3gt ltTLINK eventInstanceID"ei1"
relatedToTime"t1" relType"HOLDS"/gt ltTLINK
eventInstanceID"ei1" relatedToTime"t2"
relType"IS_INCLUDED"/gt
30
TLINK 2
(6) John taught twice on Monday but only once on
Tuesday John ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt taught lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"
signalID"s1"/gt ltMAKEINSTANCE eiid"ei2"
eventID"e1" signalID"s1"/gt ltMAKEINSTANCE
eiid"ei3" eventID"e1" signalID"s2"/gt ltSIGNAL
sid"s1"gt twice lt/SIGNALgt ltSIGNAL
sid"s3"gt on lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" temporalFunction"true"
value"XXXX-WXX-1"gt Monday lt/TIMEX3gt But
only ltSIGNAL sid"s2"gt once lt/SIGNALgt ltSIGNAL
sid"s4"gt on lt/SIGNALgt ltTIMEX3 tid"t2"
type"DATE" temporalFunction"true"
value"XXXX-WXX-2"gt Tuesday lt/TIMEX3gt ltTLINK
eventInstanceID"ei1" signalID"s3"
relatedToTime"t1" relType"IS_INCLUDED"/gt ltTLINK
eventInstanceID"ei2" signalID"s3"
relatedToTime"t1" relType"IS_INCLUDED"/gt ltTLINK
eventInstanceID"ei3" signalID"s4"
relatedToTime"t2" relType"IS_INCLUDED"/gt
31
TLINK 3
(7) John taught 5 minutes after the explosion.
ltEVENT eid"e1" class"OCCURRENCE" tense"PAST"
aspect"NONE"gt taught lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt ltTIMEX3 tid"t1"
type"DURATION" value"PT5M"gt 5
minutes lt/TIMEX3gt ltSIGNAL sid"s1"gt after lt/SIGNAL
gt the ltEVENT eid"e2" class"OCCURRENCE"
tense"NONE" aspect"NONE"gt explosion lt/EVENTgt ltMA
KEINSTANCE eiid"ei2" eventID"e2"/gt ltTLINK
eventInstanceID"ei1" signalID"s1"
relatedToEvent"ei2" relType"AFTER"
magnitude"t1"/gt
32
TLINK 4
(8) John taught from 1992 through 1995.
John ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt taught lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"/gt ltSIGNAL
sid"s1"gt from lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" value"1992"gt 1992 lt/TIMEX3gt ltSIGNAL
sid"s2"gt through lt/SIGNALgt ltTIMEX3 tid"t2"
type"DATE" value"1995"gt 1995 lt/TIMEX3gt ltTLINK
eventInstanceID"ei1" signalID"s1"
relatedToTime"t1" relType"BEGUN_BY"/gt ltTLINK
eventInstanceID"ei1" signalID"s2"
relatedToTime"t2" relType"ENDED_BY"/gt
33
TLINK 5
(9) John taught from September to December last
year. John ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt taught lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"/gt ltSIGNAL
sid"s1"gt from lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" temporalFunction"true"
value"XXXX-09"gt September lt/TIMEX3gt ltSIGNAL
sid"s2"gt To lt/SIGNALgt ltTIMEX3 tid"t2"
type"DATE" temporalFunction"true"
value"XXXX-12"gt December lt/TIMEX3gt ltTIMEX3
tid"t3" type"DATE" temporalFunction"true"
value"XXXX" anchorTimeID"t4"gt last
year lt/TIMEX3gt ltTIMEX3 tid"t4" type"DATE"
functionInDocument"CREATION_TIME"
value"1996-03-27"gt 03-27-96 lt/TIMEX3gt ltTLINK
eventInstanceID"ei1" signalID"s1"
relatedToTime"t1" relType"BEGUN_BY"/gt ltTLINK
eventInstanceID"ei1" signalID"s2"
relatedToTime"t2" relType"ENDED_BY"/gt
34
SLINK 1
(12) John taught on Monday but not on Tuesday
John ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt taught lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"
signalID"s3"/gt ltMAKEINSTANCE eiid"ei2"
eventID"e1" signalID"s4"/gt ltSIGNAL
sid"s3"gt on lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" temporalFunction"true"
value"XXXX-WXX-1"gt Monday lt/TIMEX3gt but ltSIGNAL
sid"s1"gt not lt/SIGNALgt ltSIGNAL
sid"s4"gt on lt/SIGNALgt ltTIMEX3 tid"t2"
type"DATE" temporalFunction"true"
value"XXXX-WXX-2"gt Tuesday lt/TIMEX3gt ltTLINK
eventInstanceID"ei1" relatedToTime"t1"
signalID"s3" relType"IS_INCLUDED"/gt ltTLINK
eventInstanceID"ei2" relatedToTime"t2"
signalID"s4" relType"IS_INCLUDED"/gt ltSLINK
subordinatedEventInstance"ei2" signalID"s1"
relType"NEGATIVE"/gt
35
SLINK 2
(13) If Graham leaves today, he will not hear
Sabine. ltSIGNAL sid"s1"gt if lt/SIGNALgt Graham ltEV
ENT eid"e1" class"OCCURRENCE" tense"PRESENT"
aspect"NONE"gt leaves lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt ltSLINK
subordinatedEvent"e1" signalID"s1"
relType"MODAL"/gt ltTIMEX3 tid"t1" type"DATE"
temporalFunction"true" value"XXXX-XX-XX"gt today
lt/TIMEX3gt he ltEVENT eid"e3" class"MODAL"
tense"NONE" aspect"NONE"gt will lt/EVENTgt ltMAKEINS
TANCE eiid"ei3" eventID"e3"/gt ltSIGNAL
sid"s2"gt not lt/SIGNALgt ltEVENT eid"e2"
class"OCCURRENCE" tense"FUTURE"
aspect"NONE"gt hear lt/EVENTgt ltMAKEINSTANCE
eiid"ei2" eventID"e2"/gt Sabine. ltSLINK
eventInstanceID"ei3" subordinatedEvent"e2"
relType"MODAL"/gt ltTLINK eventInstanceID"ei1"
relatedToEvent"ei2" relType"BEFORE"/gt ltSLINK
subordinatedEvent"e2" signalID"s1"
relType"NEGATIVE"/gt
36
SLINK 3
(14) Bill denied that John taught on Monday.
Bill ltEVENT eid"e1" class"OCCURRENCE"
tense"PAST" aspect"NONE"gt denied lt/EVENTgt ltMAKEI
NSTANCE eiid"ei1" eventID"e1"/gt that ltSLINK
eventInstanceID"ei1" subordinatedEvent"e2"
relType"NEG_EVIDENTIAL"/gt John ltEVENT eid"e2"
class"OCCURRENCE" tense"PAST"
aspect"NONE"gt taught lt/EVENTgt ltMAKEINSTANCE
eiid"ei2" eventID"e2"/gt ltSIGNAL
sid"s1"gt on lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" temporalFunction"true"
value"XXXX-WXX-1"gt Monday lt/TIMEX3gt ltTLINK
eventInstanceID"ei2" relatedToTime"t1"
relType"IS_INCLUDED"/gt
37
SLINK 4
(15) Bill wants to teach on Monday. Bill ltEVENT
eid"e1" class"I_STATE" tense"PRESENT"
aspect"NONE"gt wants lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt ltSLINK
eventInstanceID"ei1" signalID"s1"
subordinatedEvent"e2" relType"MODAL"/gt ltSIGNAL
sid"s1"gt to lt/SIGNALgt ltEVENT eid"e2"
class"OCCURRENCE" tense"NONE"
aspect"NONE"gt teach lt/EVENTgt ltMAKEINSTANCE
eiid"ei2" eventID"e2"/gt ltSIGNAL
sid"s2"gt on lt/SIGNALgt ltTIMEX3 tid"t1"
type"DATE" temporalFunction"true"
value"XXXX-WXX-1"gt Monday lt/TIMEX3gt ltTLINK
eventInstanceID"ei2" relatedToTime"t1"
relType"IS_INCLUDED"/gt
38
ALINK 1
(17) The boat began to sink. The boat ltEVENT
eid"e1" class"ASPECTUAL" tense"PAST"
aspect"NONE"gt began lt/EVENTgt ltMAKEINSTANCE
eiid"ei1" eventID"e1"/gt ltSIGNAL
sid"s1"gt to lt/SIGNALgt ltEVENT eid"e2"
class"OCCURRENCE" tense"NONE" aspect
"NONE"gt sink lt/EVENTgt ltALINK eventInstanceID"ei1"
signalID"s1" relatedToEvent"e2"
relType"INITIATES"/gt
39
ALINK 2
(18) The search party stopped looking for the
survivors. The search party ltEVENT eid"e1"
class"ASPECTUAL" tense"PAST" aspect"NONE"gt stop
ped lt/EVENTgt ltMAKEINSTANCE eiid"ei1"
eventID"e1"/gt ltEVENT eid"e2" class"OCCURRENCE"
tense"NONE" aspect"PROGRESSIVE"gt looking lt/EVENT
gt ltALINK eventInstanceID"ei1" relatedToEvent"e2"
relType"TERMINATES"/gt for the survivors
40
time2002.org

• DTD created
• TimeML.dtd
• Schema created
• TimeML.xsd

41
Confidence Measures

• attributes tagType tagID attributeName
  confidenceValue
• tagType CDATA
• tagID IDREF
• attributeName CDATA
• confidenceValue CDATA
• confidenceValue 0 lt x lt 1

42
Use of Confidence Measure

• The TWA flight
• ltEVENT eid"e1" class"OCCURRENCE" tense"PAST"
  aspect"NONE"gt
• crashlanded
• lt/EVENTgt
• ltMAKEINSTANCE eiid"ei1" eventID"e1"/gt
• ltTLINK eventInstanceID"ei1" signalID"s1"
  relatedToTime"t2" relType"BEFORE"
  magnitude"t1"/gt
• on Easter Island
• ltTIMEX3 tid"t1" type"DURATION" value"P2W"gt
• two weeks
• lt/TIMEX3gt
• ltSIGNAL sid"s1"gt
• ago
• lt/SIGNALgt.
• ...
• ltTIMEX3 tid"t2" type"DATE" functionInDocument"C
  REATION_TIME" value"1999-12-20"gt
• 12-20-1999
• lt/TIMEX3gt

43
Domains and Data Sets

• Document Collection (300)
• ACE
• DUC
• PropBank (WSJ)
• Query Corpus Collection
• Excite query logs
• MITRE Corpus
• TREC8/9/10
• Queries from TIMEBANK

44
Corpus Analytics

• Concordanced and indexed all training data
• DUC subset
• ACE subset
• WSJ subset
• Concordancing and indexing reference data
• BNC
• Brown Corpus
• WSJ Corpus

45
Graphical Annotation Tools

• TimeML-Alembic
• Extensions to MITREs Alembic Workbench
• Semi-Graphical Annotation Tool
• Create links by ordering events and TIMEX3s

46
Text Segmented Closure

• System-prompted queries (a la Setzer)
• Completes temporal ordering markup in a text
• Performed on document segments
• Decreases the number of queries required to
  provide closure
• Enrichments to Closure
• Persistence of states
• Negative events

47
Goals of Text Segmented Closure

• Too many temporal relations in a large document.
• The number of temporal relations is quadratic to
  the number of objects that are being linked
  temporally. An annotator may be prompted hundreds
  of times, especially for large documents where a
  lot of the relations are "unknown".
• Some temporal relations are not interesting.
• There does not seem to be a need to relate all
  time expressions to each other.

48
Architecture for TSC

• 1. Perform initial closure on all links added by
  the annotator.
• 2. Alert the user to potential identity chains.
  This is the only occasion where a user may be
  asked to specify a non-local relation.
• 3. Create a sliding window of three sentences.
  Initially, the window will consist of sentences
  one through three. The sliding window implements
  the local context. The size of the sliding window
  can be parameterize.
• 4. Prompt the user to specify a relation type for
  two time objects that are not yet linked within
  the local context. If no temporal relation
  exists, the annotator may specify "unknown".
• 5. After each added relation, recompute the
  closure using the new fact. Do this till all time
  objects within the local context are related.
• 6. If all objects in the local context are
  related, move the window up one sentence. For
  example, if the previous local context was made
  up of sentences 3-5, then the next local context
  for the closure algorithm is sentences 4-6. Start
  prompting the user for the new context.

49
Temporal Axioms

• The axioms work with a normalized set of temporal
  relations (no axiom is needed for the relType
  "unknown")
• PRE before, after, ibefore, iafter
• INC includes, is_included
• SIM simultaneous
• IDT identity
• For PRE, normalization works as follows
• Linkltx,y,beforegt gt Linkltx,y,PREgt
• Linkltx,y,ibeforegt gt Linkltx,y,PREgt
• Linkltx,y,aftergt gt Linklty,x,PREgt
• Linkltx,y,iaftergt gt Linklty,x,PREgt

50
Precedence

• PRE1 x PRE y y PRE z gt x PRE z
• ----x---- ----y---- ----z----
• PRE2 x PRE y y SIM z gt x PRE z
• PRE3 x PRE y y IDT z gt x PRE z
• ----x---- ----y----
• ----z----
• PRE4 x PRE y x SIM z gt z PRE y
• PRE5 x PRE y x IDT z gt z PRE y
• ----x---- ----y----
• ----z----
• PRE6 x PRE y x INC z gt z PRE y

51
Inclusion

• INC1 x INC y y INC z gt x INC z
• ------x------
• ----y----
• --z--
• INC2 x INC y y SIM z gt x INC z
• INC3 x INC y y IDT z gt x INC z
• ----x----
• --y--
• --z--
• INC4 x INC y z SIM x gt z INC y
• INC5 x INC y z IDT x gt z INC y
• ----x----
• --y--
• ----z----

52
Identity and Simultaneity

• SIM1 x SIM y y SIM z gt x SIM z
• SIM2 x SIM y y IDT z gt x SIM z
• IDT1 x IDT y y IDT z gt x IDT z
• ----x----
• ----y----
• ----z----

53
TIMEX3 Parser Objects (T3PO)

• Extends TIDES TIMEX2 annotation
• Broader Coverage of temporal expressions
• Larger lexicon of temporal triggers
• Delays Computation of Temporal Math
• Annotation with Temporal Functions
• Import Hobbs Semantic Web Temporal System
• Distinct Cascaded Processes
• TIMEX3 and signal recognizer
• Event Predicate recognizer
• LINK creation transducer.

54
Algorithm Overview

• Preprocessing POS, Shallow Parsing
• Three Finite State modules
• Temporal Expressions
• Events
• Links
• Discourse Information

Slide 1
55
Temporal Expressions

• Extension to Timex2
• Coverage
• Absolute ISO Values
• Signals
• Functional Representation
• Anchor Resolution
• Suite of Temporal Functions

56
Event Recognition

• In Verbal uses VG chunks
• Encodes Tense and Aspect information
• Nominal Events using
• Morphological information
• POS ambiguity
• Signals
• Semantic Information

57
Link Recognition

• Event -Timex Links
• Use of heuristics.
• Extra-sentential (Event-DCT Links)
• Event-Event Links
• Intrasentential
• SLINKS (evidential)
• SLINKS (infinitivals)
• Extrasentential

58
Discourse Information

• Reference Resolution
• Anchor Resolution
• Tense Sequence and Discourse Structure.

59
Current Development Status
60
Preliminary Tests Estimation(6 documents with
human annotated version)
61
Next Steps

• Complete Timex
• Complete Event Recognition
• Develop Signal Recognition
• Develop Event Class Recognition
• Reference/Anchor Time Recognition
• Evaluation against TimeBank

62
Graphical Visualization Tool

• Filled Yellow event with just one instance
• Yellow border event without instance or linked
  to multiple instances
• Red border Instance of the event
• Dotted link relation with a signal
• Green Link MAKEINSTANCE
• Blue diamond Timex3 tags
• Purple link TLINK
• Blue link SLINK
• Orange link ALINK
• Signal are between

63
Visualization Process

• Parses TimeML file
• Entity Extraction
• Nodes (Events, Instances and Timex3)
• Links (Temporal, Subordinate, Aspectual)
• Graph generation (graphviz dot format)
• Graphviz processing

64
Utility of Visualization

• Debugging Annotated documents
• Use it as a syntax and semantic validator
• Represent the timeline and partial ordering of
  the events

65
Temporal Math Closure

• " Two Russians and a Frenchman left the Mir and
  endured a rough landing on the snow-covered
  plains of Central Asia on Thursday. ... The
  two Russians arrived on the Mir last August ...
  Solovyov ... celebrated his 50th birthday during
  his six-month space voyage."

66
TIMEBANK STATUS

• 50 Articles Fully Annotated to TimeML 1.0
• 66 Articles Annotated with TIMEX3, Signals, and
  Events
• 100 Articles Annotated with Signals and Events
• 3 Articles from 50 with inter-annotator scores
  (for 2 annotators)

67
Final Report Leftover Items

• Add Generic Event Expressions
• Periodicity and set notation on TIMEX3
• Enhance Temporal Function Expressiveness

68
Near Term Projects

• Create TIMEBANK Gold Standard with Closure
• Adding axioms for computing closure for new LINK
  types
• Computing event ordering within articles
• Text coherence models
• Computing event ordering between articles

69
Open Research Issues

• Persistence and Entailed Events
• The terrorists kidnapped the journalist.
• The President resigned.
• Event Normalization and Quantification
• Three deaths occurred.
• Three people died.
• Generalizing the Treatment of Negation
• No survivors were found.
• The plane did not crash.

70
Impact of TERQAS

• Better understanding of limits of current
  technology
• Preliminary capabilities for answering questions
  using that output
• Standard for Temporal and Event Markup (TimeML)
• Gold Standard Corpus for use by anyone in the
  community (TIMEBANK)
• Add a new dimension to the kinds of QA possible

71
Practical Challenges of TimeML Human Annotation

• Density Highly dense annotation, mainly due to
  links
• Speed Extremely slow process
• 1K/hour per annotator (8.61K took Inderjeet 9
  hours)
• Utility Research communities carrying out other
  tasks need to adopt it

TimeML tag frequencies in 56.6K bytes (raw)
dataset
72
Addressing the Challenges

• Density
• move away from textual annotation for links
  Graphical Annotation
• Speed
• use radical mixed-initiative architecture,
  involving massive pre-processing and interactive
  post-processing and machine learning
• Relevance
• build links to other communities, by showing
  value (e.g., QA, summarization, MT)

73
Efficient Annotation Through Multi-stage Mixed
Initiative Method
Raw Corpus
Annotated Corpus
Pre-processing
Efficiency, reliability gains
Annotated Corpus
Human Tagging
Annotated Corpus
Post-processing
Machine Learning
Annotated Corpus
74
Multi-Document TimeML Annotation for Summarization
Even this simple summary is only possible using
TimeML
Multi-doc TimeML anchors single-doc events, and
merges events across multiple docs (via TimeML
graphs)
75
TimeML for MT

• Extend to multilingual annotation (re TIMEX2
  results on Spanish, French, and Korean)
• Address translation of specialized TimeML
  constructs

76
Human Annotation Accuracy
TIMEX2, 193 news docs 5 annotators
TimeML elements, 3 news docs 2 annotators
run 5/29/02 and 7/20/02
77
TimeML Event Warehousing

• Collects events 24X7 from news feeds
• Builds TimeML graphs from each document
• Aggregates events in database using graph
• Can be used for trend analysis, etc.

78
Conclusions

• TimeML aims to provide a robust markup framework
  for multiple domains and applications
• Compliant and interoperable with Semantic Web
  standards
• Goal to integrate into Document Models and Models
  of narrative structure
• Algorithms can be compared and measured against
  common TimeML-marked up corpora, starting with
  TIMEBANK.

79
www.time2002.org