Summary of Common Task

1
Summary of Common Task

• Semantic Annotation Planning Meeting
• University of Maryland
• April 14, 2005
• Presented by
• Lori Levin
• Language Technologies Institute
• Carnegie Mellon University

2
Submissions for Common Task

• Berkeley, FrameNet
• CMU, Javelin
• CMU, Scone
• CMUPitt
• Cyc
• IAMTC
• MITRE, ACE
• NYU
• Penn, Discourse Treebank

• Propbank
• Princeton, WordNet
• UMD, LCS
• UMD, AIL
• UMD, Semantic Frame
• UMBC
• UNT
• ACE
• LCC

3
Annotation Tool

• Thank you University of Maryland!
• Zooming for different levels of detail
• Faces on nodes
• Different views

4
Goal for this session

• What is the aggregate collection of phenomena
  that were annotated in the homework assignment?
• What is generally agreed?
• What is still in flux?
• What is under-represented?
• What is not represented?

5
Fitting into the goals of the workshop

• What knowledge has been derived from documents?
• What is in
  this gap?
• What are all of the knowledge elements you could
  get from a document?
• What is a practical plan to get there?

6
Outline

• Summary of shared task results
• Aggregate of meaning elements that were
  identified
• WSJ
• Enron
• Discussion
• What is under-represented

7
Close analysis of WSJ Sentences 1 and 2

• The struggle against the drug lords in Columbia
  will be a near thing.
• Meaning of "near thing"
• literal "near" plus "thing"
• hot thing
• thing with uncertain outcome
• thing that will begin soon
• thing that will end soon
• This week, the government arrested Jose Abello
  Silva, said to be the fourth-ranking cartel
  leader.

8
Apology in advance for misconstrual of your
submission
9
Inventories of Concepts and slot namesor
Ontologies

• PropBank
• NomBank
• Omega
• FrameNet
• WordNet
• Cyc
• LCS
• Mikrokosmos
• Scone
• Javelin
• Impromptu

10
Phenomena Annotated in WSJ

• Frames with named arguments
• Indexed events and entities
• Coreference
• Pre-known facts from knowledge base
• Metadata about the article, publication, and date
• Temporal information
• Truth and reality
• Familiarity
• (and other things expressed by determiners)
• Subjectivity, modality, and who to attribute it
  to
• Links to ontology
• Links between lexical items
• Links to text strings
• Part of speech, syntactic features, constituent
  structure
• Grammatical relation names or dependency relation
  names

11
Phenomena annotated/not annotated

• Not all of the phenomena addressed in the
  previous session showed up strongly in the
  annotations of sentences 1 and 2.
• Time constraints
• Leave it to the experts
• Wasnt in those sentences

12
Frames and arguments Snt1

• struggle(against drug lords)
  (nyu, iamtc, lcs, unt)
• struggle (someone, drug lords)
  (scone)
• opposition-event(drug lords, colombia, phase
  begin) (umbc)
• hostile-encounter(leadership, columbia,
  after-now, unspecified-outcome)
• 
  (pgh)
  
• hostile-encounter(dni, ini, leadership, columbia)
  (fn)
• antagonist(drug lords, struggle)
  (cyc)
• be(near)
  (fn, lcs)
• be(struggle, thing)
  (ail, javelin)
• be(struggle, will, near thing)
  (propb)
• difficulty(struggle, DNI)
  (fn)
• near-thing(struggle)
  (pgh)
• thing(near,struggle)
  (iamtc)
• lord(drug, in Columbia)
  (iamtc)
• in(drug lords, Colombia)
  (cyc, lcs, scone)
• exists(unknown winner of struggle)
  (cyc)

The struggle against the drug lords in Columbia
will be a near thing.
13
Frames and arguments Snt2

• leader(cartel, leader/abello)
  (nyu,
  pgh)
• leader(pro/abello, fourth ranking, cartel)
  (iamtc, lcs, fn)
• government(government, columbia)
  (nyu)
• hasmember(druglords, abello)
  (cyc)
• arrest(leadership/government, abello, this week)
• 
  (pgh, iamtc, fn, lcs, ail, umbc, propb,
  javelin)
• performed-by(arrest, Colombian government)
  (cyc)
• arrestee(arrest, Abello)
  
  (cyc)
• dateofarrest(arrest, week)
  
  (cyc)
• said(leader)
  
  (iamtc, lcs)
• said(speaker/generic, be)
  (fn,
  propb)
• said(be)
  
  (ail)
• someone claims abello is fourth in succession
  (cyc)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
14
Explicitly indexed entities and events Snt1

• article (the news article that contains this
  sentence) (cyc)
• author (cyc)
• publication date (cyc)
• struggle (cyc, umbc)
• drug lords (cyc, umbc, scone)
• set of drug lords (scone)
• nation/Colombia (umbc, scone)
• struggle is referential (unt)
• Underrepresented because its time consuming?
  Waiting for ACE to do it?

The struggle against the drug lords in Columbia
will be a near thing.
15
Explicitly indexed entities and events Snt 2

• Columbia/Colombia (nyu, unt)
• arrest (cyc, umbc, propb, scone)
• set of people in the government who arrested
  Abello (scone)
• saying (propb)
• week001 (cyc)
• governmental-organization (umbc)
• human (abello) (umbc, scone)
• social role (leader) (umbc)
• cartel (umbc, scone)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
16
Coreference

• social role (leader)/human (abello)
• (umbc, scone)
• abello is one of those drug lords from sentence 1
  (unt)

The struggle against the drug lords in Columbia
will be a near thing.
This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
17
Related Facts from Knowledge Base

• Colombia is a South American country (wn)
• Very under-represented in this assignment.

The struggle against the drug lords in Columbia
will be a near thing.
This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
18
Links to parent nodes in Ontology

• cyc, lcs?, wn, scone, unt

19
Temporal Snt1
The struggle against the drug lords in Columbia
will be a near thing.

• struggle is single-happening (NYU)
• be-thing is future (nyu)
• struggle is after now (pgh)
• tense future (lcs, unt)
• opposition event, phase begin (umbc)
• opposition event, a short time from now (umbc)
• end time of struggle will be near current time
  (scone)

20
Temporal Snt2

• 1989 week 41 (mitre)
• time t1, zero weeks from now (pgh)
• week001 is temporally subsumed in the calendar
  week of the date of
• publication (cyc)
• time present
• arrest is before now (propb)
• arrest is this week (propb)
• arrest started and ended this week? (scone)
• arrested, single-happening (nyu)
• said, current (nyu)
• leader, current (nyu)
• government, always (nyu)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
21
Temporal Relations

• Inconsistently represented across submissions
• Waiting for ACE?
• Temporal people are at a workshop in Dagstuhl?

22
Truth/Reality Snt1Snt2

• struggle true (NYU)
• be-thing true (NYU)
• arrest, true (nyu)
• said, true (nyu)
• be leader, true (nyu)
• government, true (nyu)
• abello is leader, epistemic value 1 (umbc)
• fourth ranking cartel leader exists (scone)

The struggle against the drug lords in Columbia
will be a near thing.
This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
Inconsistently represented in the submissions.
23
Familiarity and meaning elements expressed by
determiners in English Snts12
The struggle against the drug lords in Columbia
will be a near thing.

• struggle (cyc, ail)
• drug lords (cyc)
• a near thing (ail)
• government of Colombia (cyc)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
Seems to need some work.
24
Attitudes and who to attribute them to Snt1

• writer has a medium, externally directed,
  negative attitude toward struggle (pgh)
• writer has medium, externally directed, negative
  certainty about near thing (pgh)

The struggle against the drug lords in Columbia
will be a near thing.
25
Attitudes and who to attribute them to Snt2

• "said" is attributed to unknown (nyu, pgh)
• "said" is subjective (unt)
• the sentence is an objective speech event from
  the perspective of the writer (pgh)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
26
Rhetorical Structure

• Sentence 1 IMPLICIT-THIS IS BECAUSE Sentence 2
  (penn)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
The struggle against the drug lords in Columbia
will be a near thing.
27
Syntactic Features

• Part of speech and inflectional features
• (fn, wn, iamtc, lcs, umbc)
• Grammatical relations or dependency relations
• fn, iamtc, lcs, ail

28
Text Anchors

• "will" Future (NYU)
• "near thing" expressive element (pgh)
• "this week" (mitre)
• "said" attribution (nyu)
• lexical unit for each node (fn, umbc)
• labelled text strings (propb)

This week, the government arrested Jose Abello
Silva, said to be the fourth-ranking cartel
leader.
The struggle against the drug lords in Columbia
will be a near thing.
29
Analysis of Enron DialogueKathy Baker

• Predicates/concept frames
• Concept names
• Inventory of Arg names
• Discourse markers e.g.
• "How you doin, man" "Ha Ha" "Hey Rich"
• Discourse connectives e.g. 'cause, and so
• Dialogue acts
• Events null event reconstruction
• Event markers
• Modality
• Negation
• Roles
• Attributes

30
Phenomena in Enron Dialogue

• Veracity
• Polarity
• Speaker attitude (subjectivity)
• Attribution of subjectivity
• Fact time
• Relative Time
• Epistemic anchor (for veracity)
• Aspect
• Intensity of subjective response
• Deixis
• Discourse relations

31
So you got a -- so you're checking a switch on
the steam engine

• speech act
• request-info
  UMBC
• communication
  UMD Semframe
• speech roles
• speaker, hearer/addressee
  UMBC, UMD Semframe
• Event
• inspect
  
  UMBC
• superclasses/ontology
  IAMTC
• check switch, "got" event coreference
  Propbank
• Veracity
• fact
  
  NYU
• attribution/discourse
• Bill
  NYU,
  UMBC
• discourse connective
• so
  
  Propbank
• Location relation
• realm spatial
  UMD Semframe

32
B We want you guys to get a little creativeR
OKB and come up with a reason to go down

• Dialogue acts
• vocal pause
• request action
• acknowledge
  
• Entity co-reference
• Direct subjective frame Bill, we
  
• Speaker attitude
• text anchor want
• attitude type positive attitude
• epistemic anchor
• want NYU
• Intensity
• medium (direct subjective, expressive subjective
  and attitude frames)
• Substantiation
• insubstantial true
• veracity unclassified

33
What under-represented, not represented,
inconsistently represented?

• Social relationship between Bill and Rich.
• Inferences that can be drawn from the text?
• How much could you blow up with 1000 pounds of
  dynamite?
• Background knowledge that contributes to the
  understanding of the text?