IL Annotation Experiment Nizar Habash and Bonnie Dorr

1
IL Annotation ExperimentNizar Habash and Bonnie
Dorr

• AMTA 2002
• Interlingua Reliability Workshop

Tiburon, California October 8, 2002
2
Annotation Task

• Assigning a thematic role to each node in a
  syntactic dependency.
• Set of Thematic roles
• AGT INS EXP THM PRC PRD RSL SRC GOL LOC PTH
  TMP BEN PRP MOD MNR DGR NIL NONAGT INS EXP
  THM PRC PRD RSL SRC GOL LOC PTH TMP
  BEN PRP MOD MNR DGR NIL NON
• 20 Sentences from Penn TreeBank Wall Street
  Journal (section 00 sentences 6-25)
• Average Words/Sentence24.75

3
Annotation Example

• ltdoc doc_id"PTB-00-8" sys_id"YOUR-ID"gt
• lttreegt
• 1 a D 3 modrole wordA
• 2 lorillard PN 3 modrole
  wordLorillard
• 3 spokewoman N 4 sbjrole
  wordspokewoman
• 4 say V role
  wordsaid
• 5 , PX 4 modrole word,
• 6 PX 4 modrole word
  
• 7 this D 8 sbjrole
  wordThis
• 8 be V 4 objrole wordis
  
• 9 an D 11 modrole wordan
  
• 10 old AJ 11 modrole wordold
  
• 11 story N 8 prdrole
  wordstory
• 12 . PX 4 modrole word.
• lt/treegt
• lt/docgt

4
Annotation Example

say
mod
mod
mod
obj
sbj
,

be
spokewoman
.
prd
sbj
mod
mod
this
story
lorillard
a
mod
mod
an
old
5
Annotation Example


say
mod
mod
mod
obj
sbj
agt
nil
thm
nil
nil
,

be
spokewoman
.
prd
sbj
mod
mod
thm
prd
mod
mod
this
story
lorillard
a
mod
mod
mod
mod
an
old
6
Annotation Example

• ltdoc doc_id"PTB-00-8" sys_id"Nizar"gt
• lttreegt
• 1 a D 3 modmod wordA
• 2 lorillard PN 3 modmod
  wordLorillard
• 3 spokewoman N 4 sbjagt
  wordspokewoman
• 4 say V
  wordsaid
• 5 , PX 4 modnil word,
• 6 PX 4 modnil word
  
• 7 this D 8 sbjthm
  wordThis
• 8 be V 4 objthm wordis
  
• 9 an D 11 modmod wordan
  
• 10 old AJ 11 modmod wordold
  
• 11 story N 8 prdprd
  wordstory
• 12 . PX 4 modnil word.
• lt/treegt
• lt/docgt

7
Annotation Results

• 8 Annotators
• Technical Problems
• No use of valid roles (role)
• Inventing roles (nul)
• Invalid changes in syntax structure
• Doubling of entries
• No annotations for some words

8
Inter-annotator Agreement
9
Inter-annotatorAgreement Distribution
10
Annotator Similarity Distribution
11
Inter-annotator Agreement Part-of-Speech
POS Distribution
POS Inter-annotator Agreement
12

• Description of task http//www.umiacs.umd.edu/ha
  bash/il-wkshp/il-wkshp.html
• Results of experiment http//www.umiacs.umd.edu/
  habash/il-wkshp/il-wkshp-results.html