Annotating Committed Belief

1
Annotating Committed Belief

• Mona Diab, Lori Levin, Teruko Mitamura, Owen
  Rambow
• CMU/Columbia-CCLS
• diab,rambow_at_ccls.columbia.edu
• lsl,teruko_at_cs.cmu.edu

2
Motivating Example 1

• Suppose IR System
• Query show documents discussing instances of
  peasants being robbed of their land
• Document found 1 The people robbing Iraqi
  peasants of their land should be punished
• Document found 2 Robbing Iraqi peasants of their
  land would be bad

Relevant
Irrelevant
3
Motivating Example 2

• Suppose QA System
• Question Did the humanitarian crisis in Iraq
  end?
• Text found 1 He arrived on Tuesday, bringing an
  end to the humanitarian crisis in Iraq.
• Text found 2 He arrived on Tuesday, calling for
  an end to the humanitarian crisis in Iraq

Answer Yes
Answer I do not know
4
What is Committed Belief?

• Model two discourse participants
• speaker/writer (S)
• hearer/reader (H)
• S and H have beliefs about the world
• Note knowledge belief encyclopedic truth
  not relevant here
• Note discourse participants also have desires
  and intentions not relevant here

5
What is Committed Belief?

• A lot of (but not all) communication is about S
  informing H of Ss beliefs
• Patrick is a nice man H believes nice(Patrick)
• I hope Patrick is a nice man H has no belief
  about Patrick being nice or not, only a desire
• Typically, Ss goal is that H adopt those beliefs
• Beliefs about the world can be of different
  strengths linguistic means can signal strength
• I know Patrick is a nice man Committed Belief
• I think Patrick is a nice man Non-Committed
  Belief
• Can use numerical weights to represent strength
  we use 2 discrete categories
• Committed belief
• Non-committed belief

6
Annotation Categories

• We annotate propositions on the verb or noun
• Committed belief S indicates in this utterance
  that S believes the proposition
• I know Mark and Sandra have eloped.
• Non-committed belief S identifies the
  proposition as something which S could believe,
  but S happens not to have a strong belief in the
  proposition
• Mark and Sandra may have eloped.
• Not applicable for S, the proposition is not of
  type in which S is expressing a belief, or could
  express a belief. Usually, this is because the
  proposition does not have a truth value in this
  world.
• I wish Mark and Sandra would finally elope.

7
Why Is RecognizingCommitted Belief Important?

• Committed-Belief Annotation Distinguishes
• Propositions that are asserted as true (CB)
• Propositions that are asserted but speculative
  (NCB)
• Propositions that are not asserted at all (NA)
• Important whenever we need to identify facts
• IR
• QA
• Summarization

8
Map from Committed Belief Detection to Useful
Functionality
Traditional IR System Or QA System
Automatic Committed Belief Filter
Response including unasserted propositions
Response based on asserted propositions only
9
Committed Belied is not Tense
Past Future
CB Smith was assassinated. Smith will be assassinated tomorrow.
NA I hope Smith regretted his acts. I hope Smith will regret his acts.

• CB committed belief, NA No asserted
  belief
• We have a special feature to indicate future
  tense on CB (committed belief) and NCB
  (non-committed belief)

10
Committed Belied is not Factivity
Fact Opinion
CB Smith was assassinated. Smith was a nasty dictator.
NA Smith will be assassinated. Smith will become a nasty dictator (once he is in power).

• CB committed belief, NA No asserted
  belief
• Committed-belief annotation and factivity
  annotation are complementary

11
Some Examples

• Word File (English)
• Mona on Arabic

12
Status

• Have complete manual
• Have core corpus annotated (including Arabic),
  working on 15 double annotation

13
Proposed Annotation Work

• Perhaps annotate desire in same manner (much of
  NA is desire)
• Work on one language until prediction (across
  genres!) works, then move to next language
• We do not know how much data is needed
• We do not know how much variability in expressing
  committed beliefs between languages there is