Multi-Perspective Question Answering

1
Multi-Perspective Question Answering

• ARDA NRRC Summer 2002 Workshop

2
Participants

• Janyce Wiebe
• Eric Breck
• Chris Buckley
• Claire Cardie
• Paul Davis

• Bruce Fraser
• Diane Litman
• David Pierce
• Ellen Riloff
• Theresa Wilson

3
Part I Overview
4
Problem

• Finding and organizing opinions in the world
  press and other text

5
Our Work will Support

• Finding a range of opinions expressed on a
  particular topic, event, issue
• Clustering opinions and their sources
• Attitude (positive, negative, uncertain)
• Basis for opinion (supporting beliefs,
  experiences)
• Expressive style (sarcastic, vehement, neutral)
• Building perspective profiles of individuals and
  groups over many documents and topics

6
Task Conceptualization

• Perspective in language
• Implications for higher-level tasks

7
Task Annotation

• Manual annotation scheme for linguistic
  expressions of opinions
• It is heresy, said Cao. The Shouters claim
• they are bigger than Jesus.

(writer,Cao)
(writer,Cao,Shouters)
(writer,Cao)
(writer,Cao)
8
Task Summary Representation of Opinions

• Built upon lower-level annotations

9
Task Automate Manual Annotations

• Machine learning
• Identification of opinionated phrases, sources of
  opinions,

10
Task Organizing Perspective Segments

• Unsupervised clustering
• Text features features from the annotation
  scheme higher-level features

11
Evaluation

• Exploratory manual clustering
• Evaluation of annotation agreement
• Evaluation of automatic annotations
• Evaluation of automatic clustering

12
Part II Conceptualization
13
Introduction

• Perspective Framework/filter through which we
  see an event
• Language give insight into perspective of writer/
  speaker and the people they describe
• Our focus subjectivity (reflections of private
  states) as conveyed through language
• Goal Draw from and elaborate on the Annotation
  Guidelines to expand our knowledge of how
  subjectivity is conveyed via linguistic
  expressions.

14
Influences on Perspective

• Identity (President, Assistant Secretary of
  State)
• Attitudes held by a person
• Micro (towards specific individuals, events)
• Macro (belief systems embraced, culture values)
• Genre of the product (editorial, report, white
  paper)
• Context of the writing
• Temporal (writing now about 1960)
• Spatial (writing located in war zone)
• Events (writing now rather than before 9-11)

15
Lexical Clues to Subjectivity

• Attitude Expression
• ltSource, Attitude, Objectgt
• Explicit Attitudes Direct realization of
  private states
• Implicit Attitudes Expressive subjective
  elements
• Other linguistic clues

16
Attitude Expression

• The Source (writer, speaker, another person an
  institution, a document
• The Object (entities, actions, propositions,
  topic, context)
• Attitudes Different typologies

17
Attitude Typologies

• Negative (reject) Positive (enjoy) Other
  (surprise)
• Opinion (criticize) Emotions (fear) Speculation
  (probably)
• Other (must, poor, better)

18
Attitude Types

• Content
• Evidential (allegedly, think)
• Deontic (require, should)
• Assessment (like, excellent)
• Process
• Discourse Markers (so, but)
• Operators (very, not)

19
Types of AE Combinations

• Reinforced Attitude (A) Two AEs of the same
  type (I like her and I admire her as well)
• Weakened Attitude (A-) Two AEs, A and A
• (I like her. I cant stand her politics)
• Entailed Attitude (gtA)- AE A entails AE B
• (He terrorizes people gt people fear)
• Modified Attitude (A(B)) AE A modifies AE B
• (He cruelly criticized the people)

20
Subjectivity Relation to Ideology

• Ideology Beliefs and opinions held
• Approximation of ideology by grouping AEs for a
  given Source
• Grouping Sources with similar ideologies
• Inferring ideology from limited AEs
• Change of an ideology over time
• Prediction of position based on ideology

21
Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
22
Part III Annotation
23
Annotation

• Find opinions, evaluations, emotions,
  speculations (private states) expressed in
  language

Private state state that is not open to
objective observation or verification.
Quirk, Greenbaum, Leech, Svartvik (1985). A
Comprehensive Grammar of the English Language.
24
Two Ways of Expressing Private States

• Explicit mentions of private states and speech
  events
• The United States fears a spill-over from the
  anti-terrorist campaign
• Expressive subjective elements
• The part of the US human rights report about
  China is full of absurdities and fabrications.

25
Two Ways of Expressing Private States

• Explicit mentions of private states and speech
  events Ons
• The United States fears a spill-over from the
  anti-terrorist campaign
• Expressive subjective elements
• The part of the US human rights report about
  China is full of absurdities and fabrications.

26
Nested Sources
27
OnlyFactive
OnlyFactiveyes
The US fears a spill-over, said Xirao-Nima, a
professor of foreign affairs at the Central
University for Nationalities.
28
OnlyFactive

• An example of all OnlyFactiveyes
• The government, it added, has amended Pakistan
  Citizenship Act 10 of 1951 to enable women of
  Pakistani descent to claim Pakistani nationality
  for their children born to foreign husbands.

29
Example

• It is heresy, said Cao. The Shouters claim
• they are bigger than Jesus.

30
Example
The Foreign Ministry said Thursday that it was
surprised, to put it mildly
by the U.S. State Departments criticism of
Russias human rights
record and objected in particular to the odious
section on Chechnya.
31
Other attributes

• Strengths of various components
• Types of private states
• Objects

32
Documents and Annotators

• Over 90 annotated documents
• 100 spans of sources, subjective elements, ons,
• with type and OnlyFactive
  attributes.
• 39 sources, strengths, agent objects
• Participants part-time annotators

33
(No Transcript)
34
Pilot Agreement Evaluationons and subjective
elements

• Agreement measured by percentage of As
  constituents matching Bs, and vice versa

35
Overlapping Spans Ons

• Bush has adopted the most pro-Taiwan posture of
  any president...
• Bush has adopted the most pro-Taiwan posture of
  any president...

36
Overlapping Spans Expressive Subjective Elements

• some of Mr. Chavezs more alarming faults
• some of Mr. Chavezs more alarming faults

37
Untrained Annotators

• 2 groups of 3 untrained annotators (the
  participants) each annotated 4 documents without
  any communication

38
Spans, Untrained Annotators
39
Overlapping Spans Expressive Subjective Elements

• some of Mr. Chavezs more alarming faults
• some of Mr. Chavezs more alarming faults

40
C 74
P 196
D 75
Expressive Subjectivity Agreement
Di 76
T 153
E 29 97
41
Spans, All Annotators
42
Pilot Agreement EvaluationOnlyFactive

• For the ons identified by both annotators
• Measured agreement for Onlyfactive
• Cohens Kappa
• Krippendorf
• 0.67 Tentative conclusions
• 0.80 Definite conclusions

43
OnlyFactive Kappa Values
44
Future work

• Continue annotations by trained annotators
• Continue to explore annotations
• Train new annotators, starting with settled
  scheme
• Even more detail is possible

45
Part IV Summary Representations of Opinions
46
Example
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
47
Sentence 1
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
48
Sentence 1 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
49
Sentence 2
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
50
Sentence 2 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
51
Sentence 3
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
52
Sentence 3 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
53
Sentence 4
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
54
Sentence 4 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive

ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
55
Sentence 5
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
56
Sentence 5 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive ltwriter, chinarepgt
onlyfactive
57
Sentence 6
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
58
Sentence 6 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
59
Sentence 7
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
60
Sentence 7 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
61
Sentence 8
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
62
Sentence 8 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive
63
Sentence 9
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
64
Sentence 9 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt onlyfactive
65
Sentence 10
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
66
Sentence 10 Low-level
The Annual Human Rights Report of the US State
Department has been strongly criticized and
condemned by many countries. Though the report
has been made public for 10 days, its contents,
which are inaccurate and lacking good will,
continue to be commented on by the world media.
Many countries in Asia, Europe, Africa, and
Latin America have rejected the content of the US
Human Rights Report, calling it a brazen
distortion of the situation, a wrongful and
illegitimate move, and an interference in the
internal affairs of other countries. Recently,
the Information Office of the Chinese People's
Congress released a report on human rights in the
United States in 2001, criticizing violations of
human rights there. The report quoting data from
the Christian Science Monitor, points out that
the murder rate in the United States is 5.5 per
100,000 people. In the United States, torture and
pressure to confess crime is common. Many people
have been sentenced to death for crime they did
not commit as a result of an unjust legal system.
More than 12 million children are living below
the poverty line. According to the report, one
American woman is beaten every 15 seconds.
Evidence show that human rights violations in the
United States have been ignored for many years.
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
67
All Low-level Annotations
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
ltwritergt onlyfactive

ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
ltwritergt onlyfactive ltwriter, chinarepgt
onlyfactive
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
68
Creating Summary Representations of Opinions with
Writer As Sole Source
ltwritergt neg-attitude (medium) ? ltreportgt
ltwritergt neg-attitude (medium)
ltwritergt neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt ?neg-attitude
(medium) ? ltUSgt ltwritergt expr-subj (low)
ltwritergt neg-attitude (low)
? ltUSgt ltwritergt expr-subj (low) ltwritergt
neg-attitude (medium)
ltwritergt onlyfactive
ltwritergt onlyfactive
ltwritergt neg-attitude (low) ? ltUSgt
ltwritergt expr-subj (low)
69
Creating Summary Representations of Opinions with
Writer As Sole Source
ATTITUDE value neg strength low
ltUSgt
70
Creating Summary Representations of Opinions for
Other Sources
ltwritergt onlyfactive ltwriter,
many-countriesgt neg-attitude (medium)
? ltreportgt ltwriter, many-countriesgt
extreme
ltwriter, many-countriesgt neg-attitude (high,
high, medium)
71
Creating Summary Representations of Opinions
72
Creating Summary Representations of Opinions for
Other Sources
ltwritergt onlyfactive

ltwriter, info-officegt
neg-attitude (medium) ? ltUSgt
73
Creating Summary Representations of Opinions
ATTITUDE value neg strength low
ATTITUDE value neg strength high
ltUSgt
ATTITUDE value neg strength medium
74
Summaries

• Describe collective perspectives with respect to
  an issue or event
• Describe the perspective of a particular writer,
  individual, government, or news source with
  respect to an object or issue
• Describe the temperature and level of rhetoric
  about an event or issue
• Create perspective profiles for agents, groups,
  news sources and look for outliers and changes
  over time

75
Part V Automating Manual Annotations
76
Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
77
Learning Architecture
Evaluation
Training Data
Learner
Learner
Annotation Database
Gate NE
CASS
Pattern Finder
78
Learning problems

• Identify expressive subjective elements
• Identify private state and speech event phrases
  (ons)
• Identify nested sources
• Discriminate facts and views (onlyfactive)
• Classify strengths of opinions
• Classify types of opinions
• Identify objects of opinions

79
Pilot experiments
Precision Recall F-measure
Baseline 69.9 47.7 56.7
Naïve Bayes 46.7 76.6 58.0
KNN 69.6 63.4 66.4
80
Future Work

• Feature generation (lexical, collocational,
  contextual)
• Feature selection
• Learning strategies (bootstrapping, co-training)

81
Part VI Clustering
82
Solution Architecture
Annotation Architecture
AnnotationTool
Learning Architecture
LearningAlgorithms
Trained Taggers
Application Architecture
PerspectiveTagging
DocumentRetrieval
DocumentClustering
Question
Other Taggers
83
Application Architecture
Multi-perspective Classifiers
Document Clustering
Documents
Annotation Database
Gate NE
CASS
Feature Generators
84
End-User Evaluation Goal

• Establish framework for evaluating tasks that
  would be of direct interest to analyst users
• Do an example evaluation
• Most end-uses involve grouping opinions
• Analyzing
• Profiles
• Finding outliers (new info, changes)
• Suggests clustering as a main initial component

85
Manual Clustering Exploration

• What kind of clusters do people naturally produce
  when given no instructions?
• 6 MPQA participants Penny Lehtola manually
  clustered opinions from 1-3 topics
• 19-31 documents per topic
• Multiple opinions per document
• Everybody did one specific topic (Mugabe)
• Targeted 4 hours per participant
• Video-conference to discuss

86
Manual Clustering Analysis

• Background strongly influenced cluster type
• Linguist separately clustered every sentence
  according to purpose of sentence
• Would be useful for information extraction to
  database
• Ex-analyst clustered according to whether
  immediate threat existed
• 4 people used the proposed task framework
• Pro-con top-level clusters
• Different sub-cluster strategy

87
Manual Clustering Issues

• Treatment of supporting evidence important
• People disagreed how to include it.
• One had separate clustering for evidence
• Some included as part of opinion, others not
• There needs to be some way of linking to opinion
• Random outliers vs important outliers
• People wanted several opinions in sub-cluster
• Pro-con with sub-clusters offers ability to
  duplicate sub-cluster on both pro and con side
• Important disagreements appear on other side in a
  singleton sub-cluster

88
Manual Clusters Agreement

• Among the 4 pro-con two-level clustering
• 50-80 agreement at top level
• Very weak agreement at sub-cluster level
• Even those sub-clustering on common basis
  differed
• Does head of government task force speak for the
  government?
• Segment boundary agreement about 60 for those
  who treated evidence the same way.

89
Manual Clustering Overall

• Clustering
• Important, useful
• Everybody does it differently for different
  reasons
• Implies evaluation of clusters
• Must be relative to clearly defined task
• Gold standard evaluation very difficult for
  anything other than simple clustering
• Outlier evaluation
• When important,must be explicitly addressed,

90
User Task Topic

• U1 User states topic of interest and interacts
  with IR system
• S1 System retrieves set of relevant documents
  along with their perspective annotations

91
User Task Question

• U2 User states particular perspective question
  on topic.
• Question should
• identify source type (eg, governments,
  individuals, writers) of interest.
• Be a yes/no (or pro/con) question for now

92
User Task Question Response

• S2System clusters documents
• based on question,text,annotations
• goalgroup together documents with same answer
  and perspective (including expressive content).
• System,for now, does not attempt to label each
  group with specific answers.
• Target a small number of clusters (2-4?)

93
User Task Cluster Feature

• U3 User states constraints on clustered
  documents or segments.
• These might be geographic, date, ideological,
  political, religous
• S3 System shows subclusters or highlighted
  documents

94
User Task Results

• U4 User gets impression (visual or statistical)
  whether constraints match clusters.
• Easy visualization of exceptions

95
FBIS Document Collection

• Large collection of 270,000 foreign news
  documents from June, 2001 to May, 2002
• Almost all FBIS documents with a small number of
  other relevant docs.
• From MITRE MITAP system
• Collection available for government contractors

96
FBIS Document Collection Features

• English Language
• 60 FBIS translated
• 40 source English
• 20 TV/Radio
• 5 Identified as editorials

97
WNC Collection (Docs,Topics)

• Subset of FBIS
• Ran 8 topics on full FBIS retrieving 200
  documents each
• Identified 575 of those docs as being publicly
  available for a small fee from World News
  Connection (wnc.fedworld.gov)

98
WNC Distribution

• MITRE/MPQA arranging to have WNC distribute both
  data and annotations
• Will require a one-time setup fee then modest
  individual fee from each research group.
• WNC is a division of the federal National
  Technical Information Service (NTIS)
• Time-frame of public distribution being worked out

99
Topics

• 8 topic statements.
• Couple of clauses.
• Each topic has 40-105 identified related docs
  within WNC collection.
• 4 topics have complete answer identification
• Answer passage spans identified.
• Generally, 0-4 answer passages per doc
• Average 1 per doc
• Annotations require little training, 5 minutes/doc

100
1. Simple evaluation

• Can we usefully cluster passages of documents?
• For each of 4 topics and 40-82 WNC docs
• Find best passage in each doc answering topic
• Cluster passages
• Evaluate using answer passages
• Do Pro or Con answers tend to occur within
  the passages of a single cluster?

101
Simple Evaluation Details

• Passages Determined statically. 800 characters
  broken on sentence boundaries, and overlapping.
• Clusters
• Complete-link agglomerative clusters at top level
• Average-link agglomerative clusters afterwards.
• Target number of clusters (3)
• No cluster contains more than 2/3 docs.

102
Simple Evaluation Comparison

• Base case Determination of best passage from
  terms of passage only
• Opinion case Boost similarity of passage if
  automatically determined ON occurs.

103
Topic 1 Zimbabwe - Base
Cluster Both Yes No Neither
1 0 3 11 18
2 1 1 5 6
3 2 2 4 4
104
Topic 1 Zimbabwe - Opinion
Cluster Both Yes No Neither
1 1 5 8 15
2 1 1 8 8
3 0 1 3 6
Opinions help a bit
105
Topic 3 Kyoto Base/Opinion
Cl Both Yes No Nthr
1 0 0 7 7
2 1 3 4 7
3 3 2 3 3
Cl Both Yes No Nthr
1 0 2 2 8
2 2 2 10 8
3 2 1 2 1
Opinions hurt a bit
106
2. Retrospective Evaluation

• Is poor performance due to opinions, or to the
  clustering?
• Suppose we could find opinion passages perfectly.
  Could we then cluster them?

107
Retrospective

• Take all passages given by the topic answers.
• Often several passages per document
• Most are shorter than previous experiment.
• Some are the entire document.
• Cluster these passages using same algorithms as
  previously.
• Examine whether the Yes and No answers group
  in the clusters.

108
Retrospective Zimbabwe
Cluster Both Yes No
1 1 1 18
2 0 1 10
3 0 23 34
109
Retrospective Kyoto
Cluster Both Yes No
1 1 6 19
2 0 2 1
3 0 4 7
110
End-User Summary

• We have an end-to-end system that
• Retrieves documents
• Adds opinions annotations using NLP tools
• Clusters documents or passages partly based on
  those features
• Evaluations show
• Clustering itself needs work before we can
  evaluate the use of perspective.

111
Future work

• Evaluate the end-user task described in the
  mid-term.
• Needs more annotated WNC docs.
• Current maximum is only 4 annotated docs for any
  query. Not enough for even a toy evaluation!
• Needs more investigation of pure clustering
  before we can evaluate opinions with it.
• Retrospective evaluation is a good testbed.

112
Future Work 2.

• Techniques not yet being used
• Query expansion including explicit opinion
  signs
• Collection statistics for opinions and agents
• Just finished first opinion annotations of entire
• Collection (base heuristic ONs)
• (Hardware for large collection arrived middle of
  week 7)

113
Summary of Future Work

• More document annotations and studies
• Incorporate summary representations of opinions
  into end-user evaluations
• Feature generation, feature selection, and
  additional learning strategies for recognizing
  expressions of opinions
• Expand end-user evaluation to include richer
  aspects of the opinion annotations and summary
  representations
• Other end-user evaluation tasks