Measures of Effectiveness in PA

1
Measures of Effectiveness in PA

• Maureen Taylor, Ph.D.
• Gaylord Family Chair of Strategic Communication
• University of Oklahoma

2
Training Objectives

• To illustrate how primary and secondary research
  inform decision making and provide evidence of
  communication effectiveness

3
PA has Evolved into an Applied Science

• Collect and analyze information on the changing
  knowledge, opinions, and behaviors of key publics
• Serve as the central source of information about
  an organization and as the official channel of
  communication between the command and its publics
• Communicate significant information, opinions,
  and interpretations to keep an organizations
  publics aware of command policies and actions
• Coordinate activities that affect commands
  relationships with its publics and other groups.

4
RPIE

• RPIE is a strategic planning tool.
• R Research
• P Planning
• I Implement
• E Evaluate
• I will cover the R and E

5
Foundation of Measuring

• Formative research provides information about the
  environment BEFORE a public affairs effort. It
  provides the baseline.
• Summative research provides evidence of impact.
  It is conducted at the END of the PA effort. This
  data shows change.
• Change is what we want from our efforts.

6
Research

• What kinds of primary and secondary research have
  you conducted in your PA role?

7
Secondary Research

• Dont reinvent the wheel.
• There is a lot of free data out there.
• Previously published research can be used to
  inform decision making and develop strategic
  messages.

8
Secondary Sources

• Census
• Periodicals
• Journals
• Chamber of Commerce Reports
• Blogs
• Military Sources

9
Primary Research

• You may also be asked to collect primary research
  to know your publics or plan a campaign.
• Primary research is something that you collect
  and interpret to prove effectiveness.
• Primary research is collected using different
  methodologies.

10
Methodology

• Each method has strengths and limitations
• Two broad categories of primary research
• Qualitative (describing)
• Quantitative (measuring)

11
The Value of Descriptive Research

• Qualitative research provides narratives and
  anecdotes that are persuasive
• The school in Iraq
• The family helped by an aid organization
• The business that reopened because security
  improved
• The newspaper that could restart printing because
  it is safer now to disagree with the government

12
Useful Qualitative Methods

• Interviews
• Focus groups
• Observations

13
Interviews and Focus Groups

• Valuable for understanding WHY people think,
  believe, and act in certain ways.
• They provide anecdotal evidence, narratives,
  stories, and the words of the people.
• News is narrative.

14
Using Interviews in Program Evaluation

• RQ How have USAID political transition grants
  helped independent media outlets in the former
  Yugoslavia promote tolerance and democracy?
• Method Interviews with over 140 media
  organizations in Bosnia, Croatia, Serbia and
  Kosovo. (over a 10 year period)

15
Findings

• Saturation of the Data
• Most owners and editors reported that they were
  able to develop public affairs programming
  through grants to cover stories that would not
  normally make the nightly news.
• Allowed for the development of in-depth, pro
  social stories that did not need advertising
  support.

16
Focus Groups

• In Armenia, focus groups with media organizations
  identified gaps in training, news coverage, and
  network consolidation.
• Outcome IREX reconsidered its model for Armenian
  media.

17
Observational Methods

• Watching people interact and/or use services.
• This method tells us HOW people are doing certain
  things.

18
Example of Observational Research

• Darfur Peace Process
• How can we find out which channels of
  communication are most trusted?
• I sat in the corner of a market cafe and watched
  Darfuris listen to radio (Government and BBC
  Arabic) (also looked at cell phone usage)

19
Lessons Learned

• Sometimes we have flawed assumptions about how
  people behave.
• Observations can help us support or refute these
  assumptions.
• Bus stop ethnography is valuable in formative
  research.

20
Limitations of Interviews

• What are some limitations of interviewing for R
  and E?
• How can you avoid these limitations?

21
Reality Check

• Is one good story enough evidence for decision
  makers?
• Selection bias
• Social desirability

22
Quantitative Research

• Numbers talk.
• Surveys, content analysis, and experiments can
  provide these numbers.

23
Surveys

• Develop specific questions that allow you to
  measure awareness, attitudes, and behaviors.
• Data is only as good as the survey questions and
  the sampling method used in collection.

24
The Best Surveys

• Specific goals
• Easy flow of questions
• Use interval level questions (likert, frequency)
• Short format
• Sampling methods ensure reliable data

25
The Worst Surveys

• Fishing expeditions
• Questions jump from topic to topic
• Open ended questions
• Long and complicated
• Convenience sample

26
Collecting Data to Shape Policy

• Surveys of public opinion of media and NGOs after
  the Bosnian war
• Randomly sampled 1200 Bosnians from each of the
  regions to ascertain which media they trusted
  when they needed information to make a decision
• USAID changed its funding strategy to stop
  supporting low impact media and NGOs

27
Surveys in PA

• Learn about internal and external publics
• Cost effective to piggy back on other surveys
• Random sampling allows us to generalize from a
  few to many
• Online surveys are valuable (but must be
  interpreted cautiously)

28
Online Surveys

• For minimal cost, you can upload surveys and
  invite people to participate.
• But, consider
• selection bias (strong opinions)
• non random
• software may not allow for skipped questions

29
Experiments

• Is there a causal (or correlative) relationship
  between a strategic communication effort and an
  outcome?
• Example Communication campaigns in Malaysia for
  improved ethnic relations.

30
Neighborliness Campaign in Malaysia

• Quasi experimental design Compared attitudes and
  behaviors of those who received the communication
  campaign to those who did not.
• Findings People in the campaign neighborhoods
  were less likely to have positive attitudes about
  other ethnic groups and had less frequent
  positive interactions with members of other
  ethnic groups.

31
Content Analysis

• One way to measure the impact of our messages.
  This allows us to count the number of times our
  PA output has appeared in the media.
• But, more importantly, it also allows us to
  examine the tone, prominence and placement of
  messages.

32
Break Out Session

• In the break out session I will take you through
  the steps to content analyze news for PA
  messages.
• We will learn how to conduct a content analysis
  and then complete a practical exercise in content
  analysis.

33
Break Out Session Objectives

• To provide in depth background about content
  analysis
• To provide practical training in conducting
  content analysis
• To assist you in using content analysis to create
  and evaluate strategic messaging.

34
A Frequent PA Tool

• Take the media coverage of your organization and
  systematically analyze it for
• Tone (positive, negative, neutral)
• Prominence (where is it in the paper, news)
• Accuracy (did they get it right? Why not?)
• Other factors

35
What Can Be Analyzed?

• Newspaper articles, radio stories, and television
  news stories (and programs) can be content
  analyzed to examine the occurrence of certain
  features.
• Internet Web sites and chat rooms can also be
  studied.
• Darfur study uses chat rooms and blogs as proxy
  data.

36
Software Programs

• There are a variety of software programs that are
  used for content analysis.
• If you use this software, then your job would be
  to interpret the findings.
• Use the findings to strategize how to address
  any problems/opportunities.

37
Best Practices for Human Coding

• Most of you will not have software and will have
  to task subordinates with the actual coding.
• Your role is interpret the findings, advise the
  command, and develop strategic messaging.

38
Communication with Stakeholders

• PA communication is based on the assumption that
  we have an obligation to inform our stakeholders.
• We also have the right to try to influence their
  awareness, attitudes and behaviors.
• We must be open to being influenced as well.

39
Information Subsidies

• Organizations provide content to media
  organizations.
• Research Findings
• of content that makes it into news
• Direct quotes from news releases

40
Different Nations Different Subsidies

• Information subsidies vary across the world. In
  some nations, media accept large portions of
  materials from public relations efforts.
• PA materials may be suspect.

41
Content Analysis as a Social Science Methodology

• Social science applies many of the tools of the
  physical sciences to studying human beings.
• Social science means that PA can be studied,
  measured, and improved using data rather than
  guess work.

42
Media Content Analysis

• Method to count results of PA activities
• Usually, we look quantity vs. quality
• Tactic/output based
• Success is often defined in terms of placement of
  strategic messages in media.

43
Minimizes Subjective Analysis

• Berelson (1952) defined content analysis as a
  research technique for the objective, systematic
  and quantitative description of the manifest
  content of communication (p.18).
• It attempts to take the subjectivity out of the
  evaluation.

44
Value of Content Analysis

• It allows the researcher to make inferences by
  objectively and systematically identifying
  specified characteristics of messages (Holsti,
  1969, p. 14).
• We can judge news based on previously specified
  content areas.

45
Use in Media Analysis

• Newspaper articles, radio stories, and television
  news stories (and programs) can be content
  analyzed to examine the occurrence of certain
  features.
• Internet Web sites and chat rooms can also be
  studied.

46
Proxy Research

• When public opinion is difficult to ascertain, we
  can use content analysis of news and editorials
  as a proxy of what people are thinking.
• Darfur Example

47
Unit of Analysis

• Different Types of Units of Analysis in Content
  Analysis
• Examples Newspaper, News program, Story, Words,
  Headlines, Photos, Quotes (sources)

48
Building from simple to complex

• Clip counting
• Circulation and readership analysis
• Advertising value equivalence
• Placement analysis
• Prominence analysis
• Message analysis
• Tonality analysis
• Overall quality of coverage
• Competitive analysis (Share of discussion)
• (this portion of the lecture is from Dr. Holroyd)

49
Media Content Analysis

• 1. Clip counting
• How often did my story appear?
• 2. Circulation and readership analysis
• Add to clip counting with data gathered from
  secondary sources
• Looking for circulation, readership, demographics
  of audiences
• Helps answer the Who might have read/heard this?

50
Media Content Analysis

• 3. Advertising value equivalence (AVE)
• If I had to pay for this space, how much would
  it cost?
• I paid 30,000 for this PA effort but got an AVE
  of 100,000.
• Allows for comparison vs. advertising budgets
• To calculate costs, consider day of publication,
  location in publication, article/story length,
  etc.
• Use reluctantly
• Apples-oranges comparison since PA ? Ads

51
Media Content Analysis

• (Begin assigning values)
• 4. Simple content analysis
• Variety of characteristics to define article or
  story are assigned a value
• Front page?
• Above the fold?
• Picture? Color?
• Half (more?) of article before the jump?
• Use computer programs to develop overall scores
  of success

52
Media Content Analysis

• 5. Message Analysis
• Were my key command messages in the article?
• More points if messages are earlier in the
  article
• Must know what key messages you wanted to convey
  prior to review

53
Media Content Analysis

• 6. Tonality Analysis
• Adds subjective assessment of content
• Was the report favorable or unfavorable to my
  project?
• Variety of ways to score
• Positive, neutral or negative
• 0-100 with 50 neutral
• Assess each article individually and then add
  together to get overall impression of impact

54
Media Content Analysis

• 7. Prominence Analysis
• Considers 6 factors
• 1. Publication where article appears
• 2. Date of appearance
• -- Sunday readership higher than Monday
• 3. Overall size of article (column inches)
• 4. Where in the publication it appears
• 5. Use of photography or not
• 6. Size of headlines
• Each assigned a code and analyzed

55

• 8. Quality of Coverage
• Considers a combination of factors such as
  tonality, prominence and message inclusion
• Each factors in to create overall score for
  article

56
Media Content Analysis

• 9. Competitive Analysis
• Compare what coverage the competition is getting
• Compare clips, prominence,
• Way to assess relative performance in media
• Sometimes called Share of Discussion
• You had 8 great stories. Other side had 29 so-so
  ones. You lose.
• Not often done but VERY important
• Looks at issue from audiences view rather than
  what you think youve conveyed.

57
Walter Reed Hospital Crisis

• In 2007, the Washington Post broke a story about
  poor conditions for wounded soldiers at the
  Walter Reed Army Medical Center in Washington,
  DC.
• Here is what content analysis can do for the PAO.

58
Media Tracking
      Score Score Score Score Score
Date Title Reporter, Placement Outlet Tone Place. Prom. Msg Anal Total (Notes)
2/25 Admin Issues Cited at Walter Reed Steve Vogel, A9 W Post 0 8 3 10 21
2/24 Review at Walter Reed is Ordered Steve Vogel, A1 W Post 0 10 3 8 21
2/24 Armys Preemptive News Briefing Howard Kurtz Columnist, C1 W Post -3 6 3 6 12
2/21 Swift Action Promised at Walter Reed Dana Priest, Anne Hull, A8 W Post -3 8 3 8 16
2/20 Army Fixing Patients Housing Dana Priest, Anne Hull, A1 W Post -3 10 3 6 16
2/19 US Army Facility Reported in Poor Shape Robert Siegel (with Dana Priest, Anne Hull) NPR -3 6 3 0 10 A
2/18 Soldiers Face Neglect, Frustration at Armys Top Medical Facility Dana Priest, Anne Hull A1 W Post -3 10 3 0 10 B
Green 20 Amber 11-19 Red 10 under Notes for
RED scores A. Reporter sent corrective
memo B. Letter-to-editor sent
59
Challenges of Content Analysis

• Flaw 1 Not determining message accuracy
• Accuracy is based on 4 factors
• Basic facts included?
• Misstatements (reduce score)
• Incomplete information
• Omissions
• Flaw 2 Not linking messages to objectives
• Where is the story in the projects lifecycle?
• Awarenessthen knowledgethen interestthen
  intent/behavior

60
What Can CA Measure?

• Content analysis is a measure of level of effort,
  activity or output
• By watching media content change, we can show
  effectiveness in strategic messages
• Cannot determine impact on key audiences
• Focuses on quantity over quality
• Other methods needed to triangulate findings to
  make the claim of impact

61
End May Not Equal Objective

• Your commander says, Weve got a great story.
  Get it on the front page of the Times.
• (His request is a tactic)
• Your job is to ask, Why the Times, sir?
• Your key audiences may be reading the Times, but
  research shows they are more likely to
  read/watch/listen to another outlet.
• If you target that 2nd outlet, youre more likely
  to achieve your objective than just accomplish a
  tactic

62
Learn More about CA or Start Exercise

• We can either move on to the exercise or continue
  learning about examples of content analysis.
• Skip to slide 72

63
Content Analysis as ME

• Content Analysis is also used for Monitoring and
  Evaluation
• ME is a way for organizations to continually
  refine and adjust what they are doing

64
ME

• USAID usually requires a ME component to every
  contract and grant
• ME proves that the organization is actually
  accomplishing what it sets out to do.

65
Can Training Journalists Help to Improve Content

• Kosovo has a new legal system. Journalists were
  getting all of the legal terminology and court
  levels wrong in their stories. Their stories were
  undermining faith in the new system.
• IREX conducted 5 Saturdays of legal system
  training for reporters to help explain the system
  and introduce them to key people in the system.

66
Measuring Change

• Benchmark of news coverage of legal stories four
  weeks BEFORE the training.
• Time 1 coverage of news coverage of legal stories
  four weeks DURING the training.
• Time 2 news coverage of legal stories four weeks
  AFTER the training.

67
Human Trafficking

• Benchmark of news coverage of trafficking stories
  four weeks BEFORE the training.
• Time 1 coverage of news coverage of trafficking
  stories four weeks DURING the training.
• Time 2 news coverage of trafficking stories four
  weeks AFTER the training.

68
Findings
69
What Went Wrong?

• This analysis can help us to see the limitations
  of the training.
• The print journalists did not incorporate many
  aspects of the training into their stories. Next
  time, the training needs to be modified and the
  editors need to be included in the training.

70
Inflammatory Language in Iraqi Media

• We are coding four stations (Al Iraqiya, Al
  Hurra, Al Summariya, and Al Bagdadiya) for levels
  and frequency of inflammatory terms.
• We found that it is the anchor/presenter who most
  often uses the terms. We also found that they
  often use sound bites from politicians that have
  inflammatory terms.

71
So What?

• In Iraq, 230 journalists have been killed and the
  Committee to Protect Journalists notes that
  violence is directed at reporters who use or work
  at stations that use inflammatory terms in their
  coverage.
• We are bringing anchors and news directors to
  Beirut in Oct. to train them in content analysis.
  

72
Community Radio Stations Liberia

• We are using content analysis to help identify
  issues before they become conflicts.
• We are having the stations log calls and topics
  to scan for issues before violence occurs.
• Dialogue based programming may be able to diffuse
  rumors and mediate tensions.

73
How Can PAOs Use Content Analysis?

• It can be used to evaluate output and then create
  new strategic content
• Components of Excellent News

74
Practical Exercise 1

•  Please spend a few minutes identifying all of
  the components of a good news story.
•  
• What are the components of a good print story?
•   
•  
• What are the components of a good television
  story?
•  
•  
•  
• What are the components of a good radio story?

75
Developing Categories

• Deductive Category Development
• Take the individual generated components and
  identify/narrow down categories
• Inductive Category Development
• Identify categories ahead of time and apply them
  to content

76
Operationalization

• Defining the category in such detail as to make
  it clear for any reader/coder
• Examples Images of Political Candidates
• How often standing, who standing with, full body,
  face

77
Practical Exercise 2

• As a group, we need to define each content
  category component.
• We need to define a category in as much detail as
  possible so that anyone who reads our
  operationalization can see the same things we
  see. Ensures objectivity.

78
Intercoder Reliability

• Scotts Pi is calculated based on percentage of
  agreement between 2 or more coders
• Categories need to be mutually exclusive and
  exhaustive
• The unit of analysis can only be placed in one
  category. Definitions need to be complete.

79
Types of texts that can be content analyzed

• Newspapers
• Magazines
• Books
• PSAs
• News releases
• Annual reports

• Memos
• Chat rooms
• Web pages
• Transcripts of
• Radio news
• Nightly news
• (or listen to)

80
Practical Exercise 3

• Rating Scale
• Now that we know how to identify each news
  component, we need to evaluate the strength of
  its presence.
•  
• What does each number mean to you?
• Lets make 5 the number representing highest/best
  incorporation of the feature. Lets make 1 the
  number representing the weakest showing/missing
  from the story.
•  
•  

81
Application

• Practical Application of Content Analysis to News
• Refining the Categories

82
Practical Exercise 4

• Take out Practical Exercise 4 and write in the
  content categories and the groups definitions.
• Please code the Cuba story based on our content
  analysis criteria.

83
Lets See How Close We Are

• Establishing an informal intercoder reliability.
• Each participant will share their coding and
  rationale.

84
Revisiting the Categories

• How well do the operationalizations work in
  representing excellent news?
• How well does the rating system work?
• You may need to do this several times with the
  coders. Every new coder added to the team must
  be trained.

85
Using Content Analysis to Generate News Stories

• Although most often viewed as a post news
  evaluation methodology, content analysis can also
  be used to help generate news stories.
• This is a great training tool for developing JR
  PAOs and support staff.

86
Check List

• After we know the components of an effective news
  story, we can use the checklist to ensure that
  all components are in our strategic messaging.

87
Training Opportunities

• We can also content analyze the written materials
  of our subordinates and identify areas for
  improvement.
• The checklist allows supervisors to pinpoint
  areas for improvement.

88
Questions?

• Thank you.