Two Methodological Issues concerning Consensus Analysis

1
Two Methodological Issues concerning Consensus
Analysis

• John B. Gatewood
• Lehigh University
• SASci / SCCR meetings
• Feb 18-21, 2009, Las Vegas, Nevada

2
The two issues

• COUNTER-BALANCING ITEMS for consensus analysis
• Examples
• Perceptions of Christmas program in Bethlehem,
  PA( Cameron Gatewood 1990 )
• Employee perceptions of credit unions( Gatewood
  Lowe 2004-2008 )
• Identifying the MEANING OF 2ND FACTOR from
  consensus analysis
• Example
• Residents understandings of tourism in Turks and
  Caicos Islands( Gatewood Cameron 2006-2009 )

3
Counter-balancing items for consensus analysis
4
Example 1 The Christmas Program in Bethlehem,
PA (ratings for 20 adjectives)

• Entertaining
• Historic
• Boring
• Interesting
• Commercialized
• Meaningful
• Glitzy
• Nostalgic
• Serene
• Insufficient

• Old fashioned
• Ethnic
• Religious
• Tasteful
• High pressured
• Crowded
• Musical
• Small townish
• Enriching
• Authentic

5
Example 1 The Christmas Program in Bethlehem,
PA (ratings for 20 adjectives)

• Entertaining
• Historic
• Boring
• Interesting
• Commercialized
• Meaningful
• Glitzy
• Nostalgic
• Serene
• Insufficient

• Old fashioned
• Ethnic
• Religious
• Tasteful
• High pressured
• Crowded
• Musical
• Small townish
• Enriching
• Authentic

6
Original data (item means and standard
deviations)
7
Same data with negative items inverted
8
Evenly counter-balanced data(4 positive items
inverted)
9
Different consensus findings !
10
Example 2 Cultural Model of Credit Unions
(pilot vs. follow-up studies)

• PILOT STUDY (2006)
• Questionnaire developed BEFORE the cultural model
• Ex post facto, only 14 items directly relevant to
  the cultural models elements all phrased
  positively (using informants own words)
• FOLLOW-UP STUDY (2008)
• Expanded questionnaire AFTER constructing the
  cultural model
• 50 counter-balanced items relevant to the
  cultural models elements paired questions for
  each idea

11
Pilot Study
12
Follow-up Study
13
VERY different consensus findings !!
14
WHY counter-balancing makes a difference

• Responses as z-scores z ( X m ) / s
• m respondents mean across items
• s respondents st. dev. across items
• Pearson correlation coefficient r S (zx zy)
  / N
• COUNTER-BALANCING ITEMS
• increases within-respondent variances more
  undulations in each persons response-profile
• Hence, counter-balancing makes higher
  correlations among respondents mathematically
  possible (although not necessary). Note
  correlation is undefined between constants.
• induces respondents to use more of the
  response-scale ? finer gradations of responses,
  more interval-like data

15
CONCLUDING REMARKS 1

• Researchers using consensus analysis on rating
  data must be sure to counter-balance items
• Goal batteries of questions containing
  approximately equal numbers of positive and
  negative items
• How?
• INVERTING ITEMS ex post facto a data
  transformation step
• formulating PAIRED QUESTIONS in advance an
  improvement to data collection

16
Identifying the meaning of the 2nd consensus
factor
17
Preliminary reminders

• The informal method of consensus analysis
• rating or ranking data (not categorical
  responses)
• Pearson r is measure of similarity among
  response profiles no adjustments for guessing
• minimum residual factor analysis of the R x R
  correlation matrix (hence, factors may be
  correlated with one another)
• 1st FACTOR overall similarity among the
  response profiles (loadings indicate how well
  each person represents the entire sample)
• 2nd FACTOR the second largest source of
  pattern variability in the response profiles
  substantive meaning is unknown in advance

18
Example 3 Understandings of Tourism (ratings
on 119 counter-balanced items)

• SPECIAL SAMPLE (N 29)
• Individuals we interviewed during first year
• Questionnaire grown from these interviews, with
  119 counter-balanced items (1-to-5 scale)
• Same people also completed questionnaire during
  second year
• RANDOM SAMPLE (N 277)
• Stratified random sample of registered voters who
  completed questionnaire no additional
  information about them

19
Cultural consensus within Random Sample

• Weak cultural consensus with respect to the 119
  similarly-formatted cultural model items
• Random Sample (N277)
• Ratio of 1st to 2nd eigenvalues 4.515
• Mean 1st factor loading .499
• 9 negative loadings, or 3.2 of sample

20
(No Transcript)
21
Disaggregating Random Sample improves consensus
mostly
22
Cultural consensus within Special Sample

• Weak cultural consensus in this group (N29), too
• Ratio of 1st to 2nd eigenvalues 3.355
• Mean 1st factor loading .584, with 0 negative
  loadings
• Hence, use Special Sample to investigate the
  second largest source of variability (2nd factor
  accounts for 21.6 of variance in this R x R
  correlation matrix)
• Examining the 2nd factor loadings for these 29
  familiar informants, we began to see a very
  interpretable pattern

23
(No Transcript)
24
JOHNSONS HIERARCHICAL CLUSTERING (average
method) Cluster 1 Cluster 2
A A
A A A 1 A A A A A A 1 A
A A A A A A A A A A A A A A A A A 2 0
1 7 0 0 1 1 0 2 7 2 3 2 2 2 0 1 0 1 0 0 0 2 1 2
2 2 3 6 3 5 a 6 2 1 2 9 1 b 7 0 9 3 5
1 9 4 4 8 5 7 4 0 0 8 2 1 ------ - - - - - - -
- - - - - - - - - - - - - - - - - - - - -
- 0.7129 . . . . . . . XXX . . . . . . . . .
. . . . . . . . . . . 0.6934 . . . . . . . XXX
. . . . . . . . . . . . . . . . . . XXX 0.6613
. . . . . . . XXX . . . . . . . . . . . . . .
. . . XXXXX 0.6417 . . . . . . XXXXX . . . .
. . . . . . . . . . . . . XXXXX 0.6060 . . . .
. . XXXXX . . . . . . . . . . . . XXX . . .
XXXXX 0.6025 . . . . . XXXXXXX . . . . . . .
. . . . . XXX . . . XXXXX 0.5926 . . . . .
XXXXXXX . . . . . . . . . . . . XXX . .
XXXXXXX 0.5754 . . . . . XXXXXXX . . . . . .
. XXX . . . XXX . . XXXXXXX 0.5694 . . . . .
XXXXXXX . . . . . . . XXX . . . XXXXX .
XXXXXXX 0.5656 . . XXX . XXXXXXX . . . . . .
. XXX . . . XXXXX . XXXXXXX 0.5420 . . XXX .
XXXXXXX . . . . . . . XXX XXX . XXXXX .
XXXXXXX 0.5290 . . XXX . XXXXXXX . . . . . .
. XXX XXX . XXXXX XXXXXXXXX 0.5282 . . XXX .
XXXXXXX . . . . . . . XXX XXX XXXXXXX
XXXXXXXXX 0.5191 . . XXX . XXXXXXXXX . . . .
. . XXX XXX XXXXXXX XXXXXXXXX 0.5085 . . XXX .
XXXXXXXXX . . . . . . XXX XXX
XXXXXXXXXXXXXXXXX 0.4899 . . XXX . XXXXXXXXX .
. . . . . XXX XXXXXXXXXXXXXXXXXXXXX 0.4688 .
. XXX XXXXXXXXXXX . . . . . . XXX
XXXXXXXXXXXXXXXXXXXXX 0.4458 . . XXX
XXXXXXXXXXX XXX . . . . XXX XXXXXXXXXXXXXXXXXXXX
X 0.4440 . . XXX XXXXXXXXXXX XXX . . XXX XXX
XXXXXXXXXXXXXXXXXXXXX 0.4327 . . XXX
XXXXXXXXXXX XXX . . XXX XXXXXXXXXXXXXXXXXXXXXXXX
X 0.4132 . XXXXX XXXXXXXXXXX XXX . . XXX
XXXXXXXXXXXXXXXXXXXXXXXXX 0.3634 . XXXXX
XXXXXXXXXXX XXX . . XXXXXXXXXXXXXXXXXXXXXXXXXXXX
X 0.3483 . XXXXXXXXXXXXXXXXX XXX . .
XXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3380 .
XXXXXXXXXXXXXXXXXXXXX . . XXXXXXXXXXXXXXXXXXXXXX
XXXXXXX 0.3184 . XXXXXXXXXXXXXXXXXXXXX .
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX 0.3038 .
XXXXXXXXXXXXXXXXXXXXX XXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXX 0.2818 . XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXXXXXXXXXXXXXXXXXXX 0.2241
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
XXXXXXXXX
25
Subcultures within Special Sample

• Analyzing the clusters separately, consensus
  indicators go up sharply
• Cluster 1 (n12)
• Ratio of 1st to 2nd eigenvalues 7.061
• Mean 1st factor loading .640, with no negative
  loadings
• Cluster 2 (n17)
• Ratio of 1st to 2nd eigenvalues 9.838
• Mean 1st factor loading .653, with no negative
  loadings
• Conclusion there are two coherent viewpoints
  (different answer keys) in the Special Sample

26
The two viewpoints (in Special Sample)

• Based on the individuals who best represent each
  subcultural group (and taking into account the
  views expressed by them in interviews), the two
  viewpoints might be characterized as follows
• Cluster 1 Cautiously ambivalent
• Some concern about the long-term consequences of
  tourism tourism involves a trade-off between
  good and bad impacts
• Cluster 2 Pro-tourism, pro-growth
• Very positive about changes tourism has
  wroughtpro-growth and pro-development change
  is progress

27
Survey items that differentiate the two
viewpoints (in Special Sample)

• Independent-samples t-tests on the 119 cultural
  model items in questionnaire (Cluster 1 vs.
  Cluster 2)
• 47 items show statistically significant
  group-group differences at the unadjusted a .05
  level
• Conversely, the two groups did not differ
  significantly on 72 items (reason the Special
  Sample, as a whole, shows weak consensus)

28
(No Transcript)
29
The usual suspects dont explain the viewpoints

• NO difference with respect to
• Age Sex Education Household income
• How often think about tourism Speak with
  tourists
• Perceived overall financial benefit from
  tourism(Variable self family neighbors
  island country)
• Sources of information
• Almost significant contrast (a .057)
• Cluster 1 has traveled to more parts of the world
• One significant contrast (a .033)
• Cluster 2 reports more personal financial benefit
  from tourism(Variable self family)

30
Extrapolating from Special Sample

• EMPIRICAL QUESTIONIs there a similar
  viewpoint variation the same sort of
  subcultural attitudinal variation in the
  larger, Random Sample?
• PRELIMINARY OBSERVATIONOverall, response
  profiles across the whole battery of 119 items
  are very similar between the Special Sample (as a
  whole) and the Random Sample r .938
• Note Special Sample has greater variance among
  items means, but very similar pattern of
  ups-and-downs

31
Both samples response profiles are very similar
overall (r .938)
32
Extrapolating? two approaches

• 1. Profile Matching
• Compare each Random Sample respondent with the
  two subcultural response profiles (across 47
  items) from the Special Sample
• See whether these profile matching measures
  correlate with the Random Sample 2nd factor
  loadings from consensus analysis
• 2. Thematic Indices
• Construct multi-item, additive indices to measure
  different themes that seem to distinguish the
  Special Samples two viewpoints
• See whether one or more of these indices
  correlate with the 2nd factor loadings from
  consensus analysis (both samples)

33
Profile matching approach
Scatterplot Random Samples correlations with
respect to the Special
Samples two subcultural response profiles
34

• r2 r1 a computed variable from information
  depicted in the scatterplot, where r1 Pearson
  r vis-à-vis Cluster 1s response profile r2
  Pearson r vis-à-vis Cluster 2s response profile
• Thus,
• Positive values ? respondent is more similar to
  the Pro-Tourism (Cluster 2) viewpoint
• Negative values ? respondent is more similar to
  the Cautiously Ambivalent (Cluster 1) viewpoint

35
The attitudinal gradient found in the Special
Sample also exists in the Random Sample

• Correlation between the r2 r1
  pattern-matching variable and the 2nd consensus
  factor scores for the Random Sample is VERY high
  r .903
• Thus, the Random Samples second largest source
  of variation is very similar to the attitudinal
  gradient identified in the Special Sample

36
Thematic indices approach

• Candidate items were selected from all 119
  cultural model questions based on their face
  validity subsequently, winnowed by standard
  criteria of index construction using Random
  Samples data
• RESULT Six additive indices scaled to
  rangefrom 1-to-5 (1maximally negative,
  3neutral, 5maximally positive)
• Social Impacts (7 items, Cronbachs a .780)
• Heritage Optimism (5 items, Cronbachs a
  .737)
• General Pro-Tourism Outlook (7 items,
  Cronbachs a .717)
• Financial Impacts (5 items, Cronbachs a
  .704)
• Environmental Impacts (5 items, Cronbachs a
  .673)
• Orientation to Tourism Work (4 items,
  Cronbachs a .636)

37

• To our surprise (and delight), the six thematic
  indices could be combined to form a single,
  second-order index
• MacroIndex a two-stage additive index based on
  33 items, Cronbachs a .812
• Histogram of MacroIndex scores for Random Sample
  (mean 3.23)

38
MacroIndex correlates VERY highly with consensus
2nd factor loadings (both samples)

• MacroIndex scores are extremely highly correlated
  with the 2nd factor loadings from consensus
  analysis
• Random Sample (N277) r .922
• Special Sample (N29) r .975
• INTERPRETATION
• MacroIndexs 33 constituent items virtually are
  the substantive issues that underlie the second
  largest source of variation among respondents
• The attitudinal gradient first discovered in the
  Special Sample is also present (and now
  substantively identified) in the Random Sample

39

• Methodological aside
• It was only by having a Special Sample people
  we interviewed AND surveyed that we
• became aware different viewpoints existed,
• were prompted to investigate how these viewpoints
  are associated with distinguishable response
  patterns in the survey data

40
CONCLUDING REMARKS 2

• When overall consensus indicators are low or
  mixed, the 2nd factor deserves detailed
  investigation to identify the subcultural
  variations
• And, the substantive meaning of the 2nd consensus
  factor must be established anew for each dataset
• Sometimes the 2nd factor may be correlated with
  demographic-biographic variables, e.g., sex, age,
  education, income, residence, political party,
  etc.
• Sometimes not, or only weakly so
• When demographic-biographic variables arent
  working, there are at least two other ways to
  proceed
• Profile matching (requires a special sample)
• Constructing thematic indices

41
Finis ( now lets talk about this stuff ! )