Title: Perspectival Diversity and Consensus Analysis
1Perspectival Diversity and Consensus Analysis
- John Gatewood . . . . . . Lehigh University
- John Lowe . . . . . . . Cultural Analysis Group
- AAA Meetings, Philadelphia, Dec 5, 2009
2Preview
- INTRODUCTORY REMARKS
- Problem of culture-sharing (and non-sharing)
- Basic patterns of inter-informant agreement
addingperspectival diversity to the list - OUR CURRENT STUDY
- Assessing effects of different distributional
patterns on consensus analysiss key indicators - Some initial findings
- CONCLUSIONS
- Methodological lessons for researchers
- Future directions
3INTRODUCTORY REMARKS
4Problem of Culture-Sharing
- By definition, culture is socially transmitted
knowledge hence, it must be shared but
sharing is always a matter of degree - Hence, two related issues for any given cultural
domain - How much knowledge is shared? (the AVERAGE
cultural competence) - How is the knowledge socially distributed? (the
DISTRIBUTIONAL PATTERN) - KEY INSIGHT assess degree of culture-sharing by
examining patterning of inter-informant agreement
5Basic Patterns of Agreement
- Boster (1980, 1985) four basic patterns of
agreement (paraphrasing expanding) - UNIFORM agreement traditional view of culture
- RANDOM agreement free variation ? no culture
- EXPERTISE gradient experts tend to agree with
one another whereas non-experts deviate
randomly - SUBCULTURAL variation more than one school
of thought - Competing answer sets different groups,
different truths - Complementary knowledge different groups
systematically know different things - Romney, Weller Batchelder (1986) cultural
consensus theory consensus analysis
6Perspectival Diversity a 5th Pattern
- Pilot study of credit union employees (Gatewood
Lowe 2006) -
- No consensus in sample
-
- No identifiable subcultural groups
- gt Fish-scale overlappings of partial
knowledge perspectival diversity - i.e., social interaction and knowledge among the
employees was rather departmentalized their
understandings of credit unions reflected what
they needed to know to perform their own jobs,
not necessarily what might be relevant to other
people - _______________________
- Pilot studys conclusion about no
consensus turned out to be an artifact of our
failure to counter-balance items in the
questionnaire form (see Gatewood Lowe 2008)
but thats another story
7- To generalize, perspectival diversity occurs
when - All individuals have limited knowledge with
respect to a given domain and to approximately
the same degree - Each individuals range of knowledge only
partially overlaps with the ranges known by
others - And, consistent with this definition, different
geometries of perspectival diversity are
possible e.g., circular pattern, linear
pattern, taxonomic-hierarchical, overlapping
polygons on a surface, etc.
8OUR CURRENT STUDY
9- RESEARCH QUESTION
- Ceteris paribus, do the different distributional
patterns affect the key indicators of consensus
analysis? - As the average knowledge in a sample varies, do
different distributional patterns show
consensus more readily than other patterns? - Do some distributional patterns mask cultural
consensus when other patterns reveal it? - If NO ? nothing to worry about yippee!
- If YES ? distributional patterning has an
independent effect that needs to be taken into
account when interpreting results of consensus
analyses
10- ITEM FORMAT
- Counter-balanced Likert-style questions, i.e.,
6-point strongly agree to strongly disagree
response scale because these are so common in
survey research - ANALYSES
- Such data can be analyzed two ways
- INFORMAL MODEL of consensus analysis i.e.,
input to factor analysis is a Resp x Resp
correlation matrix (data treated as
interval-scale) - FORMAL MODEL of consensus analysis i.e., input
to factor analysis is a chance-corrected
agreement matrix (data treated as nominal-scale,
e.g., dichotomized responses)
11Research Design with average knowledge and
distributional pattern as manipulated variables
Distributional Patterns Key Indicators Key Indicators Key Indicators variety of other measures
Distributional Patterns Ratio of eigenvalues Mean 1st factor loading Number of negative loadings variety of other measures
Uniform-to-Random ? ? ? ?
Expertise ? ? ? ?
Subcultures ? ? ? ?
Perspectival ? ? ? ?
12Theoretical Predictions
Distributional Pattern Prediction
Uniform-to-Random ? None serves as benchmark for other patterns
Expertise Gradient ? INCREASE consensus indicators
Subcultures ? DECREASE consensus indicators
Perspectival ? ???
13Implementation
- How to experimentally manipulate key parameters
for different distributional models while holding
others constant ?? computer simulation to the
rescue ! - See Excel data-generating file Excel
findings file
14SOME INITIAL FINDINGS
15(No Transcript)
16(No Transcript)
17(No Transcript)
18(No Transcript)
19(No Transcript)
20(No Transcript)
21(No Transcript)
22(No Transcript)
23(No Transcript)
24(No Transcript)
25(No Transcript)
26Key Findings
- Distributional pattern has an independent effect
with respect to consensus indicators - w/r/to RATIO OF EIGENVALUES
- ( compared to the Uniform-to-Random model )
- Expertise patterns INCREASE the ratio
- Subcultural patterns DECREASE the ratio
- Perspectival patterns DECREASE the ratio
- w/r/to MEAN 1st FACTOR LOADING
- Distributional patterns have little effect on
this indicator, AND consensus analysis estimates
actual competence very well with one exception - Expertise (triangular) pattern INFLATES mean
competence as well as the ratio of eigenvalues
(because it violates the homogeneity of items
assumption?)
27- Expertise (rectangular) pattern
- The range of expertise about the same average
competence also makes a difference greater
range ? larger ratio of eigenvalues - Subcultural patterns
- As expected, systematic differences in sub-group
knowledge undermine consensus - By question sub-groups may still show consensus
overall, with the groups showing up on the 2nd
factor - Different answer keys just destroy consensus
- Formal consensus model (on dichotomized data)
and informal consensus model yield very similar
results
28CONCLUSIONS
29Lessons for Researchers
- Since the ratio of eigenvalues is particularly
sensitive to the distributional pattern of
knowledge, REPORT MORE than just the ratio - Minimally, include
- Ratio of 1st to 2nd eigenvalues
- Mean 1st factor loading (and st.dev. of those
loadings) - Number of negative loadings
- And, comparable guidelines should be
established for evaluating these additional
measures e.g., 0.500 for mean loading fewer
than 5 negative loadings in sample - These output statistics are necessary for more
meaningful interpretations of ones data
30- IF your data show a hefty mean 1st factor loading
but a low ratio of eigenvalues DO NOT leap to
the conclusion that either (a) subcultures exist
or (b) there is free variation in the domain - You may be dealing with a case of PERSPECTIVAL
DIVERSITY which would warrant further
investigation, such as examining the inter-person
correlation matrix and the response-profiles of
individuals one at a time to see if you can
detect a subtle social patterning to
who-knows-what
31- Try to formulate questions that are EQUALLY
DIFFICULT ( and ask lots of questions ) - Violations of Assumption 3 will inflate both the
obtained ratio of eigenvalues the mean 1st
factor loading - e.g., Expertise (triangular) pattern INFLATES
both indicators - So ex post factoif you notice that some
questions were much easier than others, then
either (a) use higher threshold criteria before
claiming the data conform to the cultural
consensus model, and/or (b) remove the very
easy questions and re-analyze
32Future Directions
- Developing additional geometries of
perspectival overlapping - Analyzing relations between a variety of measures
describing the initial Resp x Resp correlation
matrix and the key indicators from consensus
analysis - Exploring different instantiations of guessing
(binomial, truncated-normal, beta distributions) - Exploring other possible measures from the factor
analysis as predictors of culture-sharing, e.g.,
1st eigenvalue divided by sample size
33Thank you
- and we would be happy to continuetalking with
interested folksafter the session
34Scalar data analyzed via Informal Method
(vertical axis)VSDichotomized data analyzed via
Formal Method (horizontal axis)
Mean 1st factor loadings( r .969 )
Ratios of eigenvalues( r .933 )
35Real Survey Item
Real Survey Item
Simulated Item
POTENTIAL PROBLEM Frequency distributions of
itemsfrom real surveys (top panels)are more
graded than oursimulated data (lower
right) something were trying to resolve,but
not there yet