How to Interpret Aggregated Survey Data:

1
How to Interpret Aggregated Survey Data The
Ecological Fallacy Problem
Chris Welzel International University Bremen
(IUB) c.welzel_at_iu-bremen.de
Canadian Humanities and Social Science Annual
Meeting, Halifax / Canada (05 / 2003)
2
The Problem
With the increasing proliferation of
cross-national survey data, such as the WVS, the
CSES, the ESS, the Latinobarometer and other
regional barometers, aggregated survey data
become an increasingly important source in
comparative research. At the same time,
methodological reservations against aggregated
survey data become also more widespread.
3
Classical and Recent References

• Robinson, William S. 1950. Ecological
  Correlations and the Behavior of Individuals.
  American Sociological Review 15 351-7.
• Przeworski, Adam Henry Teune. 1970. The Logic
  of Comparative Social Inquiry, New York Wiley
  (chapter 3).
• Seligson, Mitchell A. 2002. Renaissance of
  Political Culture or the Renaissance of the
  Ecological Fallacy. Comparative Politics 34 (3)
  273-92.
• Inglehart, Ronald Christian Welzel. 2003.
  Democratic Institutions and Political Culture
  Misconceptions in Addressing the Ecological
  Fallacy. Comparative Politics 35 (1),
  forthcoming.
• Welzel, Christian, Ronald Inglehart Hans-Dieter
  Klingemann. 2003. The Theory of Human
  Development A Cross-Cultural Analysis. European
  Journal of Political Research 42 (2) 341-80.

4
The Flaw
Przeworski Teune (1970) An aggregate
correlation that is not reflected at
the individual level within each nation is
spurious.
Seligsons (2002) correct application of the
flaw
The association between civic culture attributes
and stable democracy that Inglehart and his
co-authors have demonstrated at the
aggregate-level is spurious because at the
individual-level within nations there is no
consistent relationship between these civic
culture attitudes and support for democracy.
Hadeniuss (2003) correct application of the
same flaw
The syndrome of self-expression values that
Inglehart, Welzel and others examine at the
aggregate-level is an artifact of aggregation
because this syndrome does not show the
same coherence at the individual-level within
nations.
5
What did Robinson (1950) really warn against?
Remember the example with Afro-American
immigrants and support for segregationist
positions There was a positive correlation
between the proportion of immigrants and support
for segregationist positions at
the district-level, but at the individual-level
the correlation between the status of an
immigrant and support of segregationist positions
was negative.
Based on this finding, Robinson warned against
concluding that an existing relationship at the
aggregate-level must also be present at the
individual-level (and vice versa). Otherwise,
one may fall victim to an unwarranted
cross-level inference (in either the form of an
ecological or individualistic fallacy).
Whether an existing relationship is an artifact
of another relationship can only be decided at
the level of analysis at which this relationship
exists, for instance by controlling for omitted
influences at this level of analysis.
Hence, deciding whether an aggregate-level
relationship is spurious by testing if it
exists in similar direction and strength at the
individual-level involves an unwarranted
cross-level inference.
6
Yet, the impossibility of cross-level inferences
does not save us from the necessity of looking
at a relationship at different levels of
analysis.
For although it is true that one cannot conclude
from a relationship at one level of analyses to
the existence of the same relationship at
another level of analyses, it is also true that
the ways in which relationships can differ
between different levels of analyses, provide
meaningful insights into the nature of the
social mechanisms shaping such cross- level
differences.
In other words, each particular way in which
relationships differ between the levels of
analyses represents a specific social mechanism
that explains these differences.
7
Three ideal-typical ways in which the same
relationship can differ between the
aggregate-level and the individual-level
(1) A significantly positive relationship at the
aggregate-level is significant at the
individual-level as well but operates into the
opposite direction Opposite Sign!
(2) A significantly positive relationship at the
aggregate-level is not significant at the
individual-level, regardless of its
sign Insignificance!
(3) A significantly positive relationship at the
aggregate-level is significant at the
individual-level and operates into the
same direction but is considerably weaker at this
level Weakness!
8
Case (1) Opposite Sign
Example Consider the case that increasing crime
rates at the district-level correlate strongly
positively with the percentage of people
supporting hard punishment. Would one expect this
to be the case because criminals show stronger
support for hard punishment? Probably not! At the
individual-level within districts one would
plausibly assume to find a negative correlation
between the status of a criminal and support for
punishment. Would this negative correlation rule
out crime as a cause for growing support of hard
punishment at the district-level? Certainly not,
because increasing crime causes the majority on
non-Criminals to be more supportive of hard
punishment.
Explanation of the social mechanism Increases in
a certain minority (e.g., criminals) cause a
defense reaction (e.g., hard punishment) among
the majority not belonging to this minority, such
that increases in the minority are related with
corresponding increases in the defense reaction
among the majority, producing a positive
aggregate-level relationship. At the
individual-level, however, there is a negative
relationship between the minority attribute and
the defense reaction because the defense reaction
only occurs among non-carriers of that attribute.
Only if the minority would itself become a
majority, would the negative relation at the
individual-level switch into a negative
relationship at the aggregate-level. Hence,
opposite signs reflect a COMPOSITION EFFECT that
can occur under majoritarian defense reactions
against increasing minorities.
9
Case (2) Insignificance
Example Consider the case that increasing
unemployment rates at the district-level
correlate strongly positively with the percentage
of people supporting extremist positions, such
that districts with higher unemployment rates
show larger proportions of people voting for
extremist parties. At the individual-level within
districts, however, one might see absolutely no
difference in the propensity of unemployed and
employed citizens to vote for extremists. In this
case we would have no significant correlation
between unemployment and extremism at the
individual-level, despite the positive
correlation at the district-level. How can this
be possible?
Explanation of the social mechanism The solution
of such a puzzle is that the critical attribute
(e.g., unemployment) is not important as a
personal property but as a property of ones
social context. In other words, people do perhaps
not become more extremist because they are
themselves unemployed but because they consider
the risen unemployment in their social context as
the symptom of a crisis that requires radical
solutions. In this case, the general threat
perception linked with unemployment ratios would
cause both unemployed and employed people to be
more radical, such that within districts no
difference in political extremism between
employed and unemployed people would be visible,
although districts with higher unemployment still
would show higher levels of political extremism.
Hence, a significant relationship at the
aggregate-level is insignificant at the
individual-level, if a certain attribute acts on
a certain behavior as a CONTEXTUAL EFFECT (and
not as a personal property).
10
Case (3) Weakness
Example The linkage between the attitudes and
behaviors that generate the syndrome of
emancipative values--namely liberty esteem,
tolerance, trust and civilian protest
activitiesis considerably weaker at the
individual-level than at the aggregate-level.
11
Example for weak individual-level linkages and
strong aggregate-level linkages The
Syndrome of Emanci- pative Values
12
Case (3) Weakness
Example The linkage between the attitudes and
behaviors that generate the syndrome of
emancipative values--such as liberty esteem,
tolerance, trust and civilian protest
activitiesis considerably weaker at the
individual-level than at the aggregate-level.
Explanation of the social mechanism This is the
result of the combination of two factors CENTRAL
TENDENCY EFFECTS and the UNCERTAINTY RANGES
inherent in probabilistic relationships between
societal variables. More precisely, the
populations central tendencies tend to condense
variable distributions into the uncertainty
ranges within which the relations between these
variables do not systematically occur. However,
across populations with disparate central
tendencies the relations become fully visible
because variance/covariance structures exceed the
uncertainty ranges.
13
Example of an Uncertainty Range
14
Population-bound Central Tendencies
Concentrated Distributions within Populations
15
Population-bound Central Tendencies Disperse
Distributions between Populations
16
Concentration and Dispersal
17
Example The Syndrome of Emanci- pative Values
Weakest linkage at the individual- level within
nations
Stronger linkage at the pooled individual- level
Strongest linkage at the aggregate
cross- national level
18
An example showing how much sense it makes to
look at different levels of analyses.
19
The differing relevance of support for
democracy and
emancipative values
20
Explaining Individual-level Support for Democracy
21
Cross-cultural Comparability The Question of
Equivalence
22
Postmaterialistic Liberty Aspirations
23
Cross-cultural Comparability Similar Linkage to
Civilian Protest
24
Cross-cultural Comparability Similar Linkage to
Tolerance
25
The End?
No, because the WVS is not saved from the needs
of methodological improvement!
One possibility is the implementation of
vignettes to provide anchors for cross-cultural
comparisons, as proposed by King et al. (2003).