GUIDELINES FOR VALID CAUSAL RELATONS: EXTERNAL VALIDITY

1
GUIDELINES FOR VALID CAUSAL RELATONS EXTERNAL
VALIDITY

• Mike Shields
• Michigan State University

2
OUTLINE

• Concepts and variables
• Linearity
• Causal-model form
• Directionality
• Level of analysis
• MA as IV or DV
• Cross-level models
• Luft and Shields (AOS 2003)

3
1. CONCEPTS AND VARIABLES

• 2 types of partially shared meanings
• Some variables are from practice (e.g., ABC, BSC)
  and some are social-science theory constructs
• Formalization, informativeness, role ambiguity
• How disentangle possible multiple meanings of
  practice-defined variables like ABC and BSC?
• Do practice-defined and social-science constructs
  have the same meaning?
• How, if at all, are ABC and BSC related to
  aggregation, informativeness, timeliness,
  uncertainty, visibility of power relations?

4
GUIDELINES FOR DEFINING VARIABLES

• If a practice-defined variable is used, then
  clearly define its underlying theoretical
  properties not only those that are of
  particular interest in the current study, but
  also other properties that the practice-defined
  variable is likely to possess.
• If a practice-defined variable can represent
  multiple theoretical properties, then gather
  evidence that identifies their separate causes
  and effects.
• If the theoretical property of interest belongs
  to only a definable subset of instances of the
  practice-defined variable (e.g., only some ABC
  systems), then state this limitation explicitly.

5
CONTINUED

• Different theories define their variables with
  differing degrees of broadness
• Research question and theory determine
  appropriate breath of definition How specific
  are concepts and theory?
• Uncertainty? Task and environmental uncertainty?
  Competitor, regulatory or consumer uncertainty?
• Too broad includes noise in cause-effect relation
• More likely effects other than predicted effects
  will be detected and wrongly interpreted
• Too narrow captures only part of cause-effect
  relation
• Less likely the expected effect will be detected
• Guideline
• 4. A variable definition should not include
  content irrelevant to the research question and
  theory employed or exclude relevant content.

6
2. LINEARITY

• Theories predict curvilinear relations but few
  studies predict them
• Linearize by range of evidence (typical, not
  extreme cases) and transforming data
• Need extremes to test boundary conditions
• Practice wants to know not just that more/less MA
  is better, but how much is optimal
• Guidelines
• 5. If theory predicts nonlinearities in the
  relation examined, then consider the value of
  capturing nonlinearities in the study.
• 6. If a linear model is used for the sake of
  simplicity, then be explicit about the resulting
  limitations.

7
3. CAUSAL-MODEL FORM

• Consider the participative budgeting
  satisfaction relation
• Studies predict and find that the relation is
  additive, intervening-variable,
  independent-variable interaction, and
  moderator-variable interaction
• Because some of these causal-model forms are in
  conflict, not all of them can be valid
  representations of this relation
• Some of these causal-model forms are in conflict
  and some are compatible

8
ADDITIVE, INTERACTION AND INTERVENING MODELS

• Does it matter which model is used?
• Are these models in conflict or are they
  compatible?

9
ADDITIVE VS INTERVENING VARIABLE

• Additive model that predicts X1 ? Y is not in
  conflict with an intervening variable model that
  predicts X1 ? X2 ? Y
• Intervening variable model provides supplementary
  information about the X1-Y relation can can help
  explain week X1-Y results
• Additive model that predicts X1, X2 ? Y is in
  conflict with an intervening variable model that
  predicts X1 ? X2 ? Y
• If the intervening variable model is valid, then
  using an additive model can show that neither X
  affects Y because of multicollinearity between X1
  and X2
• If the additive model is valid, then using an
  intervening variable model with no direct X1-Y
  path can show no effect of X1 on Y because there
  is not effect of X1 on X2 even though the X1-Y
  relation is strong

10
ADDITIVE VS INTERACTON

• If a relation is interactive the effect of X1
  on Y is conditional on X2 and vice versa but an
  additive model is used (X1, X2 ? Y)
• If X2 is held constant, then the conclusion about
  X1 ? Y is valid only at the level of X2
• If X2 is omitted or included as an additive
  variable, then the detected effect of X1 on Y is
  a weighted average of the different X1 effects
  that occur at different levels of X2
• If a relation is an ordinal interaction, then
  using an additive model affects the estimated
  magnitude of the effect of X on Y
• If a relation is a disordinal interaction, then
  using an additive model affects the estimated
  magnitude and possibly the sign of the X-Y
  relation

11
INTERVENING VARIABLE VS INTERACTION

• Consider a setting in which X1 ? X2 and X1 by X2
  ? Y can be represented by 2 models
• Intervening variable X1 ? X2 ? Y
• Interaction X1-Y depends on X2 and X2-Y depends
  on X1
• Testing both models is problematic
• If the X1-X2 relation is strong, then there will
  be insufficient variation in the sample to
  provide a powerful test of the interaction

12
INTERACTING INDEPENDENT- AND MODERATOR-VARIABLE
MODELS

• An IV has a causal influence on the DV
• A moderator variable influences the IV-DV
  relation, but is not associated with either the
  IV or DV
• Important practice and theory implications
  whether an interaction involves IVs only or IVs
  and MVs

13
GUIDELINES

• If the causal model proposed is additive, then
  indicate both the reasons for assuming there are
  no important intervening-variable or interaction
  relations and the consequences of omitting these
  relations if they exist.
• If the causal model proposed is conditional, then
  indicate the type of conditionality (intervening
  versus interacting).
• For interaction models, indicate whether the
  interaction is ordinal or disordinal.
• For interaction models, indicate whether the
  interaction involves independent variables only
  or independent and moderator variables.

14
4. DIRECTIONALITY

• Does budget goal difficulty influence performance
  or vice versa?
• Cross-sectional studies (Map A)
• budget goal difficulty ? performance
• A study with 2 points in time (Map B)
• past performance ? current budget goal difficulty
• A study of ratchet systems with 3 time points
  (Map F)
• performance ? budget goal difficulty ?
  performance
• Answers to these questions depend on answers to 2
  questions about time length
• What is the time frame of the study over how
  long a time period and at what intervals within
  that period should evidence be collected?
• What is the causal interval of the relation how
  long does it take for a change in the IV to cause
  a change in the DV?

15
CONTINUED

• For bidirectional relations, common for research
  to assume unidirectionality
• When do so, usually treat slower changing
  variable as exogenous (IV) because its response
  to changes in the other variables is too slow to
  be captured within the studys time frame
• Consider MA and national culture
• NC ? MA has a shorted causal interval than MA ?
  NC
• Therefore, most studies treat NC as the IV and MA
  as the DV
• If changes in NC have had time to cause changes
  in MA but changes in MA have not had time to
  cause changes in NC, then a unidirectional model
  of NC ? MA is valid

16
TYPES OF BIDIRECTIONALITY

• Bidirectional model is required if the interest
  is on the slower effect (variable with the longer
  causal interval) or if both effects have similar
  causal intervals
• Cyclical model is valid if the causal interval of
  the relation and time frame are matched such that
  causality goes in one direction for one time
  period and the other direction for the next time
  period
• Reciprocal model is valid if the causal
  influences are simultaneous or causal intervals
  are shorter than the intervals at which evidence
  is collected so that influences in both
  directions are captured

17
GUIDELINES FOR DIRECTIONALITY

• If unidirectional causality is assumed, then
  indicate the reasons for excluding
  bidirectionality.
• Align the time frame of the study (length and
  frequency of evidence collection) and the causal
  interval (the time required for the cause
  examined in the study to have an effect).

18
5. LEVEL OF ANALYSIS

• What is the level of analysis?
• Is participative budgeting at the individual,
  subunit, or organization level of analysis? Map
  A vs. B?
• For single-level studies, need to align
• Level of theory What is being explained?
• Level of variable measurement What is the source
  of the evidence?
• Level of data analysis What is treated as an
  independent datum for analysis purposes?

19
MULTIPLE LEVELS OF ANALYSIS

• Effects on performance from different levels can
  be additive
• Industry market structure ? industry component of
  managers performance
• Organizational strategy ? organizational
  component of managers performance
• Subunit budget ? subunit component of managers
  performance
• Individual skill ? individual component of
  managers performance
• Additive effects is not cross-level model because
  a variable defined at one level cannot
  systematically affect a variable defined at
  another level but can add noise to measurements
  of variables at other levels
• Use hierarchical or nested models to partial out
  the additive effects at different levels either
  to remove noise if some levels are not of
  interest or to identify the multiple-level
  effects separately

20
CROSS-LEVEL MODELS

• Must be interactive causal models with an
  interacting IV or MV at the level of the DV
• Organizational MA can affect individual
  performance only if individuals are different on
  some characteristic (knowledge, preferences) that
  causes them to respond differently to
  organization MA
• Cross-level models can be top-down (as most are
  due to exogenizing the slower changing variable,
  which tends to be a variable at a higher level)
  and/or bottom-up

21
GUIDELINES FOR MULTI-LEVEL MODELS

• Indicate whether the variable of interest varies
  across individuals, organizational subunits,
  organizations, or beyond-organization entities
  like markets and societies.
• Align the level of theory (what is being
  explained), level of variable measurement (source
  of evidence), and level of data analysis (unit of
  data).
• If the theoretical variables at multiple levels
  affect the observed measures, then separate the
  effects from multiple levels.
• If cross-level effects are proposed, then use an
  interaction causal-model form, with at least one
  interacting independent or moderator variable at
  the level of the dependent variable.
• If the variation of interest in a variable is
  variation in its value relative to a subset of
  other values in the sample, then use an
  individual-within-group-level model.

22
6. MA AS IV OR DVLINKING CAUSES AND EFFECTS

• Some studies explain only causes of MA and others
  explain only effects of MA
• Questions arise about whether the explanations of
  cause are consistent with explanations of effect
• If the assumptions about performance effects used
  to explain causes are correct, then causal-model
  forms should be consistent across studies of MAs
  causes and effects

23
STUDIES OF CAUSES AND EFFECTS ARE SOMETIMES IN
CONFLICT

• A study shows that competition has an additive
  effect on MA
• It assumes MA is used to increase performance as
  any type of competition increases
• conflicts with
• Another study in which an interaction model
  predicts that the performance effects of MA are
  conditional on the type of competition (price or
  quantity)
• 3 possible explanations for such conflicts
• Across-study differences in the meaning of
  similar variables
• Across-study differences in the level of
  analysis Causes and effects vary with level
• Assumptions about the performance effects of MA
  that explain its causes are not correct (e.g., MA
  is performance beneficial for only some kinds of
  competition)
• This explanation is controversial because
  economics-based research excludes it because
  economics assumes organizations will only use
  practices that are optimal, thus there is no
  performance effect of MA

24
CAUSES, EFFECTS AND EQUILIBRIUM

• Economics assumes equilibrium
• Explanations of causes of MA are explanations of
  why MA is an equilibrium solution to the economic
  problem of performance maximization
• If MA is an equilibrium solution, then it is
  possible to provide evidence for explanations of
  its causes but not its performance effects
• If all organizations use the MA that is optimal
  for them, then they would all have the same
  performance, controlling for the effects of other
  variables, thus there is no variance in
  performance due to MA

25
CONTINUED

• Psychology and sociology do not assume
  equilibrium
• Assume that causes and effects of MA can differ
• Assumes that MA can be used for reasons other
  than performance maximization, or if performance
  maximization is the goal then variables like
  judgments and decisions can cause errors that do
  not result in MA maximizing performance
• Contingency theory assumes that organizations use
  the MA that is best for them, but that the
  process of adapting to a new equilibrium can take
  a long time
• During the adjustment process, it is possible to
  test for performance effects as different
  organizations are at different stages of
  converging on a new equilibrium

26
CONTINUED

• Critical issue in determining whether MA (and
  performance) is an IV or DV is whether an
  equilibrium condition exists or there is
  convergence on an equilibrium
• Research indicates that the dynamics of many
  complex systems result in non-equilibrium
  behavior
• cyclical or chaotic behavior
• Equilibrium behavior is less likely as the length
  of the causal interval increases, time lags
  increase, and relations are curvilinear and
  indirect
• Causal intervals are important in choosing the
  valid causal model form and whether MA should be
  an IV or DV

27
7. CROSS-LEVEL MODELS

• Individuals decisions are affected by
  organizational MA
• Design of organizational MA requires individuals
  to make decisions about the design

28
MA AS IV AND DVCONSTRUCTING ONE MAP OF MA
PRACTICE

• Unlikely that a study will include this model,
  but studies should be conducted that allow their
  theory-consistent evidence to construct maps that
  contain this model of MA as IV and DV
• How do we combine 275 studies into 1 map?
• Consider our 3 questions and 17 guidelines
• What variables are researched?
• What is the shape and direction of explanatory
  links?
• What is the level of analysis?

29
BIBLIOGRAPHY

• GENERAL
• Campbell, D. and J. Stanley. 1963. Experimental
  and Quasi-Experimental Designs for Research (Rand
  McNally).
• Kerlinger, F. 1985. Foundations of Behavioral
  Research (Harcourt Brace).
• Nunnally, J. and I. Bernstein. 1994. Psychometric
  Theory (McGraw Hill).
• Rosenthal, R. and R. Rosnow.1969. Artifact in
  Behavioral Research (Academic Press).
• Runkel, P. and J. McGrath. 1972. Research on
  Human Behavior A Systematic Guide to Method
  (Holt, Rhinehart and Winston).
• ACCOUNTING
• Birnberg, J., M. Shields, and M. Young. 1990.
  "The Case for Multiple Methods in Empirical
  Management Accounting Research," Journal of
  Management Accounting Research, pp. 33-66.
• Libby, R. R. Bloomfield, and M. Nelson. 2002,
  "Experimental Research in Financial Accounting,"
  Accounting, Organizations and Society, pp.
  775-810.
• Luft, J. and M. Shields. 2003. "Mapping
  Management Accounting Graphics and Guidelines
  for Theory-Consistent Empirical Research,"
  Accounting, Organizations and Society, pp.
  169-249.