Issues of Theory, Validity and Reliability

1
Issues of Theory, Validity and Reliability
Theory is self-referential. Any claim about what
theory is, itself relies upon a particular
theory about the nature of theory. That said,
another theory may be invoked to identify
inconsistencies between what is claimed to be
theory and the theory that is more or less
explicitly engaged to make the claim. This
curse also applies to theories about how
validity and reliability are attained, or
performed.

• Hugh Willmott
• Research Professor in Organizational Analysis
• Cardiff Business School
• Home Page http//dspace.dial.pipex.com/town/clos
  e/hr22/hcwhome

2
Outline of Session

• Positivism The Legacy of Natural Science
• Issues of Validity and Reliability
• Questioning Positivism
• Alternatives to Positivism Conventionalism,
  Critical Theory, Critical Realism, Discourse
  Theory
• Conclusions

3
Positivism The Legacy of Natural Science
4
Some Key Theoretical Questions

• In what respects is the social world equivalent
  or different to the natural world?
• Whatever answer we give, it will be informed by a
  theory (e.g. a theory of what the social world is
  and what the natural world is)
• To what extent are the methods developed within
  the natural sciences transferable to the social
  sciences?
• Again, whatever answer we give, it will be
  informed by a theory (e.g. a theory about how
  (valid, reliable) ways of generating
  knowledge of the natural / social world can be
  developed
• On what basis would we determine the equivalence
  / difference of the natural and social worlds or
  the transferability of methods?
• Whatever answer we give, it will be informed by
  theory (e.g. about whether the methodology of
  determination is valid, reliable, etc).

5
Positivism
Adapted from A. Giddens (1974), ed., Positivism
and Sociology, London Heinemann, pp3-4 and A.
Thomas (2006), Research Concepts for Management
Studies, London Routledge, p 63

• Methodological procedures of natural science are
  applicable to the social world. Subjectivity,
  volition, will, reflexivity etc. do not present
  any insuperable barrier to the treatment of
  social phenomena as objects
• The social scientist is an independent observer
  of the social world equivalent to the
  pre-quantum? natural scientific observer.
  Observation language is theory-free.
• Theories are formulated (inductively) from
  observational data. Data are generated by
  specifying precisely how variables are being
  defined and measured (operationalization)See
  next slide
• Data are conceived to be independent of theory,
  so can be used to (deductively) test the adequacy
  of, competing theories
• The objective is to formulate laws or law-like
  generalizations. Guided by a concern to improve
  the capacity to predict and control
• Neutrality with respect to values. Theories
  provide explanations but cannot be used to
  justify prescriptions

6
Operationalism

• Simply stated, operationalism seeks to remove
  the ambiguity in the concepts that are typically
  embedded in scientific theories by specifying the
  operations by which they are to be measured. Once
  concepts have been operationalized, we would
  conceive of them almost exclusively in terms of
  the procedures developed for their measurement.
• Further, the doctrine of operationalism implies
  that concepts for which operational definitions
  cannot be devised should have little or no place
  in the subsequent development of scientific
  theories in a particular field of inquiry
• A. Bryman (1988), Quantity and Quality in Social
  Research, London Routledge, p. 17

7
Example The Aston StudiesDeductive approach to
comparison of organizations
Adapted from A. Bryman (1988), Quantity and
Quality in Social Research, London Routledge, p.
24
8
Issues of Validity and Reliability
9
Issues of Validity and Reliability

• Validity
• With respect to whether the measure (e.g.
  questions and scales developed to generate
  construct? data) does capture the phenomenon
  conveyed by the concept (e.g. formalization)
• Measurement generate different results, depending
  upon who is asked to complete the questionnaire
  (e.g. one representative of an organization v.
  random or structured samples of the membership)
• Reliability
• With respect to the consistency of the measure
• Is it internally coherent. For example, does
  formalization refer to one phenomenon or a
  diversity of phenomena? Is its meaning internally
  stable?
• Is the measure consistent over time (external
  reliability). Does it produce the same results?
  Link with issue of generalizability.

Bryman (1988 29) comments researchers seem to
more inclined to report that reliability tests
have been carried out. This creates the illusion
that reliability is more important, and allows
measures to be evaluated mainly in terms of this
criterion. The real reason is probably that
validity testing is highly time-consuming and can
easily turn into a major project in its own
right See also Mason (2002 187).
10
An Example of Validity Issues Studying
Everyday Views about National Government

• With regard to this research topic
• How will you demonstrate that you are indeed
  studying everyday views? And that these are views
  of national government, not about a particular
  issue or politician?
• How will you demonstrate that the views are not
  excessively influenced by a recent event?

These questions of validity involve ontological
and conceptual clarity in the sense that you will
need to be clear about what it is you mean by,
for example, everyday views or attitudes, and
they also involve relevant epistemology in that
you will need to demonstrate that your research
strategy has appropriately honed in on these
elements (Mason, 2002 188, emphases added)
11
Some Threats to External Validity (Consistency in
Generation of Data)
Adapted from Box 3.5 in P.Johnson and J. Duberley
(2000), Understanding Management Research,
London Sage, p. 51

• Selection. Findings are specific to the group
  studied
• Setting. Findings are specific to, or dependent
  upon, the particular context in which the study
  took place
• History. Specific and unique historical issues
  may determine or influence the findings
• Construct effects. Particular constructs studied
  may have a particular meaning for the group being
  investigated

12
Replication as an Audit of Validity and
Reliability Claims

• Replication is advocated as a means of checking
  for bias and testing for generalizability
• Replication relies upon explicated data
  collection procedures that can be copied by other
  researchers

Replication and Case Study Research. With regard
to multi-case design of case study research, for
example, cases may be selected using a
replication logic (see Yin, 1994). It may be
predicted that they will exhibit the same
characteristics (empirical replication) or
contrasting ones (theoretical replication). They
may serve to confirm hypotheses or to suggest
areas for revising the theory from which the
hypotheses are drawn.

• Replication per se does not resolve issues of
  validity and reliability but may raise questions
  about them

13
Validity and Reliability in Qualitative Data
Generation

• Is it reasonable or appropriate to apply notions
  of validity and reliability derived from the
  natural sciences to discipline and audit
  qualitative research? Masons response is that

given the non-standardization of many methods
for generating qualitative data, a researcher
will be unable to perform simple reliability
testsbecause the data they generate will not
take the form of a clearly standardized set of
measurements (Mason, 2002 187).
So, what is to be done? Abandon methods that
cannot comply with scientific tests of
reliability? Develop alternative criteria for
assessing the status, value and contribution of
social science?
14
Questioning Positivism
15
One Possible Response
Adapted from Mason (2002188)

• Qualitative researchers are inclined to argue
  that the rigidity and standardization of
  quantitative measures compromises claims to
  validity and that the flexibility of less
  structured methods is more capable of conveying
  the complexity and richness of the social world
  (i.e. is more valid)
• more ready to acknowledge uncertainty and
  indeterminacy? professional ideology of
  qualitative researchers?
• Qualitative researchers are inclined to argue
  that quantitative approaches are preoccupied with
  issues of reliability and ease of quantification
  as an escape from more difficult issues of
  disclosing richness and complexities (and thus
  gaining validity, in this sense)

16
One Possible Response (contd)
Adapted from Mason (2002188)

• Qualitative researchers may emphasise that they
  are no less concerned with overall questions of
  accuracy in their research practice, albeit that
  there are particular challenges associated with
  justifying claims to accuracy
• Qualitative researchers may accept an obligation
  to ensure and demonstrate to how data generation
  and analysis have not only been appropriate to
  the research questions, but also thorough,
  careful and honest (as distinct from true or
  correct terms which many qualitative
  researchers would, of course, wish to
  problematize)
• This means being able to convince others that you
  have not, for example, invented your data, or
  been careless or slipshod in its recording and/or
  analysis
• Qualitative researchers may be urged to provide
  an account of exactly how they achieved the
  degree of accuracy being claimed
• Presentation of analysis must include an
  explanation of why it is that the audience should
  believe the data to be reliable and accurate and
  how the researcher came to develop your
  particular interpretation of the data

Exercise. 1. What, if anything, do you see as
problematical or contentious in Masons
position? 2. What assumptions about the social
world (ontology) and our theory of how to gain
knowledge of it (epistemology) inform Masons
position?
17
Anyone for Triangulation?

• As an alternative, or supplement, to
  replication, triangulation the use of a variety
  of methods to study the same phenomenon is
  widely advocated as a means of gaining greater
  confidence in the results.
• Triangulation reduces dependence upon a single
  method and is said to enable a richer picture to
  be built up of the research phenomenon

• However, the logic of triangulation is based
  upon particular ontological and epistemological
  assumptions that are problematical
• It disregards how different methods may
  construct, rather than reflect, different views
  of the phenomenon
• It assumes that there is one, objective, and
  knowable social reality, and all that social
  researchers have to do, is to work out which are
  the most appropriate triangulation points to
  measure it by a view with which many
  researchers in the qualitative tradition would of
  course take issue (adapted from Mason, 2002190)

18
Standpoint and Respondent Logics of Validation

• Standpoint logic. Suggests that the personal
  background or social location of the research
  provides them with a measure of epistemological
  privilege (parallels the idea that women know
  best about womens experience or that managers,
  for example, know best about management).
• Respondent logic. Suggests that research
  subjects occupy a privileged epistemological
  position for evaluating the validity of the
  findings or the interpretation placed upon them

Mason (2002193-4) comments If you think that
your respondents do have epistemological
privilege enabling them to provide a quick fix
to the problem of interpretive validity, just
as if you think that you yourself have
epistemological privilege based on a standpoint
position, then you will need to demonstrate how
and why they (and you) have come to hold that
privilege
19
Alternatives to Positivism (1) Conventionalism
20

Challenge to Assumption of Theory-Neutral
Observation Language

• Challenges the possibility of a theory-neutral
  observation language for testing claims
  attentiveness to involvement of scientists
  particular frame of reference in making
  observations / developing theories
• Scientific accounts are not viewed as true or
  accurate descriptions of external reality but as
  representations that are taken to be true by
  members of particular epistemic communities.
• Shift of analytical focus from external reality
  to conventions that are understood to be
  productive of a specific sense of truth

Affinity of conventionalism with some
characterisations of postmodernism there is no
single discoverable true meaning, only numerous
different interpretationsthe free play of
signifiers means that they get their meaning only
from other signifiers within language and do not
refer to anything outside themselves such as an
independent realitywhat we take to be knowledge
is constructed in and through languageit is
language and the social negotiation of meaning
themselves that need to be illuminated to display
their constructive properties and processes
(Johnson and Duberley, 2000 96-7)
21
Alternatives to Positivism (2) Consensus
Habermasian Critical Theory
22
Challenge to the Assumption of Value-Freedom

• Understands knowledge to be developed in a
  context where all communication is distorted by
  relations of power/domination (e.g.
  military-industrial complex)
• Contemporary domination of instrumental
  conception of reason ends are taken-for-granted
  - naturalization of the present
• Three types of science (and associated interests
  that are constitutive of these sciences) 1.
  Empirical-analytical (prediction and control) 2.
  Historical-hermeneutic (interpretation mutual
  understanding) 3. Critical (emancipatory) see
  box below
• Argues that current scientific practice is
  dominated by 1. to the detriment of 2. and
  especially 3.
• Counter-factual of the ideal speech situation
  (free from distorted communication) provides the
  basis for exposing bias and thereby cleansing
  knowledge of relations of power/ domination

critical science seeks to free people from
overt and covert forms of domination. It unites
aspects of the empirical-analytic and the
historical-hermeneutic sciences within a project
aimed at self-reflective understanding as a
basis for emancipatory transformation (Johnson
and Duberley, 2000 120).
23
Alternatives to Positivism (3) Critical Realism
(Bhaskar)
24
Challenge to the Assumption that Data can be Used
as Basis for (deductively) generating, and
testing adequacy of competing, theories

• Insists that there is a fundamental difference
  between what exists (the intransitive dimension)
  and what we know (the transitive dimension).
• Rejects as the epistemic fallacy - the
  (empirically realist / positivist) belief that it
  is possible to know what exists.
• Also rejects the conventionalist view that
  conventions alone should be the focus of study
  (and the implied understanding that what exists
  is irrelevant to this project)
• Asserts the existence of unobservable entities or
  structures (causal mechanisms) that can be
  retroductively deduced from observations of
  empirical events
• Avoids relativism by arguing that some knowledge
  provides more compelling explanations of the
  structures or mechanisms that cause events but
  also emphasises the fallibility of this knowledge
• Causality is not conceived in terms of constant
  conjunction of (empirical) events involving
  dependent and independent variables but, rather,
  unobservable, generative mechanisms (e.g.
  magnetism patriarchy) that underlie regular
  events

25
A Key Problem for Critical Realism

• How is the claimed existence of an intransitive
  realm to be warranted?
• Invoking an intransitive realm provides a counter
  to the self-referentiality of conventionalism
  (and postmodernism) but how can it be accepted
  that the (transitive) representation of the
  intransitive is anything other than a discourse
  that provides a possible way of accounting for,
  and guiding, (scientific) knowledge production?
  Or as Johnson and Duberley, 2000 156) put it
• how does science involve socially mediated
  transitive transactions with the common
  referent an intransitive reality?
• And as Halfpenny (1994 156 cited in Johnson and
  Duberley, 2000 156)) asks, rather more
  pointedly,
• what restrictions are there upon the mechanisms
  that can be invoked as causal explanationwhy not
  demons or witches spells?
• The problem is that compelling explanations, or
  identifications of generative mechanisms, are
  formulated and assessed within particular
  discourses (the conventionalism point). The
  idea of epistemic fallacy alludes to this issue
  but, arguably, it is not fully incorporated into
  critical realism
• This critique of critical realism presumes and
  invites consideration of some kind of discourse
  theory that addresses directly the problems of
  conventionalism disclosed by critical realism.

26
Alternatives to Positivism (4) Discourse Theory
(Laclau and Mouffe)
27
Challenge to neutrality with respect to values
(1)

• Rejects assumption that knowledge can be cleansed
  of power /domination either through
  counter-factuality of ideal speech situation
  (Habermas) or through the refinement of
  retroduction to disclose causal mechanisms
  (critical realism)
• Rejects those forms of discourse theory/ analysis
  that assume that nothing exists beyond language
  or that what exists is irrelevant for social
  science as the focus is exclusively upon the
  conventions that serve to construe social reality
  in various ways
• Assumes the existence of a reality that exists
  independently of diverse efforts to represent it
  , and argues that this is recurrently disruptive
  of efforts to know it. This reality what Lacan
  terms the Real - is not a product of discourse
  rather it provides a recurrent reminder of the
  limits of discourse (including the limits of all
  scientific discourses conventionalism.discourse
  theory)
• It is possible to produce /constitute
  intersubjective knowledge of reality only through
  the development of varieties of discourse. It is
  not possible to study or know reality directly
  it is possible only to study discourse as
  constitutive of a sense of reality. The sense, or
  senses, of reality are held together
  hegemonically (through the operation of
  power/knowledge relations).

28
Challenge to neutrality with respect to values (2)

• In contrast to versions of critical realism,
  reality is not reducible to something that simply
  imposes limits upon our beliefs for example by
  demonstrating that a window is not simply
  whatever, discursively, we construct it to be but
  that its meaning and significance is formed in
  relation to its particular properties.
• Discourse theory stresses that while the
  properties of windows are not to be conflated
  with our knowledge of them, these properties are
  not self-evident but must be identified through
  discourse that is necessarily imperfect in
  communicating these properties. The sense of
  imperfection is evident in the limits of every
  attempt to provide a comprehensive account of the
  object. Each attempt is understood to be
  conditioned and sustained by an hegemonic
  exercise of power.
• In contrast to critical realism which retains the
  positivist spirit of striving to produce
  objective knowledge of the world (subject to the
  epistemic fallacy clause), Laclau and Mouffes
  discourse theory is attentive to how particular
  discourses become sufficiently hegemonic to
  transform their particularity into a (precarious
  because impossible) sense of universality such
  as the establishment and widespread acceptance of
  a general theory of windows.
• Laclau and Mouffe reject the claim that it is
  possible to differentiate between more or less
  pragmatically-adequate beliefs about the material
  world (A. Sayer (1992), Method in Social
  Science, London Routledge, p83) cited in Johnson
  and Duberley 2000 161) independently of the
  discourse that produces such differential
  evaluations. In this respect, there is an
  affinity with conventionalism but, unlike
  conventionalism, the notion of the Real is
  understood to disrupt the allure of convention
• Methodology guided principally by a
  politico-ethical commitment to the disruption of
  hegemonic knowledge rather than an aspiration to
  produce objective knowledge. This is one respect
  in which discourse theory departs from
  conventionalism. Critical realism may also
  incorporate a politico-ethical commitment to
  emancipation (e.g. Bhaskar) but does not
  radically problematize the aspirations of
  positivism.

29
Conclusions The Selection and Justification of
Approach to Study (1)
Adapted from A. Bryman (1988) Quantity and
Quality in Social Research, London Routledge, Ch
5

• Epistemological
• Selection based upon what is deemed to produce,
  or pass as, warrantable knowledge (e.g.
  positivism v. conventionalism or survey v.
  participant observation). Pledge of allegiance to
  one paradigm or methodology or another.
  Combinations produce big headache.
• So, we might differentiate qualitative and
  quantitative approaches on basis of them
  represent(ing) fundamentally different
  epistemological frameworks for conceptualizing
  the nature of knowing, social reality, and
  procedures for comprehending these phenomena
  (Filstead, 1979 45). Or we might suggest that
  there is a continuum of epistemological
  frameworks, with considerable overlap between
  some kinds of qualitative and some kinds of
  quantitative analysis
• If we follow the second suggestion, then the
  quantitative / qualitative distinction becomes
  less important than differences of ontological
  and epistemological assumptions. So, for example,
  quantitative and qualitative researchers may
  share the same leanings towards positivism or
  critical realism, etc.

30
Conclusions The Selection and Justification of
Approach to Study (2)

• Technical
• Based upon assessment of the suitability of a
  particular method for a specific research
  question or topic
• e.g. survey when information is information
  sought is specific and familiar to respondents
  participant observation when study requires
  examination of complex social relationships
• This approach brackets, or selectively uses,
  epistemological and ontological positions in
  order to do what is acceptable or seems to work
• Absence of reflection upon how acceptability or
  workability is assessed or claimed
• Vulnerability to questioning about how the design
  of the study can be warranted in relation to its
  scientific claims (philosophy of science)

31
Conclusion The Selection and Justification of
Approach to Study (3)

• Ethical
• Based more or less explicitly, or knowingly, upon
  an ethical/ political/ normative orientation to
  the topic and the research process
• e.g. feminists have tended to favour and advocate
  methods that are not seen to exemplify and
  reproduce the researcher as possessing a monopoly
  of knowledge. On this basis, conversational
  interviews and ethnography are preferred to
  surveys
• Laclau and Mouffes discourse theory is informed
  by commitment to radical democracy that is
  consistent with their negative ontology or
  ontology of lack which accepts and prompts the
  continuous challenging and renewal of hegemonised
  institutions, including science

32
Final Thoughts (1)

what happens when social science tries to
describe things that are complex, diffuse and
messy ?. The answer I will give is that it
tends to make a mess of it. This is because
simply, clear descriptions dont work if what
they are describing is not itself very coherent.
The very attempt to be clear simply increases the
mess J. Law (2004), After Method Mess in Social
Science Research, London Routledge, p.2
33
Final Thoughts (2)

• Researchers have a sense of responsibility for
  the consequences of a particular way of
  representing the words and practices of other
  peopleThis sense of responsibility can be a
  source of liberation, rather than simply an
  unwelcome burden it is now possible to write
  extraordinarily rich, and even sometimes
  extraordinarily readable, ethnographies that are
  quite open about their limitations and
  partiality, and which manage to acknowledge the
  complexity of the world, and thus the difficulty
  of rendering it through words on a page, without
  sacrificing coherence or clarity J. Spencer
  (2001), Ethnography After Post-Modernism in P.
  Atkinson et al, Handbook of Ethnography, London
  Sage, p. 450 cited in Mason, 2002 194)

34
Additional Recommended Reading

• P. Halfpenny (1992), Positivism and Sociology
  Explaining Social Life, London Allen and Unwin
• Kirk, J., Miller, M. L. (1986). Reliability and
  Validity in Qualitative Research. Beverly Hills
  Sage Publications.
• G. Burrell and G. Morgan (1979), Sociological
  Paradigms and Organizational Analysis, London
  Heinemann
• H. Willmott (2003), Organizational Theory as
  Critical Science The Case of "New
  Organizational Forms"' in C. Knudsen and H.
  Tsoukas (eds), Organization Theory as Science
  Prospects and Limitations, Oxford University
  Press
• H. Willmott (2005),'Theorizing Contemporary
  Control Some Postructuralist Responses to Some
  Critical Realist Questions', Organization, 12,5
  747-780
• P. Johnson and J. Duberley (2000), Understanding
  Management Research, London Sage (but not the
  final chapter)
• M. Alvesson and S. Deetz (1990), Doing Critical
  Management Research, London Sage