Research quality and multimethod research

1
Research qualityandmultimethod research

• Module 1
• Communication Theories and Methods
• Session 8
• Kim Schrøder

2
The credibility of empirical research

• See handout with brief definitions and agendas
  reliability, validity, generalizability.
• Exercise (handout) the communicative validity of
  a case study about music videos.

3
(No Transcript)
4
The Madonna study and the credibility criteria
discussion points

• University undergraduates are a convenient, not a
  representative sample of the ethnic groups
  (representativeness)
• Too few respondents to warrant generalization
  (representativeness)
• Intercoder agreement of categorization acceptable
  (reliability)
• Classroom not an ideal setting for exploring
  leisure time activities (validity)
• Medium of writing does not enable respondents to
  comfortably express their feelings (validity)

5
Madonna study credibility criteria (cont.)

• The time of research should be closer to the time
  of video release (validity)
• Interpretations are speculative, maybe prejudiced
  by subjectivity of white researchers
  (reliability)
• Interpretations should be validated by a
  follow-up focus group study, probing into the
  connection between sociocultural background and
  video experience (validity)

6
(No Transcript)
7
Multimethod research

• The reasons for methodological innovation
• Recognizing the inherent limitations of
  single-method investigation (next slide)
• Desire for greater relevance and explanatory
  power
• Perfecting our work in the service of greater
  'truthlikeness'

8
Strong and weak propertiesof single methods

• "Quantitative observations provide a high level
  of measurement precision (reliability) and
  statistical power (representativeness,
  generalization), while qualitative observations
  provide greater depth of information about how
  people perceive events in the context of the
  actual situations in which they occur (validity)"
  Frey et al. 199199).

9
Complementary strengths and weaknesses

• Friessen Punie (1998)
• Quant. "tell us a little about a lot of people"
• Qual. "tell us a lot about a few people"
• Multimethod/triangulation ambition
• tell us a lot (or at least more) about a lot of
  people
• compensating for each other's weaknesses towards
  a better insight

10
Credibility-enhancing initiativeswithin the camps

• Self-critique in survey research (UG as ex.)
• Reliability misunder-standing of question-naires
  (Roe)(Stax)
• Validity shallow findings, superficiality, out
  of context (Gantz)
• Remedy supplement with depth interviews (Gantz)

• Self-critique in reception research
• Reliability falling for 'the typical example' in
  the data (Höijer)
• Reliability opaqueness of data interpretation
  (Bergman)
• Remedy adopt rigid (Höijer) and transparent
  (Liebes) analytical procedures

11
Qualitative generalization a contested issue

• Against qual. generalization ('analytical')
• Radical constructionism "Generalizations are
  necessarily violations to the concrete
  specificity of all unique micro-situations" (Ang,
  Hermes)
• Multi-dimensional social reality should not be
  reduced

• For qual. generalization ('analytical') (Halkier)
• Obligation to generalize prerequisite for being
  heard in policy-defining contexts
• Reductionism can be counteracted by supplementary
  'thick description'
• Quantification is better than pseudo-quantificatio
  n!

12
Triangulation

• Combining the methodological approaches in order
  to compensate for their reciprocal weaknesses
• Combination within the paradigms
• Combination between paradigms
• 'Auxiliary' triangulation one method is used in
  order to facilitate the use of another (qual.
  pilot before quant. survey, or vice versa)

13
Primary triangulation

• Combining equal methods using different methods
  for obtaining primary data
• Definition The sequential execution of two or
  more primary studies of the same communicative
  phenomenon using different methods (Res. Aud.
  356)
• Exploiting respective strong aspects so as to
  cancel out respective weak aspects
• Case study Friessen Punie (1998)

14
The positivist delusion about triangulation

• "Using both types of observation also provides a
  way of assessing the accuracy of the findings
  from one operational procedure by comparing it
  with a different operational procedure. If the
  findings support each other, both procedures are
  corroborated. If the findings are different,
  however, this does not necessarily mean that the
  data are questionable. The difference could be a
  result of the types of date that are acquired
  through quantitative and qualitative observation"
  (Frey et al. 199199)

15
Positivist delusions (cont.)

• A logic of having your cake and eating it!
• The lens metaphor of social research we should
  expect different methods to produce different
  results, not sameness
• Qual. a complexifying lens of lifeworld
  discourses
• Quant. a homogenizing lens of pre-defined
  categories
• Kvale Triangulation "contains assumptions of a
  fixed point or object that can be triangulated"
  (p. 243-4). (fixed pointtruth)
• The challenge to understand how to relate the
  different findings from different methods

16
Complementarity of methods?

• "Different methodologies may be best suited to
  examine different aspects of a research question,
  and not necessarily in the same empirical domain"
  (Jensen 2002272).
• Example 3 studies of 3 different aspects of
  'national identity' (semiotic, survey,
  ethnography)
• 'Complementarity' each study provides one piece
  for "a sociocultural jigsaw puzzle whose full
  picture will eternally elude us" (Res.Aud.360).

17
Beyond triangulation?

• Triangulation is a strategy that really requires
  no-one to change fundamentally each method
  enters the joint design fully intact
• I'll do my thing, you'll do yours!
• From juxtaposition to integration of methods?

18
An integrative multimethod approach?

• A research design that incorporates all three
  quality criteria simultaneously?
• Q-methodology
• Statistical factor analysis (typology)
• Think-aloud analysis (process monitoring)
• Depth-interview (discursive reflection)
• Combining the best of both worlds another
  methodological delusion?