Metaanalysis and systematic reviewsin qualitative research: Mission impossible

1
Meta-analysis and systematic reviews in
qualitative research Mission impossible?
Ellen MacEachen (Institute for Work Health,
University of Toronto) Scott Reeves (Li Ka Shing
Knowledge Institute, University of Toronto)

• Qualitative Inquiry Group Seminar
• University of Toronto, March 27, 2009

2
Overview 4 parts

• 1. Where did systematic reviews come from why
  do we do them?
• 2. Brief overview of the systematic review
  process
• 3. Cases that draw out systematic review issues
• Case I (Scott)
• Case II (Ellen)
• 4. Reflections on the conduct and usefulness of
  systematic reviews of qualitative studies

3
1. What are systematic reviews?

• Positivistic roots / standardization
• Synthesis of primary studies
• Multiple check points / dual reviewers /
  transparency
• Assess quality of evidence (only include
  rigorous studies)
• Inference through statistical analysis
• Newer Qual types (meta-ethnography )

4
Why do SRs?

• Useful
• Knowledge translation (policymakers/
  practitioners)
• Initial entry into field (academics)
• Help define field (academics)
• Outline areas of future research (acad/policy)
• Grant applications (academics)

5
2. Brief overview of the systematic review
process

• A walk through the recipe followed for
  systematic reviews

6
(No Transcript)
7
What databases will be searched?
What keywords are used in the search?
What is the scope of the search? Languages? Dates?
What stakeholders are involved and at what stages?
Medline Total 1620
EMBASE Total 1683
CINAHL Total 443
PsycINFO Total 1174
Sociological Abstracts Total 279
ASSIA Total 74
ABI Inform Total 381
EconLit Total 58
Business Source Premier Total 585
Content Experts Refs Total 840
How will we find papers not listed on databases?
Other Total 240
Quality control process for what is being
included/excluded
Step 1 Library Search
What articles are included/excluded?
Step 2 Study Relevance
Excluded at this level n 4256
Studies considered for QA (qualitative,
quantitative and mixed methods (n 609 )
Supplemental articles from reference list n 63
Two reviewers per paper, consensus approach
Step 3 Division of QNT/QL Studies
Excluded at this level n 20
Excluded at this level n 13
How will we assess quality?
Two reviewers per paper at this stage, consensus
approach
Step 4 Quality Appraisal
What data will we extract?
Studies passing quality appraisal n 5
Studies passing quality appraisal n 14
Step 5 Data Extraction
Step 6 Finding Synthesis
n 19
8
3. SR cases

• Case I Making decisions about how to
  construct systematic reviews the early days
• Interprofessional education (Scott)
• Case II Struggles with the paradigm of
  systematic reviews later days
• Work health in small businesses (Ellen)

9
Interprofessional education (IPE)

• Case I Making
• decisions about how
• to construct
• systematic reviews
• the early days

10
Mixed method studies Quality assessment, data
extraction, synthesis
Qualitative studies Quality assessment, data
extraction, synthesis
11

• Context information
• Policymaker demands for IPE evidence (early
  1990s)
• SR political response
• In dark about SR process new activity
• Side line work (enthusiasm)

12

• SR processes
• Inclusive approach to review team 9 members
• Conceptually inclusive (implicit/explicit IPE)
• Methodologically inclusive (qual/quan)
• Open stance seeking guidelines/standards
• Pragmatic trial error
• Abstraction pre-determined categories

13

• Thinking about Quality
• quality of study (appropriate design,
  sampling, recruitment, validity, reliability)
• quality of information (good contextual info,
  explicit rationale, clear research questions,
  clear results)
• (e.g. CASP, EPPI Centre, Popay et al 1998)

14

• Large and small SRs
• Big was good 107 studies
• Practitioners/policymakers - larger numbers
  (width)
• Small was better - 21 studies
• Researchers small numbers (higher quality)

15

• Some reflections
• Lots of discussion/debate create parameters
  first IPE review
• Proceed with caution
• One step forward, two back
• Gradual movement from inclusion to exclusion

16

• Some reflections
• Different types of qualitative research difficult
  to synthesize
• Qual / Quan / mixed methods more problems
• Quality assessment best effort (pragmatics)
• SR team dynamics

17
Case II Systematic review on work and health in
small businesses (Ellen)

• Struggles with the paradigm of systematic reviews
• 2 parts
• qualitative review
• mixed method review

18
(No Transcript)
19
Struggles in a Qualitative review

• How to observe the broad focus that can be taken
  with qualitative research while also subjecting
  it to the recipe of a SR process
• The SR process is systematic, not exploratory.
• Pre-defined parameters

20
Struggles in a Qualitative review

• How to synthesize the findings insights of
  Qualitative studies while also preserving their
  context and theory
• The systematic review extracts findings to
  answer a specific question.
• What insights do we end up with? What do we
  gain/lose?

21
Struggles in a Qualitative review

• What is being counted/considered in a qualitative
  systematic review?
• Varying approaches to data extraction.
• pre-set-findings categories?
• grounded theory process?

22
Struggles in a Qualitative review

• What counts as a qualitative study?
• Studies can self-identify as qualitative but
  not use recognizable qualitative methods.
• E.g. Participatory studies
• Our criteria Some qualitative data some
  qualitative analysis of it

23
Struggles in a Qualitative review

• How to handle interesting data from studies that
  dont meet quality criteria?
• Nuggets Pawson (2006).
• Descriptions of process that are a nugget of
  gold for the question being asked.

24
Struggles in a Mixed Method review

• How to achieve a steady process for both
  Qualitative Quantitative sides of the review
• Quantitative team-- focused on definitions,
  outcomes
• Qualitative team--no fixed categories for
  outcomes.
• Qual quan work happens at different
  times--affects possibilities for concurrent
  synergy.

25
Struggles in a Mixed Method review

• Theoretical differences about conflict of
  interest during the quality assessment process
• Qualitative team--sensitive to social/power
  relations
• heated issue, agreed to disagree.

26
Struggles in a Mixed Method review

• Differences about levels of evidence
• QN additive approach
• E.g. "best evidence synthesis guidelines.
• How much evidence is there that this has an
  effect?
• QL studiesnot about effectiveness
• Some additive approach e.g. concepts in 3
  studies theme
• No overall conclusions about strength of evidence

27
IV. Reflections on the conduct and
usefulness of systematic reviews

• SR process developed from Quantitative paradigm
  we are trying to adapt Qualitative literature to
  it
• Can this be meaningfully done?

28
Issues

• Reviews are not entirely systematic
• Messiness
• Paired reviewers anti-bias ideal
• Politics of agreement/consensus
• BUT
• Power relations among reviewers
• Assessing papers own merits or against overall
  standard?
• QA approaches change over course of the review

29
Issues

• The issue of the podium
• Arrogance decisions on inclusion of peer
  reviewed papers
• A side effect of systematic reviews
• Claims of ownership of a field from SR work

30
Issues

• Debates
• Can the synthesis integrate research using
  different methods/theories
• Although there are multiple descriptions/
  explanations of data, these all ultimately relate
  to some underlying reality/truth
• (Bondas Hall 2007, Mays et al 2005)

31
Some gains

• Think carefully about papers
• Learn a lot about a field
• Read a lot of poor papers (learn good, bad ugly)
• Hone critical appraisal skills (teaching, journal
  reviewing/editing)
• Opportunities to engage with people (outside
  academia)

32
Thank you

• Questions? Comments?