ReSIS Realist Synthesis Implementation Strategies

1
ReS-ISRealist Synthesis Implementation
Strategies
Alison Hutchinson, Australia/Canada Alyce
Schultz, USA Brendan McCormack, UK Bridie Kent,
New Zealand Cheryl Stetler, USA Erna
Snelgrove-Clarke, Canada Jo Rycroft-Malone,
UK Kara DeCorby, Canada Lars Wallin,
Sweden Marita Titler, USA Tracey Bucknall,
Australia Val Wilson, Australia

• The story so far.

2
Outline

• Background methodology
• Searching retrieval
• Process issues
• Emerging findings

3
Background

• KU colloquia
• Frustration
• Existing evidence base for effective
  interventions

4
The state of the evidence base

• Systematic review evidence
• Some indications
• Facilitative approaches, feedback, education.
• Some limitations

e.g. Thomas et al 2000, OBrien et al , 2003,
Foxcroft Cole 2003, Grimshaw et al
2004, Thompson et al 2007
5
Limitations

• Equivocal findings
• Dont necessarily relate to the professional
  practice of all practitioners
• Use of behaviour change as outcome measure
• Lack of acknowledgement about complexity and
  context

6
A different approach different lens

• Want to know about what is working, what isnt
  working
• Different contexts ( levels)
• Different stakeholders
• Address meaningful questions
• Theoretical driven

7
Methodology

• Realist synthesis (Pawson et al, 2004)
• A method for studying complex interventions using
  diverse bodies of data
• Clear objectives and inclusion criteria
• Benefits
• Flexible but rigorous
• Capacity for providing detailed and practical
  recommendations

8
The Realist Alternative

• Review method based on principles of realistic
  evaluation
• Focus is on reviewing complex social
  interventions e.g. policy, management, service
  delivery
• Review takes place at the level of theories that
  underpin complex interventions
• Explanatory focus seeking answers to the
  question What works, for whom, in what
  circumstances, in what respects and why?

9
Characteristics of Complex Social Interventions

• Consist of theories
• Involve actions of people
• Consist of a chain of steps or processes that
  interact
• Constituent steps and processes are rarely linear
• Embedded in social systems
• Prone to modification
• Open systems that change through learning
• (Pawson et al, 2004)

10
Rethinking the Standard Systematic Review Template

• Clarifying the scope of the review
• Searching for evidence
• Appraisal of primary studies
• Data extraction
• Data synthesis
• Interpretation of findings/ presentation of
  results and recommendations

11
Clarifying the Scope of the Review

• Identify the review question
• Nature and content of the intervention
• Circumstances or context for its use
• Policy intentions or objectives
• Refine the purpose of the review
• Theory integrity, adjudication, comparison
• Articulate key theories to be explored
• Draw up long list
• Group, categorise or synthesise
• Design the evaluative framework

12
Purpose of the Review

• What are the interventions and
• strategies that are effective in enabling
• evidence informed healthcare?

13
OUTCOMES
ReS-IS Theoretical Model
1. Change Agency
2. Systems change

• Change agent characteristics
• Change agent interventions
• Interplay between change agent characteristics
  and interventions

• Systems change interventions
• Interplay between setting and interventions

DOSE
LEVELS
Theory and Contextual Factors
what works, for whom, in what circumstances, in
what respects and why
3. Technology
4. Education and Learning

• Characteristics of technological interventions
• Impact of technological interventions

• Characteristics of education interventions
• Impact of education interventions

14
Four Theories and 13 Theoretical Foci

• Theory area 1 - Properties of change agency in
  KU
• What impact do the characteristics of the change
  agent have on KU
• What is the overall impact of the change agent
  intervention on KU?
• What impact does the interaction between the
  change agent and the setting have on KU?
• Theory area 2 system change in KU
• What impact do characteristics of the systems
  change intervention(s) have on KU?
• What is the overall impact of the system change
  intervention(s) used?
• What impact does the interaction between the
  system change and the setting have on KU?
• What impact do senior leadership roles have in
  creating practice environments that integrate
  daily use of evidence at the point of care
  delivery?

15

• Theory area 3 properties of technologies (paper
  electronic) used in KU
• What impact do the characteristics of the
  technological intervention(s) have on KU?
• What is the overall impact of the technological
  intervention(s) used?
• What impact does the interaction between the
  technological intervention and the setting have
  on KU?
• Theory area 4 education interventions in KU
• What impact do the characteristics of the
  education intervention(s) have in enabling KU?
• What is the overall impact of the education
  intervention(s) used?
• What impact does the interaction between the
  education intervention and the setting have on
  KU?

16
Core and Quality Analysis

• Core Areas Analysis
• Is there evidence of particular theoretical
  perspective(s) impacting on the effectiveness of
  the intervention?
• Is there evidence of contextual factors impacting
  on the effectiveness of the intervention?
• Is there evidence of the level of the
  intervention impacting on the effectiveness of
  the intervention?
• Is there evidence of the intervention dose
  impacting on the effectiveness of the
  intervention?
• Quality Analysis
• Is the evidence provided in this theory area good
  and relevant enough to be included in the
  synthesis (consider issues of sample size, data
  collection, data analysis and claims made)

17
Search for Evidence

• Searching by stages to
• Get a feel for the literature
• Identify key programme theories (and refine
  inclusion criteria)
• Test and refine programme theories
• Purposive and snowball sampling
• Look across policy domains
• Final search for additional studies when review
  nearing completion

18
Data Extraction

• Develop bespoke set of data extraction forms
  and notation devices
• Extract different data from different studies to
  populate evaluative framework with evidence
• Interest is in the chains of inference in the
  original study this body of evidence allows me
  to draw the following conclusions

19
Data Synthesis

• Basic task is to refine the programme theory i.e.
  determine what works, for whom, in what
  circumstances, in what respects and why
• Use contradictory evidence to generate insights
  about the influence of context

20
Searching, Relevance Testing, Data Extraction

• search strategy
• relevance, 2 stream approach
• data extraction, final relevant papers

21
Search Strategies

• broad, not discipline-specific, corresponding to
  healthcare in general
• approx 39 lines of search text used with Boolean
  operators

22
Search Results

• 6 online databases (1997-2007)
• (Medline, CINAHL, Embase, PsycInfo, Sociological
  Abstracts, Web of Science)
• Health Sciences Librarians (Dalhousie University,
  McMaster University) consulted on search
  strategies run March 5, 2007 in OVID

23
Search Results

• 24,021 electronic references
• Medline 4,530
• Embase 8,482
• CINAHL 5,683
• PsycInfo 4,993
• Sociological Abstracts 248
• Web of Science 130
• 196 potentially-relevant papers based on title
  and abstract
• (return rate of 0.8)

24
Inclusion Criteria

• relate to purpose statement
• What are the interventions and strategies that
  are effective in enabling evidence-informed
  health care?
• published in the last 10 years
• english-language
• healthcare related
• all disciplines
• not limited by research focus or design
• (e.g. papers addressing intervention application
  in addition to effectiveness)

25
Data Extraction

• papers divided among 5 work groups
• further papers eliminated based on full versions
• data extraction of each paper by 2 reviewers
• final data extraction resulted in a number of
  papers being omitted

26
Overall Process What challenges were overcome

• Methodology
• Questions
• asking the right question to get the literature
• Search terms
• Volume
• Choice of one area first to refine process
• Inclusion or exclusion criteria
• Health non-health papers
• Empirical non empirical papers
• Theory papers
• Interventional papers
• Process of analysis,
• 2 independent reviews
• double checking all papers
• Clarity, confidence in process
• Assumptions on data extraction

27
Overall Process What worked well

• Communication-
• face to face and phone
• email
• Yahoo discussion group
• Variable perspectives
• International group
• Research methods used
• Range of specialties
• Variable work processes
• Established process
• Agree rules of engagement, responsibilities and
  outputs
• Small and large group
• Product Development
• Established database
• Completed preliminary review
• Refined processes
• Learning
• Access and exposure to new literature
• Realist synthesis method and philosophy

28

• Preliminary Findings

29
Theory Area1 Impact of Change Agents on KU

• Character of the literature
• Crosses a number of theory areas
• Poor linkages between processes, context and
  outcomes
• Methodological problems and study design
  challenges

30

• Impact of the Characteristics of the Change
  Agent?
• Role characteristics
• Knowledge characteristics
• Overall impact of change agent interventions?
• Potential to change practice context
• Increasing knowledge of others
• Providing feedback
• Awareness raising
• Some practice change
• How?

31

• Impact of Education (overlap with theory area 4)
• Change agent and education interventions
• Education that includes content and process
• Barriers and how to overcome these
• Specific projects and professional/personal growth

32

• Impact of the interaction between change agent
  and setting?
• Relationship between character of context and
  impact of change agent
• Illogical inferences!

33
Next Steps

• Refine data extraction templates
• Complete synthesis for theory area 1
• Write up process, challenges, findings
• Begin review process for other theory areas
• Need funding!!!!!!

34
Strengths of the Realist Approach

• Firm roots in philosophy and the social sciences
• Not a method or formula, but a logic of enquiry
• Pluralist and flexible
• Explanatory, as opposed to judgemental
• Learns from, rather than attempting to control,
  real world phenomena
• Engages stakeholders in a systematic way
• Has the potential to maximise learning across
  policy domains

35
Some Limitations

• Stage of development of the realist approach
  small number of completed reviews
• Cannot be used as a protocol driven approach
• Handling potentially large volume of evidence
• Developing/documenting reproducible methods
• Demonstrating and maintaining objectivity
• Requires a high level of experience on the part
  of the reviewer
• Leads at best to tentative recommendations