Planning Systematic Reviews

1
Planning Systematic Reviews
2
Systematic Reviews

• An ordered, systematic analysis
• Criteria for study inclusion are decided either
  in advance, or after doing a search and seeing
  what evidence exists.
• Stronger systematic reviews are those carried out
  on RCTs, but this is not always possible

3
State objectives and outline eligibility
criteria Search for evidence (research) that
meets this criteria Tabulate characteristics of
each trial identified and assess its
methodological quality Apply criteria and justify
exclusions
4
Planning Your Review

• What is the need for the review?
• What is the problem, who has it, and what has
  been done in the past.
• Consider the background of the problem
• How many have it, what is the exact nature, does
  it lead to other problems?
• What is the cost to society?

5
Article Inclusion Criteria
6
General Article Inclusion Criteria

• Specified diagnostic groups
• Specified intervention that reviewers feel can be
  identified by techniques described in the
  articles
• Age categories when appropriate
• Specified Holms levels or types of research
  design

7
Article Inclusion Strategy

• The language in which the studies are
  published. (Typically only English)
• Articles important to the practice
  question.
• Interventions typical of occupational
  therapy specified in your clinical question.
• Does an occupational therapist have to
  provide the intervention? (If so, this may overly
  limit your search)
• Studies have to be published in
  peer-reviewed journals or have to be theses or
  dissertations
• Dates of published articles, typically
  involves a range, such as 1990 2002.
• Other criteria specific to selecting
  literature to answer your practice question.
  Typically these specific criteria include
  limitations created by client characteristics
  that are of interest.

8
For example

• Articles specifically targeted the use of NDT as
  an intervention
• Intervention stated as NDT but combined with
  other sensorimotor techniques
• Intervention could be identified by reviewers as
  NDT by the techniques described
• All subjects had CP
• OR all subjects with CP had data that could be
  separated from other participants

9
Example 2

• NDT was used as a treatment
• Did the subject group consist of children ages
  0-18 years of age with a neurological
  dysfunction?
• Was there a clinical outcome used in the study
  (standardized assessment or non-standardized
  observations)?
• Were subjects randomly assigned?
• Published after 1975

10
Including the highest levels of evidence possible

• The stronger the evidence, the more creditable
  your review
• However, finding RCTs or meta-analyses may not be
  feasible
• Start with Level 2 or 3 studies if you know there
  are not RCTs

11
So whats in your article inclusion criteria????
12
Search Strategy

• Set keywords and combinations of keywords
• Document ALL keywords and combinations you will
  need to list them in your paper
• List all databases you search
• Specify how many references came from each
  database
• Document your hand search strategy

13
Criteria for Judging Articles
14
Criteria for Reviews

• Used to judge the methodological aspects of
  studies included in a review
• Criteria include information relating to internal
  validity and external validity of studies

15
Blinding in Research
The purpose of blinding is to minimize bias in
the process of recording or collecting
information about the outcome of the treatment.
16
So who gets blinded?

• Subjects
• Persons administering the treatment
• Persons collecting the data
• Persons analyzing the data

17
Threats to Internal Validity

• History
• Maturation
• Attrition
• Testing
• Instrumentation
• Statistical Regression
• Selection
• Ambiguity about Direction of Causal Influence
• Treatment Diffusion
• Demoralization of respondents receiving treatment
  assumed to be less desirable

18
Threats to Internal Validity

• History - events occurring after administration
  of the independent variable that could affect or
  cause outcomes
• Maturation - changes in behavior/skills occurring
  over time after administration of the independent
  variable
• Attrition - subjects who drop out of the study,
  causing groups to be asymmetrical

19
Threats to Internal Validity

• Testing - the effects of having repeated tests on
  behaviors under study OR the effect of certain
  kinds of assessment on subjects behavior
• Instrumentation - the effect of instrumentation
  with poor reliability on measures
• Statistical regression - another reliability
  concern extreme scores on a given variable can
  change to approach the mean over time (regression
  to the mean) a real issue when subjects are
  chosen because of extreme scores on a variable

20
Threats to Internal Validity

• Selection - When subjects are selected for groups
  rather than randomly assigned, the groups end up
  being different. Matching can help but does not
  balance out the selection factor completely
• Diffusion - occurs when one group becomes aware
  of the treatment used for the other group and
  integrates some aspect of the treatment into
  their own intervention
• Compensatory Equalization - when persons
  delivering one intervention believe the treatment
  the other group receives is more desirable and
  works harder, ultimately delivering compensatory
  services

21
Threats to Internal Validity

• Compensatory rivalry and resentful demoralization
  - this is the situation when the participants in
  one group believe the other groups treatment is
  desirable and work extra hard to compensate,
  thereby affecting the outcomes. Resentful
  demoralization occurs when one group becomes
  angry with the treatment they receive and dont
  work as hard as they should
• Ambiguity about causal influence - when the
  cause-effect sequence is unclear - does the
  independent variable cause the dependent variable
  or vice versa?

22
Threats to Internal Validity

• Selection and history - when the history of
  non-randomized subjects interacts with selection
  to influence the outcomes
• Subjects and maturation - when selected subjects
  in experimental and control groups mature at
  different rates, affecting the outcomes
• Subjects and instrumentation - when the
  instrumentation results in different measures for
  each group involved in the study

23
Can the results of the study be generalized to
the population of whom the sample is
representative in a given study? This question
asks about the external validity of a study.
24
Threats to External Validity

• Interaction of Treatment and Selection
• Interaction of Treatment and Setting
• Interaction of Treatment and History

25

• Were inclusion and exclusion criteria of the
  study population well described and followed?
• Was the intervention well described and was there
  adherence to the intervention assignment?
• Were the measures used clearly described? Were
  they valid and reliable for measuring the
  outcomes of interest?
• Was the outcome assessor unaware of the
  intervention status of the participants (i.e.
  blind assessment?
• Did the authors conduct and report appropriate
  statistical evaluation, including power
  calculations?
• Were dropout/loss to follow-up reported and less
  than 20? For 2-group designs, was dropout
  balanced between the groups?
• Considering the study design, were appropriate
  methods for controlling confounding variables and
  limiting potential biases used?