Research synthesis

1
Research synthesis

• Iain Chalmers
• Editor, James Lind Library
• (www.jameslindlibrary.org)

2
Research synthesis is the application, in
practice, of the principle that science is
cumulative.
3
How has the science of research synthesis evolved?
4
The evolution of measures to reduce biases in
research synthesis
5
(No Transcript)
6
As it is no easy matter to root out prejudices,
. it became requisite to exhibit a full and
impartial view of what had hitherto been
published on the scurvy, and that in a
chronological order, by which the sources of
these mistakes may be detected. Indeed, before
the subject could be set in a clear and proper
light, it was necessary to remove a great deal of
rubbish. James Lind, 1753
7
Joseph Goldberger, 1907? identified 44
studies and provided comprehensive references in
a bibliography ? used a newly developed serum
agglutination test to separate reliable studies
from those he considered unreliable ? tabulated
the raw data from 26 selected studies ?
calculated the mean rate of bacteriuria from the
pooled data. Winkelstein, 1998
8
More recently, physicists and social scientists
led the way
9
(No Transcript)
10
Light RJ, ed. (1983). Evaluation Studies Review
Annual. Beverly Hills, CA SAGE
Publications. Light RJ, Pillemer DB (1984).
Summing up the science of reviewing research.
Cambridge, Mass. Harvard University Press.
11
Mulrow C (1987). The medical review article
state of the science. Ann Int Med 106485-488.
 "Current medical reviews do not routinely use
scientific methods to identify, assess, and
synthesize information." 
12
The evolution of statistical methods for
reducing the play of chance in research synthesis
- meta-analysis
13
(No Transcript)
14
(No Transcript)
15
Ronald Fisher (1932) ... although few or no
statistical tests can be claimed individually as
significant, yet the aggregate gives an
impression that the probabilities are lower than
would have been obtained by chance.
16
In 1976, Gene Glass, an American social scientist
coined the term meta-analysis to refer to
the statistical analysis of a large collection
of analysis results from individual studies for
purposes of integrating the findings
17
Rosenthal and Rubin (1978) Smith and Glass
(1977) Glass and Smith (1979) Hunter et al.
(1979).Rosenthal R (1984). Meta-Analytic
Procedures for Social Research. Newbury Park,
CA Sage Publications.Hedges L, Olkin I
(1985). Statistical Procedures for
Meta-Analysis. Orlando Academic Press. Yusuf
S, Peto R, Lewis J, Collins R, Sleight P (1985).
Beta blockade during and after myocardial
infarction An overview of the randomised trials.
Prog Cardiovasc Dis XXVII (5), 336-371. 
18
By the end of the 20th century, research
synthesis was beginning to be taken more
seriously
19
There are research syntheses in such diverse
topics as advertising, agriculture, archaeology,
astronomy, biology, chemistry, criminology,
ecology, education, entomology, law,
manufacturing, parapsychology, psychology, public
policy, zoology and even of eyewitness accounts
of the Indian rope trick. Mark Petticrew,
BMJ 2001.
20
As science is cumulative, scientists should
cumulate scientifically!
21
Luc de Clapiers Vauvenarques, 1715-47
(Réflexions et Maximes)."Il est plus aisé de
dire des choses nouvelles que de concilier celles
qui ont étés dites. It is easier to say
something new than to reconcile things that have
already been said.orSynthesis of ideas is
harder than merely proposing a new idea. 
22
Lord Rayleigh, 1884If, as is sometimes
supposed, science consisted in nothing but the
laborious accumulation of facts, it would soon
come to a standstill, crushed, as it were, under
its own weightThe work which deserves, but I am
afraid does not always receive, the most credit
is that in which discovery and explanation go
hand in hand, in which not only are new facts
presented, but their relation to old ones is
pointed out.
23
(No Transcript)
24
Hill AB. Reasons for writing. BMJ 19654870.
People reading research reports want
investigators to answer four questions Why
did you start? What did you do? What answer
did you get? What does it mean anyway?

25
PremiseThe results of a particular research
study cannot be interpreted with any confidence
unless they have been considered, systematically,
together with the results of other studies
addressing the same or similar questions.
26
How well is this premise reflected in papers
published in major general medical journals?
27
Classification of Discussion sections in RCT
reports published in Ann Int Med, BMJ, JAMA,
Lancet, and N Eng J Med
28
Classification of Discussion sections in RCT
reports published in Ann Int Med, BMJ, JAMA,
Lancet, and N Eng J Med
29
A contemporary example of the consequences of not
taking research synthesis seriously Drugs to
prevent heart rhythm abnormalities
(arrhythmias)after heart attacks
30
The theoryPatients with arrhythmias are at
increased risk of early death after heart attack
Anti-arrhythmic drugs reduce arrhythmias after
heart attack These drugs should reduce early
death after heart attack
31
The evidenceA 1983 systematic review of 14
randomized controlled trials of anti-arrhythmic
drugs in heart attack

• The theoretical potential for a preventive or
  prophylactic effect of antiarrhythmic drugs ..in
  the treatment of coronary patients with
  ventricular arrhythmias has not been realized.
  
• Furberg, 1983

32
A 1993 systematic review of 51 randomized
trials of anti-arrhythmic drugs in heart
attack660 deaths among 11,712 patients
allocated drugs571 deaths among 11,517 patients
allocated to control Teo et al. JAMA
1993.

33
The consequenceAt the peak of their use in the
late 1980s, it has been estimated that
anti-arrhythmic drugs were causing every year
comparable numbers of deaths to the total number
of Americans who died in the Vietnam war (Moore
1995).
34
The discovery that these drugs are lethal could
have been made a decade earlier if the Discussion
sections in each report of a new trial had set
the new results in the context of a systematic
review of the results of all previous trials in
other words, if scientists had cumulated evidence
scientifically.
35
The human costs of failing to recognise the
importance of cumulating evidence
scientificallyAdvice on some life-saving
therapies has been delayed for more than a
decade, while other treatments have been
recommended long after controlled research has
shown them to be harmful. Antman et al.
1992Antman et al. JAMA 1992268240-8.
36
Setting the results of new results in the context
of a systematic review of the results of all
other relevant studies would become
straightforward if systematic reviews were always
done before embarking on new research.
37
Would any of you have agreed to participate in
a placebo controlled trial of prophylactic
antibiotics for colorectal surgery after
1975?
38
Reduction of perioperative deaths by antibiotic
prophylaxis for colorectal surgery
39
Would you ever have put babies to sleep on their
tummies?
40
(No Transcript)
41
(No Transcript)
42
Would a healthy young woman volunteer at Johns
Hopkins have died after inhaling hexamethonium if
the investigators had done a systematic review of
relevant evidence?Clark O, Clark L, Djulbegovic
B. Is clinical research still too
haphazard?Lancet 20013581648.
43
(No Transcript)
44
Would 7000 stroke patients have been persuaded to
participate in controlled trials of nimodipine if
researchers had done systematic reviews of the
relevant animal experiments? Johanna Horn et al
(2001). Nimodipine in animal model experiments
of focal cerebral ischaemia. Stroke 322433-2438.
45
Pound P, Ebrahim S, Sandercock P, Bracken MB,
Roberts I. The Reviewing Animal Trials
Systematically (RATS) Group.Where is the
evidence that animal research benefits humans?
BMJ  2004328514-17 (28 February)
46
Principles of good practiceAll new research
studies should be designed in the light of
scientifically-defensible syntheses of relevant
existing research evidence reported using the
new evidence to update these research syntheses
(thus making clear what contribution the new
study has made to the total evidence)
47
An important innovation by the BMJ
48
The main obstacles to applying these principles
in practice are psychological and social, not
practical. The practical challenge is being met
by very rapid evolution and application of the
science of research synthesis and electronic
publishing.
49
Can the public rely on scientists to use research
synthesis to ensure that limited resources for
research are used more efficiently and ethically?
50
Needed academic recognition that research
synthesis is research
51
.. The work which deserves, but I am afraid
does not always receive, the most credit is that
in which discovery and explanation go hand in
hand, in which not only are new facts presented,
but their relation to old ones is pointed out.
Lord Rayleigh 1884
52
If academia cannot be relied on to ensure that
new research projects begin and end with
syntheses of the results of other relevant
studies, who will protect the public from the
adverse consequences of current scientific
indiscipline?
53
Danish Research Ethics Committee System,
1997For a research ethical evaluation of
scientific experiments involving man it is
crucial that all relevant literature has been
reviewed by the research group before
submission....This will be a precondition
when the evaluating committee is judging the
originality of the project and, for example, the
permissibility of using placebo and not an
already known treatment in a control group.
54
UK Medical Research Council requirements of
people applying for support for new controlled
trials, 1999.Give references to any relevant
systematic review(s) and discuss the need for
your trial in the light of the(se)
review(s).If you believe that no relevant
previous trials have been done, give details of
your search strategy for existing trials.
For definition of a systematic review, see
Oxman, AD. Checklists for review articles, BMJ,
1994 309648-51.
55
The European Science Foundation (2001) endorses
the view that embarking on new research without
first preparing systematic reviews of relevant
existing evidence is indefensible on scientific
and ethical grounds, and recommends that its
Member Organisations should Require
applicants for support of new trials to refer to
scientifically defensible reviews of relevant
existing trials, or demonstrate that no other
relevant trials exist. Require data monitoring
committees to take account of new evidence
accumulating in updated systematic reviews.
56
In conclusionScience is cumulative, and
scientists must cumulate scientifically!As the
main funder of science, the public has a right to
expect that this will be reflected more
effectively in the way that science is conducted
and reported.It remains unclear how effectively
academia will respond to this fundamental
challenge to its traditional ways of working.
57
This talk has been based in part on material
published inChalmers I, Hedges L, Cooper H.A
brief history of research synthesis Evaluation
and the Health Professions20022512-37.
58
Francis Galton, 1901 "I have begun to think
that no one ought to publish biometric results,
without lodging a well-arranged and well-bound
copy of his data in some place where it should be
accessible, under reasonable restrictions, to
those who desire to verify his work.