Guidelines for reporting research

1
Guidelines for reporting research

• Doug Altman
• Centre for Statistics in Medicine Oxford
• NCRI, 24 January 2006

2
Biased reporting is scientific misconduct

• Failure to publish an adequate account of a
  well-designed clinical trial is a form of
  scientific misconduct which can lead those caring
  for patients to make inappropriate treatment
  decisions. Chalmers, 1990

3
Clinical trials

• Focus on randomised clinical trials (RCTs)
• Principles of good reporting apply to all types
  of research

4
Outline

• Importance of clinical trials
• Importance of good reporting
• Evidence of bad reporting
• Reporting guidelines - CONSORT
• Publication (dissemination) bias
• Bias from unpublished research
• Bias in published research

5
Chalmers Altman. How can medical journals help
prevent poor medical research? Some opportunities
presented by electronic publishing. Lancet 1999

• Electronic publication of a protocol could be
  simply the first element in a sequence of
  threaded electronic publications, which
  continues with reports of the resulting research
  (published in sufficient detail to meet some of
  the criticisms of less detailed reports published
  in print journals), followed by deposition of the
  complete data set. Not only would this approach
  allow alternative explorations of the data, it
  would help to address some of the growing
  concerns about research misconduct.

6
Data for clinical trials

• Knowledge of existence (registration)
• Trial protocol
• Including detailed analysis plan
• Results of analyses
• Raw data
• Other documents
• e.g. Data collection forms

7
Importance of good reporting

• The whole of medicine depends on the
  transparent reporting of clinical trials
• Rennie D. CONSORT revisedimproving the reporting
  of randomized trials. JAMA 20012852006-7.

8
Importance of good reporting

• Assessment of quality and relevance of research
  is seriously impeded by inadequate reporting
• Randomised controlled trials (RCTs) should be
  reported fully and accurately
• Existence of trials and main findings, to ensure
  that we see all the evidence, not a biased subset
• Clear details of what was done, to enable
  evaluation of the reliability of findings
• Full reporting of all results (outcomes), to
  contribute to the overall evidence of
  interventions (benefits and harms)
• Much evidence indicates that all areas are
  deficient

9
Poor reporting of RCTs is common 1

• Not reporting an adequate method for generating
  random numbers
• 68 of 206 trials in obstetrics gynecology
  journals
• 52 of 80 trials in general medical journals
• Not reporting the mechanism used to allocate
  interventions
• 89 of 196 trials in rheumatoid arthritis
• 48 of 206 trials in obstetrics gynecology
  journals
• 44 of 80 trials in general medical journals
• 93 of 73 trials in one dermatology journal
• 96 of 2000 trials in schizophrenia
• 99 of 122 trials of SSRIs

10
Poor reporting of RCTs is common 2

• Not stating whether blinding was used
• 51 of 506 trials in cystic fibrosis
• 33 of 196 trials in rheumatoid arthritis
• 38 of 68 trials in dermatology
• Poor reporting of adverse effects of
  interventions
• 61 of 192 reports in 7 medical areas

11
The CONSORT statementBegg et al 1996 Moher et
al 2001

• 22 items which should be reported in a paper
• Also a flow diagram showing patient progress
  through the trial (to be included in the trial
  report)
• Evidence-based, whenever possible
• Accompanied by a long explanatory document

www.consort-statement.org
12
Assessed for eligibility
(n)
13
CONSORT checklist (22 items)

• TITLE ABSTRACT
• INTRODUCTION
• Background
• METHODS
• Participants
• Interventions
• Objectives
• Outcomes
• Sample size
• Randomization
• Sequence allocation
• Allocation concealment
• Implementation
• Blinding (Masking)
• Statistical methods

• RESULTS
• Participant flow
• Recruitment
• Baseline data
• Numbers analyzed
• Outcomes and Estimation
• Ancillary analyses
• Adverse events
• DISCUSSION
• Interpretation
• Generalisability
• Overall evidence

14
CONSORT items

• There are really gt22 items
• Item 6
• Clearly defined primary and secondary outcome
  measures and, when applicable, any methods used
  to enhance the quality of measurements (e.g.,
  multiple observations, training of assessors).

15
Avoid unhelpful labels

• CONSORT does not use term double blind
• Item 11
• Whether or not participants, those administering
  the interventions, and those assessing the
  outcomes were blinded to group assignment.

16
Secondary goal of CONSORT

• To encourage and provide incentives for
  researchers to conduct high-quality, unbiased
  randomized trials
• Arguably even more important

17
Extensions

• Cluster RCTs (BMJ 2004)
• Harms (Ann Intern Med 2004)
• Noninferiority RCTs (JAMA 2006)
• More planned

18
Factors in success of CONSORT

• Membership of Group
• Methodologists
• Trialists
• Editors
• Reporting rather than conduct
• Focus on main issues
• One side of paper
• No competitors
• High profile publications
• Supported by Editorial groups, gt150 journals,
  peer review granting agencies

19
Process 1

• Preparation
• Assessment of existing literature on reporting
• Identification of relevant empirical evidence
• Meeting of experts
• Clear scope
• Clear aim reporting not conduct
• Develop checklist ( flow diagram)
• Small group fine-tunes checklist and develops
  publications
• Seek input from wider group (e.g. on web)

20
Process 2

• Publications
• Statement
• Explanatory document
• Web site
• Extra material
• Invite comments
• etc
• Periodic revision
• Take account of feedback / criticism
• Take account of new evidence
• NB no funding!!

21
Some history of reporting guidelines developed by
consensus

• 1993-94 SORT RCT
• 1994 Asilomar RCT
• 1995-96 CONSORT RCT
• 1996-99 QUOROM SR/M-A of RCTs
• 1997-00 MOOSE M-A of obs studies
• 1999-01 CONSORT II RCT
• 2000-03 STARD Diagnostic
• 2003-04 TREND NonRCT/Behavioural
• 2001-05 REMARK Prognostic (cancer)
• 2004-?? STROBE Observational
• 2005-?? QUOROM II SR/M-A of RCTs

22
Impact of CONSORT

• Review of Instructions to Authors of 167 high
  impact medical journals in 2003
• 36/166 (22) referred to CONSORT
• 8 published studies show that adoption of CONSORT
  by journals is associated with improved reporting
  of RCTs
• Much room remains for improvement

23
(No Transcript)
24
Publication (dissemination) bias

• Study not published
• Only selected findings published
• Stopping a trial early for benefit

25
From trial to review
Trial 1
Trial 3
Trial 2
Publication
Publication
Publication
Outcomes
Outcomes
Outcomes
Systematic review
26
Publication biases

• Study publication bias
• selective publication of entire studies
• Outcome reporting bias
• selective reporting of outcomes within published
  studies

27
Study publication bias

• Studies reported at conferences are less likely
  to be fully published if not significant
• Statistically significant results are about 20
  more likely to be published Scherer et al, CDMR
  2004
• Even when published nonsignificant studies take
  longer to reach publication that those with
  significant findings
• Trials with null or negative findings took an
  average just over a year longer to be published
  than those with positive results Hopewell et
  al, CDMR 2001

28
Full publication bias (abstracts not leading
to full papers)

• Cohort of 500 oncology trials with gt200
  participants Krzyzanowska et al, JAMA
  2003
• Preliminary results presented at ASCO
• 26 not published in full within 5 years
• 81 with Plt0.05 were published
• 68 with Pgt0.05 were published
• So not just a problem of small trials!

29
3 studies of selective reporting of trial results
Chan et al, 2004 2005

• Study of published reports of RCTs
• All Medline-indexed RCTs published in Dec 2000
• 519 RCTs, gt10000 outcomes
• Comparison of protocols and publications
• 102 RCTs submitted to the Copenhagen and
  Frederiksberg Research Ethics committee in
  1994-95
• 48 RCTs funded by CIHR 1990-98
• Questionnaire sent to all authors in each cohort

30
Pooled OR for outcome reporting bias (fully
versus incompletely reported outcomes) by study
design and cohort
31
Impact of these biases

• Study publication bias odds ratio of 2.0
• 81 of studies with Plt0.05 were published
• 68 of studies with Pgt0.05 were published
• Selective reporting odds ratio of 2.4
• 71 of outcomes with Plt0.05 are fully reported
• 50 of outcomes with Pgt0.05 are fully reported
• Biases are cumulative
• Hard to quantify overall impact

32
Major discrepancies in the specification of
primary outcomes between protocols and
publications
33
Protocols

• Clearly, trial protocols need to be in the public
  domain
• Preferably before the trial finishes

34
Biased reporting is scientific misconduct

• In return for the altruism and trust that make
  clinical research possible, the research
  enterprise has an obligation to conduct research
  ethically and to report it honestly.
• Clinical Trial Registration A Statement from
  the International Committee of Medical Journal
  Editors, September 2004

35
Data for clinical trials

• Knowledge of existence (registration)
• Trial protocol
• Including detailed analysis plan
• Results of analyses
• Raw data
• Other documents
• e.g. Data collection forms

36
(No Transcript)