Evaluating methodological quality in the criminal justice system literature

1
Evaluating methodological quality in the criminal
justice system literature
2
Dr Amanda Perry

• Centre for Criminal Justice Economics and
  Psychology, University of York.

3
Overview of the session

• Background to RCT in criminal justice
• Quality assessment
• The Maryland Scientific Methods Scale (SMS)
• Problems associated with the SMS
• Where next.

4
Background

• Few RCT conducted in the UK
• Systematic review revealed 125 conducted between
  1957-2005
• (Farrington Welsh, 2005)
• Concurrent findings from UK systematic review
  between 1990-2002
• (Perry, McDougall Farrington, 2005)

5
Resistance to RCT

• Historical resistance
• Ethical and moral reasons
• Practical difficulties

BUT... New era Campbell Collaboration
(2000) Department of Health/Home Office (2000)
6
New age of RCT
Feasibility studies (Farrington Joliffe,
2002) (Farrington et al., 2002) Matrix (2006)
Criteria for assessing feasibility

• Current RCTs
• Evaluation of Cognitive behavioural skills
  programmes (McDougall, Bowles, Perry Clarbour,
  ongoing).
• Evaluation of Restorative Justice Programmes
  (Strang Sherman, 2006)

7
Quality assessment

• Reviews of quality assessment tools The medical
  field
• Moher et al. (1995) identified 25 scales devised
  up to 1993.
• Juni et al (1999) compared 25 scales for purpose
  of inter-rater reliability (r.72).

8
Quality assessment

• Reviews of quality assessment tools The
• Social Sciences
• Gibbs (1989) social work
• STROBE STrengthening the Reporting of
  OBservational studies in Epidemiology
  http//www.strobe-statement.org/index.html
• Maryland Scientific Methods Scale Criminal
  Justice (Sherman et al., 2002)

9
Quality assessment

• The Maryland Scientific Methods Scale (SMS)
• (Sherman et al., 2002, based on work by Cook
  Campbell, 1979)

• Purpose of the SMS
• Simple (measuring internal validity)
• To provide policy makers with information about
  the evidence
• Aim to classify all programmes into 1 of 4
  categories

10
The SMS quality assessment..
Five point scale Rating 1-5 Rating 1
Correlational study Rating 2 Pre and post test
study Rating 3 Observational cohort with
comparable group Rating 4 Quasi-experimental/c
ontrolled trial Rating 5 Randomised controlled
trial

11
The SMS quality assessment..

• Statistical conclusion validity
• Was the statistical analysis appropriate?
• Did the study have low statistical power to
  detect effects because of small samples?
• Was there a low response rate or differential
  attrition?
• Construct validity
• What was the reliability and validity of
  measurement of the outcome?

12
Evidence and use of the SMS

• What Works Evidence from 2 or more studies
  reporting positive results scoring 3 or above on
  the SMS showing statistical significance and
  desirable effects and the preponderance of all
  available evidence showing effectiveness.
• What Doesnt Work Evidence from 2 or more
  studies reporting negative results scoring 3 or
  above on the SMS

13
Evidence and use of the SMS

• What s Promising Reporting evidence from 1
  study reporting positive results scoring 3 or
  above on the SMS.
• What Unknown Evidence from 1 study with a
  negative or inconclusive result scoring 3 or
  above on the SMS..

14
Use of the SMS

• What works an example
• CCTV in car parks
• (evidence from Welsh and Farrington, 2002)
• Street lighting
• (evidence from Painter Farrington, 1997
  1999b2001b Farrington Welsh, 2002)
• Burglary reduction schemes
• (evidence from Ekblom, 1996a 1996b)

15
Limitations of the SMS

• Not fully assessing all threats to the validity
  of a study
• Categorising study designs
• Does not take into consideration at What cost?

16
Limitations of the SMS

• Is designed to apply equally to all experimental
  units
• Does not embrace all study designs
• Method of drawing conclusions on what works based
  on statistical significance rather than effect
  size

17
Improving the SMS.

• Farrington (2003)
• Based on five key criteria
• Internal validity
• Descriptive validity
• Statistical conclusion/validity
• Construct validity
• External validity

18
Improving the SMS
Information for policy makers 3 five point
scales

• Design
• (internal validity)
• Execution
• (construct validity/statistical conclusion
  validity /sampling elements of external
  validity)
• Reporting of the trial

19
Beyond the SMS..

• Assessing the cost of an intervention
• Adequate and standardised follow-up periods for
  outcome measures
• (e.g., reconviction rates)
• Encouraging journal editors to use a new
  scale/standard similar to CONSORT statement

20
Beyond the SMS..

• Development of a specific quality assessment tool
  evaluating RCT in the criminal justice
• To incorporate all elements of validity
• To include an assessment of cost/cost
  effectiveness/cost-benefits of an intervention
• To include guidance on adequate and standardised
  follow-up periods for outcome measures
• (e.g., reconviction rates)

21
SUMMARY

• RCT rarely used in UK criminal justice system
• Use of the SMS as quality measure
• Room for improvement
• Development of a new scale.

22
CONTACT

• Amanda Perry
• Centre for Criminal Justice Economics and
  Psychology
• University of York
• Email aep4_at_york.ac.uk