Assessment of Reliable Change: Methods and Assumptions

1
Assessment of Reliable Change Methods and
Assumptions

• Michael Basso, Ph.D.
• Associate Professor and Director of Clinical
  Training
• Department of PsychologyUniversity of Tulsa
• Clinical Associate Professor
• Department of PsychiatryUniversity of Oklahoma

2
Objectives

• Provide background concerning methods of
  assessing reliable change
• Describe assumptions and applications of reliable
  change scores
• Illustrate use of reliable change scores

3
Assessment of Clinical Change

• Two Basic Approaches
• Assessment of Group Differences Across Time
• Assessment of Individual Differences Across Time

4
Assessment of Group Differences Across Time

• Assessment of statistically reliable change
• Does the treatment yield significant benefits
  for groups of patients?
• i.e., do average scores at T1 and T2 come from
  different distributions
• This approach describes the average rate of
  change over groups primarily
• It is accomplished with repeated measure ANOVA
• Problem You could have a statistically
  significant difference with a very small effect
  size, but it might not be a clinically meaningful
  change

5
(No Transcript)
6
Assessment of Individual Differences Across Time

• Assessment of Clinically Meaningful Change
• Did the patients change in performance at T2
  exceed base rates of change?
• i.e., did the individual show change that
  exceeded expectations based on measurement error,
  practice effects, and regression to the mean?
• This method describes the base rate of change
• Change that is exceeds the base rate is not
  normal, and is therefore clinically meaningful
• Our focus is on the assessment of clinically
  meaningful change in individuals, but this method
  can be applied to group data as well

7
Assessment of Clinical Change for Individuals

• How do you establish the base-rate of change?
• Bear into consideration that
• It would be improbable to obtain the exact same
  score twice
• There is no perfect test-retest correspondence
  because of
• measurement error
• regression to the mean
• practice effects

8
Two Methods of Assessing Base Rates of Change

• Reliable change Index scores
• Does change exceed what would be expected based
  on measurement error alone?
• This method is based on Reliability of
  measurement
• It is used for typical performance tests
• i.e., attitude, personality, psychopathology,
  etc.
• Standardized Regression-Based Change Scores
• Does change in scores exceed expectations based
  on T1 (baseline) scores?
• This method is based on a validity coefficient
  (i.e., what T2 score is predicted by the T1
  score)
• It is used for maximal performance tests
• i.e., IQ, neuropsychological, etc.

9
Reliable Change Index Scores

• Elaborated by Jacobson and Truax (1991)
• Based on the standard error of the difference
• Which in turn is based on the reliability
  coefficient
• This reflects the sampling distribution of
  difference scores
• it implies the magnitude of differences between
  two test scores that vary by chance alone
• Assumptions
• Error components are mutually independent and
  independent of true pretest and posttest scores
• Error is normally distributed with a mean of 0
• SE of error is equal for all subjects
• These assumptions are questionable in clinical
  settings (cf. Maassen, 2004)

10
Standard Normal CurveDistribution of Difference
Scores
11
Reliable Change Index Scores--Method

• To use the RCI, you must compute the SE of
  difference between two scores
• SEdiff(2(SD(1-rxx)1/2)2)1/2
• Then, compute a confidence interval for change
  scores
• for 95 confidence, you multiply 1.96 SEdiff
• for 90 confidence, you multiply 1.60 Sediff
• Does the raw score change between T2 and T1
  exceed the confidence interval?
• If so, it represents change that exceeds the base
  rate expected based on measurement error
• Thus, clinically meaningful change has occurred
• If not, then the change is consistent with the
  base rate expected based on measurement error
• Thus, no clinically meaningful change has occurred

12
Reliable Change Index ScoresAn Example

• Ferguson, Robinson, Splaine (2002)
• SF-36 in 200 patients who had undergone a
  Coronary Artery Bypass Grafting (CABG) surgery
• SF-36 contains 8 scales
• Physical Functioning
• Role Functioning Physical
• Bodily Pain
• General Health
• Vitality
• Social Functioning
• Role Functioning-Emotional
• Mental Health

13
Reliable Change Index ScoresAn Example

• Ferguson, Robinson, Splaine (2002)
• Physical Functioning
• Reliability.93 (from normative sample of 2474)
• Mean of normative sample84.15
• SD of normative sample23.28
• SEdiff(2(SD(1-rxx)1/2)2)1/2
• SEdiff(2(23.28(1-.93)1/2 ) 2) )1/29.85
• 95 CI (SEdiff)1.9619.32
• T1 Mean40.97
• T2 Mean63.39
• Mean Diff22.42
• The mean difference exceeds 19.32
• Thus, clinically meaningful change has occurred
  as a result of surgery

14
Reliable Change Index ScoresAn Example

• Ferguson, Robinson, Splaine (2002)
• Mental Health
• Reliability..84 (from normative sample of 2474)
• Mean of normative sample75.01
• SD of normative sample21.40
• SEdiff(2(SD(1-rxx)1/2)2)1/2
• SEdiff(2(21.40(1-..84)1/2 ) 2) )1/210.92
• 95 CI (SEdiff)1.9621.40
• T1 Mean72.08
• T2 Mean71.84
• Mean Diff-0.24
• The mean difference fails to exceed 21.40
• Thus, no clinically meaningful change has
  occurred as a result of surgery

15
(No Transcript)
16
Standardized Regression Based Change Scores

• Elaborated by Charter (1996)
• Used primarily for maximal performance tests
• The RCI of Jacobsen and Truax is used for typical
  performance tests
• It assumes that errors between test scores at
  baseline and time 2 are uncorrelated
• This assumption is untenable in maximal
  performance tests because of practice effects

17
Standardized Regression Based Change Scores

• Based on the standard error of prediction
• SEpredSDY2((1-rY1Y22)1/2)
• The SE reflects the sampling distribution of
  predicted scores
• It implies the range of scores that might be
  expected at time two that may be expected from
  the baseline score and prediction error
• This method requires you to compute the estimated
  true score
• Y2TrueM((rY1Y2)(Y1-M))
• The T2 score is prone to error, and this formula
  permits an unbiased estimate of the true score
• The SEpred is used to compute a confidence
  interval around the estimated true score

18
Standard Normal CurveDistribution of Standard
Error of Prediction Around Estimated True Score
19
Standardized Regression Based Change
Scores--Method

• To use the SRB, you must compute the estimated
  true T2 score
• Compute the confidence interval around this
  estimated true T2 score
• For 95 confidence, you multiply 1.96 SEpred
• For 90 confidence, you multiply 1.60 SEpred
• Does the obtained T2 score fall outside the
  confidence interval around the estimated true
  score for T2?
• If so, it represents change that exceeds the base
  rate expected based on measurement error,
  regression to the mean, and practice
• Thus, clinically meaningful change has occurred
• If not, then the change is consistent with the
  base rate expected based on measurement error,
  practice, and regression to the mean
• Thus, no clinically meaningful change has occurred

20
Standardized Regression Based Change Scores--An
Example

• Basso, Carona, Lowery, Axelrod (2002)
• WAIS-III re-tested in a group of control subjects
  over a 3-6 month interval
• FSIQ
• Test-Retest Reliability.90
• T1 Mean T1109.4 (11.6)
• T2 Mean T2115.0 (12.1)
• SEpredSDY2((1-rY1Y22)1/2)
• SEpred(12.1((1-.902) 1/2))5.29
• 95 CI (SEdiff)1.9610.36
• Mean Diff5.60
• The mean difference fails to exceed the 95 CI
• No individual had a score exceeding the 95 CI
• To apply the SRB, the T2 True Score is estimated
• If the obtained score falls within the CI around
  the T2 True score, then no clinically meaningful
  change has occurred

21
Standardized Regression Based Change Scores--An
Example

• Basso, Carona, Lowery, Axelrod (2002)
• An example application
• T1 obtained score104
• T2 obtained score116
• Estimated True T2 Score
• YTrueM((rY1Y2)(Y1-M))
• YTrue100(.90)(104-100)103.6
• 116 exceeds 10.36 points from 103.6
• Thus, meaningful change has occurred

22
Standardized Regression Based Change Scores--An
Example

• Basso, Carona, Lowery, Axelrod (2002)
• An example application
• T1 obtained score103
• T2 obtained score106
• Estimated True T2 Score
• YTrueM((rY1Y2)(Y1-M))
• YTrue100(.90)(106-100)105.4
• 105 falls within 10.36 points of 106
• Thus, no meaningful change has occurred

23
Questions?