Reliability of Individual Standard Deviations

1
Reliability of Individual Standard Deviations

• Ryne Estabrook
• Kevin Grimm
• Preparation for GSA, Nov 16-20
• Design and Data Analysis
• Nov 2, 2006

2
Individual Standard Deviations

• Measure of intraindividual variability.
• Scaled in the units of the measure.
• Easily calculable and interpretable.
• Measures of intraindividual variability are
  useful both as outcomes and predictors.
• Used in cases where meaningful interoccasion
  variability exists, but shows no discernable
  trend.

3
Problems with ISDs

• ISDs are treated as theoretically identical to
  means, and differences between them are not
  checked.
• Classical test theory violations
• ISDs assume meaningful true score variance.
• CTT assumes all variance is error variance.
• Means and ISDs may be differentially reliable.
• ISDs may require more occasions to be reliable
  than means do.

4
Objectives of the Simulation

• To explore how true-score variance affects
  reliability, simulations were carried out.
• Discern the requisite number of occasions and
  reliability of the original test to produce valid
  ISDs.
• Discover the variance conditions under which ISDs
  are most reliable, and allow researchers to
  estimate reliability.

5
Setting Up the Simulation

• For each iteration
• A sample of 100 records are assigned both a mean
  and an ISD from a bivariate normal distribution.
• Means are distributed in the population N(0,k).
• ISDs are distributed in the population N(f(k),
  g(k)) such that f(k) and g(k) vary across
  iterations.
• µISD 0.5-5.0 sMEAN, sISD .25-2.5
  sMEAN
• µISD3sISD to keep the distribution of ISDs above
  zero.
• Alterations to the ranges of the above parameters
  did not significantly alter results.

6
Setting Up the Simulation

• For each iteration
• A number of observations (t) and test reliability
  (?) were assigned.
• t within 3103, ? within .09.99
• Each record is observed t times, with an observed
  score at every occasion calculated from µj and
  sj.
• Observed Scoreij v? Trueij v(1-?) Errorij
• An observed mean and ISD is calculated for each
  record.
• Each dataset thus included 100 records for each
  possible combination of µISD, sISD,t and ?.
• The procedure was repeated for 75 datasets.

7
First Simulation

• Investigate the reliability of across occasion
  means.
• Correlation between true and observed means
  constitutes the reliability index.
• Regress reliability of the across occasion mean
  on the µISD, sISD, number of occasions and
  ?MEASURE.
• Stepwise regression of the parameters, their
  transformations and 2-way interactions yielded an
  average R2 of .936.

8
Results - Mean
µISD .5 sMEAN
µISD sMEAN
µISD 1.5 sMEAN
9
Results - Mean

• Inclusion of intraindividual variation
• Increasing µISD negatively affects reliability.
• Increasing sISD has a negligble negative affect
  on reliability.
• May result from a non-linear effect of µISD and
  the correlation between µISD and sISD .
• Highlights the CTT issue
• Any increase in variance is treated as error
  variance, decreasing the estimated reliability.

10
Results - Mean

• Increasing the number of observations provides
  the greatest benefit to reliability.
• Reliability of the original measure has little
  effect relative to other predictors.
• Multiple iterations of identical tests serve to
  increase the test length by a factor of t.
• Test reliability indicates the within occasion
  maximum for reliable measurement aggregating
  increases reliability very quickly.

11
Second Simulation

• Investigate the reliability of across occasion
  individual standard deviations.
• Correlation between true and observed ISDs
  constitutes the reliability index.
• Regress reliability of the ISD on the µISD, sISD,
  number of occasions and ?MEASURE.
• Stepwise regression of the parameters, their
  transformations and 2-way interactions yielded an
  average R2 of .955.

12
Results - ISD
13
Results - ISD

• Increasing the ratio of sISD to µISD increases
  reliability.
• Increasing sISD and µISD in proportion changes
  the units, but not the meaning, of ISDs.
• The number of occasions has a greater effect for
  ISDs than for means.
• The reliability of the measure used again has a
  negligible effect.

14
Conclusions

• Reliability of means and ISDs depend on the
  relationship between µISD, sISD sMEAN.
• Means are most reliable when variance (µISD or
  error) is minimized.
• ISDs are most reliable when sISD is maximized
  relative to µISD and, to a lesser extent, sMEAN.
• ISDs are less reliable than means.
• ISDs based on very few occasions may be too
  unreliable to interpret.

15
Implications

• Tests that are reliable for mean levels may not
  be reliable for measuring variability.
• ISDs from few occasions lack statistical power.
• ISDs do not differ theoretically from
  single-occasion population standard deviations.
• Standard deviations based on 10 observations of
  an individual are as reliable as a standard
  deviation from a sample of 10.

16
Thanks

• John Nesselroade
• Current and former colleagues at CDHRM at the
  University of Virginia
• NIA T32 AG20500-01