Sensitivity and Specificity of FDR Methods in Neuroimaging

1
Sensitivity and Specificity of FDR Methods in
Neuroimaging

• Thomas Nichols, Wei Xie
• Department of Biostatistics
• University of Michigan
• ENAR 2006

2
Outline

• Background
• Functional Neuroimaging
• fMRI Characteristics
• Spatially smooth noise
• Spatially structured (blobby) signal
• FDR methods perform under smoothness
• BH FDR method
• Adaptive FDR methods
• Conclusions

3
Functional Neuroimaging and fMRI

• ? Neuronal Activity ? Blood Flow
• Many functional neuroimaging methods
  measurecorrelates of blood flow
• fMRI Functional Magnetic Resonance Imaging
• BOLD Blood Oxygenation Level Dependent effect
• ? Blood flow ? fMRI Signal

Tap fingers
Rest
4
Characteristics of Functional Neuroimaging
Statistic Images

• Spatial Structure of Noise
• Always have some correlationdue to physics and
  physiology
• Data often smoothed as pre-processing step
• Spatial Structure of Signal
• Sparse, sharp activations
• Spatially extended patterns of activation

5
DataExamples

• Smooth noise
• Small signal

Single subj. fMRI t237Oddball Target vs.
Standard Window -4 4
6
DataExamples

• Some-what smoothnoise
• Extensive signal

2 group FDG PET t82Norm AlzheimersWindow
-10 10
7
DataExamples

• Less smooth noise
• Subtle, sparse signal

1 group FDG PET t50AlzheimersFDG on
MMSEWindow -5 5
8
DataExamples

• Smooth noise
• Extended signal

50 subj. fMRI t49Oddball Target vs.
Standard Window -15 15
9
Statistic Image CharacteristicsWhy smooth!?

• Ideally no one would smooth
• Spatial structure of signalwould be explicitly
  modeled
• Why is smoothing common
• Part of the culture
• Increases sensitivity inexchange for spatial
  detail
• Imperfect intersubjectregistration
  necessitatessome smoothing

10
Evaluating FDR under Smooth Noise

• Benjamini Hochberg (1995) FDR (BH)
• Linear step-up procedure
• Developed assuming independence
• Controls FDR q m0/m ? q
• Conservative by null-fraction of the image
  (m0/m)
• Benjamini Yekutieli (2001)
• Showed BH is valid under positive dependence
• Valid, but possibly conservative
• Degree of conservativeness unknown
• How BH FDR perform under smoothness?

m0 Null Voxels m Total Voxels
11
Notation

• False Discovery Proportion
• FDP V / R IRgt0
• False Discovery rate
• FDR E( FDP )
• False Discovery Exceedance
• FDX P ( FDP gt q)

V False Positives R Positives
12
Control of FDX

• Common misinterpretation
• Threshold image with FDR at level-q
• Users assume q ? R false positives present,
  butE ( V / R IRgt0 ) q ?? E( V ) q?? E(
  R IRgt0 )
• FDX avoids this problem
• Threshold image w/ FDX at level-q with ?
  confidence FDX P(FDP gt q)
  ? ? 1 - FDX ?P(FDP q) ? 1-
  ?
• Can be confident no more than q?? R false
  positivesP ( V / R IRgt0 q) 1 - ? ?? P (
  V lt q ? R) 1 - ?

FDR control
Expd FPs
FDX control
Confidence of FPs being less than q ? R
13
Simulation Methods

• Benjamini Hochbergs FDR, q 0.05
• Images Gaussian T(6) images
• 10,000 realizations
• 32x32x32 voxels
• Smoothed with Gaussian kernels
• FWHM 0, 1.5, 3, 6 and 12 voxels
• Signal Spherical constant signals
• Radii 0, 1, 2, 4 and 8 voxels
• 0, 0.02, 0.10, 0.85 and 6.64 of the volume
• Signal magnitudes of 0.5, 1, 2, 4 8

14
Specificity Metrics

• FDR
• Small FDR, good specificity
• FDR gt q 0.05, method invalid
• FDR lt q 0.05, method inexact
• FDX
• Small FDX, good specificity
• FDX is controlled if this is less than 0.05
• No guarantee that FDX is controlled

15
Sensitivity Metrics

• Average Power
• The probability of detecting a given voxel with a
  (non-null) signal.
• Equivalently, the chance, averaged over all
  signal voxels, of a detection.
• Familywise Power
• The probability of detecting one or more signal
  voxels.
• Conditional Familywise Power
• Conditional on detecting at least one voxel of
  any kind at all, the probability of detecting one
  or more signal voxels.

16
ResultsSpecificity

• FDR falls below nominal as smoothness increases
• FDX happens to be controlled for small radius
  signals...

Signal mag. 2
17
ResultsSpecificity

• FDR falls below nominal as smoothness increases
• FDX happens to be controlled for small radius
  signals... but for larger magnitudes is not

Signal mag. 4
18
ResultsSensitivity

• Average power slightly increases with smoothness
• Familywise power actually falls with increasing
  smoothness

Signal mag. 2
19
ResultsSensitivity
Average Power
1
0.8

• Average power slightly increases with smoothness
• And of course w/ mag.
• Familywise power actually falls with increasing
  smoothness

0.6
P( True Detection at a Voxel )
0.4
0.2
0
0
2
4
6
8
10
12
Smoothness, Voxels FWHM
Signal mag. 4
20
Perplexing Results

• Smoothness ? Average Power ?
• Smoothness ? Familywise Power ?
• Strange?
• Of course, a familywise true positive can only
  occur if there are any positives
• Maybe no positives are more likely

21
ResultsSensitivity

• Condl Familywise Power grows with smoothness
• Ruling out the no-detection cases, the chance of
  any true positives does indeed grow

Signal mag. 2
22
ResultsSensitivity

• Condl Familywise Power grows with smoothness
• Ruling out the no-detection cases, the chance of
  any true positives does indeed grow

Signal mag. 4
23
Strangeness Discussion

• As smoothness grows
• Average power doesnt change
• Familywise power falls
• Conditional familywise power rises
• For high smoothness
• On data sets where you detect anything you are
  probably getting more true positives than at
  lower smoothness

24
Results Summary

• BH-FDR controls FDR
• When signal subtle, FDX is controlled
• Familywise power falls with smoothness But
  average power doesnt

25
Conservativeness of BH FDR Procedure
m0 Null Voxels m Total Voxels

• BH FDR
• Can control the FDR at precisely qm0/m ? q
• But conservative by the same factor m0/m
• If m0 is known, then we can improve BH procedure.
• Oracle BH Procedure
• Use qqm/m0 in BH procedure.
• Control FDR at precisely level q in independent
  case
• More powerful

26
Two-Stage FDR Procedures

• How to estimate the unknown m0 from the data
• Two-Stage FDR Procedures
• Benjamini, Krieger Yekutieli (2005)
• Idea m0 can be estimated from one-stage BH
  procedure
• 3 Procedures selected for simulation
• TST Two-Stage Linear Step-Up Procedure
• MTST Modified TST
• ASD Adaptive Step-Down Procedure

27
Two-Stage FDR Procedures

• TST Two-Stage Linear Step-Up Procedure
• Stage I Use BH at
• r1 rejected hypothesis
• r1 0 or r1 m, then stop
• Otherwise,
• Stage II Use BH at

28
Two-Stage FDR Procedures

• MTST Modified TST
• Stage I Use BH at level q, estimate m0
• Stage II Use BH at
• ASD Adaptive Step-Down Procedure
• Simplified version of a multi-stage procedure
• Question
• Will these adaptive procedures overcome the
  conservativeness under smoothness?

29
Results for FDR
30
ResultsSpecificity
Mag. 2Rad. 8

• Oracle has the best performance
• ASD is the most conservative
• Others perform similar to BH

31
ResultsSpecificity
Mag. 4Rad. 8

• ASD is the best only for large signal magnitude
• FDX is still not controlled

32
ResultsSensitivity
Mag. 2Rad. 8

• ASD is the most conservative one
• Others dont have much difference

33
ResultsSensitivity
Mag. 4Rad. 8

• ASD is the most conservative one
• Others dont have much difference

34
Results Summary

• Still have conservative control on FDR
• FDX is not controlled
• In general, the two-stage FDR procedures dont
  overcome the conservativeness for smoothed data

35
Conclusions

• BH-FDR
• Appropriate for smooth data
• Controls FDR
• Controls FDX, when signal subtle
• Familywise power falls with smoothness, but
  average power does not
• Two-stage FDR procedures
• Do not overcome the conservativeness under
  smoothness