Statistical Nonparametric Mapping SnPM Thresholding without many assumptions

1
Statistical Nonparametric Mapping -
SnPMThresholding without (m)any assumptions

• Thomas Nichols, Ph.D.
• Director, Modelling GeneticsGlaxoSmithKline
  Clinical Imaging Centre
• http//www.fmrib.ox.ac.uk/nichols
• ICN SPM Course May 8, 2008

2
Overview

• Multiple Comparisons Problem
• Which of my 100,000 voxels are active?
• SnPM
• Permutation test to find threshold
• Control chance of any false positives (FWER)

3
Nonparametric InferencePermutation Test

• Assumptions
• Null Hypothesis Exchangeability
• Method
• Compute statistic t
• Resample data (without replacement), compute t
• t permutation distribution of test statistic
• P-value t gt t / t
• Theory
• Given data and H0, each t has equal probability
• Still can assume data randomly drawn from
  population

4
Nonparametric Inference

• Parametric methods
• Assume distribution ofstatistic under
  nullhypothesis
• Needed to find P-values, u?
• Nonparametric methods
• Use data to find distribution of statisticunder
  null hypothesis
• Any statistic!

5
Permutation TestToy Example

• Data from V1 voxel in visual stim. experiment
• A Active, flashing checkerboard B Baseline,
  fixation
• 6 blocks, ABABAB Just consider block
  averages...
• Null hypothesis Ho
• No experimental effect, A B labels arbitrary
• Statistic
• Mean difference

6
Permutation TestToy Example

• Under Ho
• Consider all equivalent relabelings

7
Permutation TestToy Example

• Under Ho
• Consider all equivalent relabelings
• Compute all possible statistic values

8
Permutation TestToy Example

• Under Ho
• Consider all equivalent relabelings
• Compute all possible statistic values
• Find 95ile of permutation distribution

9
Permutation TestToy Example

• Under Ho
• Consider all equivalent relabelings
• Compute all possible statistic values
• Find 95ile of permutation distribution

10
Permutation TestToy Example

• Under Ho
• Consider all equivalent relabelings
• Compute all possible statistic values
• Find 95ile of permutation distribution

0
4
8
-4
-8
11
Permutation TestStrengths

• Requires only assumption of exchangeability
• Under Ho, distribution unperturbed by permutation
• Allows us to build permutation distribution
• Subjects are exchangeable
• Under Ho, each subjects A/B labels can be
  flipped
• fMRI scans not exchangeable under Ho
• Due to temporal autocorrelation

12
Permutation TestLimitations

• Computational Intensity
• Analysis repeated for each relabeling
• Not so bad on modern hardware
• No analysis discussed below took more than 3
  hours
• Implementation Generality
• Each experimental design type needs unique code
  to generate permutations
• Not so bad for population inference with t-tests

13
MCP SolutionsMeasuring False Positives

• Familywise Error Rate (FWER)
• Familywise Error
• Existence of one or more false positives
• FWER is probability of familywise error
• False Discovery Rate (FDR)
• R voxels declared active, V falsely so
• Observed false discovery rate V/R
• FDR E(V/R)

14
FWER MCP Solutions

• Bonferroni
• Maximum Distribution Methods
• Random Field Theory
• Permutation

15
FWER MCP Solutions Controlling FWER w/ Max

• FWER distribution of maximum
• FWER P(FWE) P(One or more voxels ? u
  Ho) P(Max voxel ? u Ho)
• 100(1-?)ile of max distn controls FWER
• FWER P(Max voxel ? u? Ho) ? ?

u?
16
FWER MCP Solutions

• Bonferroni
• Maximum Distribution Methods
• Random Field Theory
• Permutation

17
Controlling FWER Permutation Test

• Parametric methods
• Assume distribution ofmax statistic under
  nullhypothesis
• Nonparametric methods
• Use data to find distribution of max
  statisticunder null hypothesis
• Again, any max statistic!

18
Permutation Test Exchangeability

• Exchangeability is fundamental
• Def Distribution of the data unperturbed by
  permutation
• Under H0, exchangeability justifies permuting
  data
• Allows us to build permutation distribution
• Subjects are exchangeable
• Under Ho, each subjects A/B labels can be
  flipped
• Are fMRI scans exchangeable under Ho?
• If no signal, can we permute over time?

19
Permutation Test Exchangeability

• fMRI scans are not exchangeable
• Permuting disrupts order, temporal
  autocorrelation
• Intrasubject fMRI permutation test
• Must decorrelate data, model before permuting
• What is correlation structure?
• Usually must use parametric model of correlation
• E.g. Use wavelets to decorrelate
• Bullmore et al 2001, HBM 1261-78
• Intersubject fMRI permutation test
• Create difference image for each subject
• For each permutation, flip sign of some subjects

20
Permutation TestOther Statistics

• Collect max distribution
• To find threshold that controls FWER
• Consider smoothed variance t statistic
• To regularize low-df variance estimate

21
Permutation TestSmoothed Variance t

• Collect max distribution
• To find threshold that controls FWER
• Consider smoothed variance t statistic

t-statistic
variance
22
Permutation TestSmoothed Variance t

• Collect max distribution
• To find threshold that controls FWER
• Consider smoothed variance t statistic

SmoothedVariancet-statistic
mean difference
smoothedvariance
23
Permutation TestExample

• fMRI Study of Working Memory
• 12 subjects, block design Marshuetz et al (2000)
• Item Recognition
• ActiveView five letters, 2s pause, view probe
  letter, respond
• Baseline View XXXXX, 2s pause, view Y or N,
  respond
• Second Level RFX
• Difference image, A-B constructedfor each
  subject
• One sample, smoothed variance t test

24
Permutation TestExample

• Permute!
• 212 4,096 ways to flip 12 A/B labels
• For each, note maximum of t image
• .

25
Permutation TestExample

• Compare with Bonferroni
• ? 0.05/110,776
• Compare with parametric RFT
• 110,776 2?2?2mm voxels
• 5.1?5.8?6.9mm FWHM smoothness
• 462.9 RESELs

26
uRF 9.87uBonf 9.805 sig. vox.
uPerm 7.67 58 sig. vox.
t11 Statistic, RF Bonf. Threshold
t11 Statistic, Nonparametric Threshold
27
Does this Generalize?RFT vs Bonf. vs Perm.
28
RFT vs Bonf. vs Perm.
29
Reliability with Small Groups

• Consider n50 group study
• Event-related Odd-Ball paradigm, Kiehl, et al.
• Analyze all 50
• Analyze with SPM and SnPM, find FWE thresh.
• Randomly partition into 5 groups 10
• Analyze each with SPM SnPM, find FWE thresh
• Compare reliability of small groups with full
• With and without variance smoothing
• .

30
SPM t11 5 groups of 10 vs all 505 FWE
Threshold
Tgt10.93
Tgt11.04
Tgt11.01
10 subj
10 subj
10 subj
2 8 11 15 18 35 41 43 44 50
1 3 20 23 24 27 28 32 34 40
9 13 14 16 19 21 25 29 30 45
Tgt10.69
Tgt10.10
Tgt4.66
10 subj
10 subj
all 50
4 5 10 22 31 33 36 39 42 47
6 7 12 17 26 37 38 46 48 49
31
SnPM t 5 groups of 10 vs. all 505 FWE
Threshold
Tgt7.06
Tgt8.28
Tgt6.3
10 subj
10 subj
10 subj
2 8 11 15 18 35 41 43 44 50
1 3 20 23 24 27 28 32 34 40
9 13 14 16 19 21 25 29 30 45
Tgt4.09
Tgt6.49
Tgt6.19
10 subj
10 subj
all 50
4 5 10 22 31 33 36 39 42 47
6 7 12 17 26 37 38 46 48 49
32
SnPM SmVar t 5 groups of 10 vs. all 505 FWE
Threshold
Tgt4.69
Tgt5.04
Tgt4.57
10 subj
10 subj
10 subj
2 8 11 15 18 35 41 43 44 50
1 3 20 23 24 27 28 32 34 40
9 13 14 16 19 21 25 29 30 45
Tgt4.84
Tgt4.64
10 subj
10 subj
4 5 10 22 31 33 36 39 42 47
6 7 12 17 26 37 38 46 48 49
33
Conclusions

• t random field results conservative for
• Low df smoothness
• 9 df ?12 voxel FWHM 19 df lt 10 voxel
  FWHM(based on Monte Carlo simulations, not
  shown)
• Bonferroni not so bad for low smoothness
• Nonparametric methods perform well overall

34
Monte Carlo Evaluations

• Whats going wrong?
• Normality assumptions?
• Smoothness assumptions?
• Use Monte Carlo Simulations
• Normality strictly true
• Compare over range of smoothness, df
• Previous work
• Gaussian (Z) image results well-validated
• t image results hardly validated at all!

35
Monte Carlo EvaluationsChallenges

• Accurately simulating t images
• Cannot directly simulate smooth t images
• Need to simulate ? smooth Gaussian images
• (? degrees of freedom)
• Accounting for all sources of variability
• Most M.C. evaluations use known smoothness
• Smoothness not known
• We estimated it residual images

36
Monte Carlo Evaluations

• Simulated One Sample T test
• 32x32x32 Images (32767 voxels)
• Smoothness 0, 1.5, 3, 6,12 FWHM
• Degrees of Freedom 9, 19, 29
• Realizations 3000
• Permutation
• 100 relabelings
• Threshold 95ile of permutation distn of maximum
• Random Field
• Threshold u E(?u Ho) 0.05
• Also Gaussian

FWHM
37
FamilywiseErrorThresholds
Inf. df

• RFT valid but conservative
• Gaussian not so bad (FWHM gt3)
• t29 somewhat worse

29df
more
38
FamilywiseRejectionRates
Inf df

• Need gt 6 voxel FWHM

29 df
more
39
19 df
FamilywiseErrorThresholds

• RF Perm adapt to smoothness
• Perm Truth close
• Bonferroni close to truth for low smoothness

9 df
more
40
FamilywiseRejectionRates
19 df

• Bonf good on low df, smoothness
• Bonf bad for high smoothness
• RF only good for high df, high smoothness
• Perm exact

9 df
more
41
FamilywiseRejectionRates
19 df

• Smoothness estimation is not (sole) problem

9 df
cont
42
Performance Summary

• Bonferroni
• Not adaptive to smoothness
• Not so conservative for low smoothness
• Random Field
• Adaptive
• Conservative for low smoothness df
• Permutation
• Adaptive (Exact)

43
Understanding Performance Differences

• RFT Troubles
• Multivariate Normality assumption
• True by simulation
• Smoothness estimation
• Not much impact
• Smoothness
• You need lots, more at low df
• High threshold assumption
• Doesnt improve for ?0 less than 0.05 (not shown)

HighThr
44
Conclusions

• t random field results conservative for
• Low df smoothness
• 9 df ?12 voxel FWHM 19 df lt 10 voxel FWHM
• Bonferroni surprisingly satisfactory for low
  smoothness
• Nonparametric methods perform well overall
• More data and simulations needed
• Need guidelines as to when RF is useful
• Better understand what assumption/approximation
  fails

45
References

• TE Nichols and AP Holmes.Nonparametric
  Permutation Tests for Functional Neuroimaging A
  Primer with Examples. Human Brain Mapping,
  151-25, 2002.
• http//www.sph.umich.edu/nichols

MC ThrRslt
EstSmCf
MC P Rslt
Data ThrRslt
46
(No Transcript)
47
Permutation TestExample

• Permute!
• 212 4,096 ways to flip A/B labels
• For each, note max of smoothed variance t image
• .

Permutation DistributionMax Smoothed Variance t
Maximum Intensity Projection Threshold Sm. Var. t
48
Back