Exact tests from estimation followed by maximisation

1
Exact tests from estimation followed by
maximisation

• Chris J. Lloyd MBS
• Max V Moldovan

2

• Preview

• For discrete models (such as 2x2 table)
• Well known approximate tests (eg. Chi-square,
  McNemar, CLT-LR inferiority) can have poor
  frequentist properties when viewed exactly.
• Maximisation of the P-value is the only
  theoretically sound way to produce an exact test
  with guaranteed properties.
• Exact tests can have poor efficiency and poor
  logical properties if the basic test statistic is
  bad.

3

• Preview

• For discrete models (such as 2x2 table)
• Estimating the nuisance parameter in an exact
  tail probability gives a more pivotal P-value.
• Maximising after estimation gives an exact, more
  powerful and more pivotal test.
• Can be computationally demanding but monte carlo
  methods should be available.

4
Emerson Moses, Biometrics, 1985

• Instability of P-values Example

5
Emerson Moses, Biometrics, 1985

• Instability of P-values Example

Go though all 48x28413632 possible contingency
tables and accumulate probability of all those
with T2.0846 Pr(T2.0846) depends on p
6
Emerson Moses, Biometrics, 1985

• Instability of P-values Example

Plot of Pr(T2.0846p) versus p (x114,x247)
7

• Full Maximisation

For testing H0??0, nuisance parameter ? using
(approx) generating P-value P(Y) P?(y)
Pr(P(Y)P(y) ?0,?) versus ? is called
significance profile. The maximised P-value
(Bickel and Doksum) is P(y)sup?P?(y.
8

• Full Maximisation

P is the unique and smallest exact P-value that
orders samples in the same order as P. Full
maximisation is required. If we dont like the
resulting test then change the ordering i.e.
change test statistic T or approximate P-value P!
9
Emerson Moses, Biometrics, 1985

• Instability of P-values Example

Plot of Pr(T2.0846p) versus p (x114,x247)
0.0611
10

• Estimation Pre-pivoting

Start with approximate P whose profile
is Pr(P(Y)P(y)?0,?) Substitute to obtain
estimated P-value
Not exact. Can be made so by maximisation of the
profile of the new statistic.
Beran, JASA 1988 Storer Kim, JASA, 1990 Kang
Chen, SIM, 2000 Skipka, Munk Freitag, CSDA, 2004
11
Emerson Moses, Biometrics, 1985

• Estimation Pre-pivoting

Profile(darker) of estimated P-value is flatter
so test is more pivotal. Maximum value very close
to estimated P-value of 0.0368 so close to exact.
12

• Matched Binary Pairs

Interest ?p01-p10 Nuisance ?p01p10
13

• Matched Binary Pairs

TX10X01 is Binomial (n,?) X01Tt is
Binomial (t,?)
14

• Matched Binary Pairs

Common Test statistics Conditional Binomial
sign P-value McNemar (1947) (X01-X10)/vT
(CLT) LR statistic (CLT)
15
Jones and Kenward, 1987 (Analgesic dosage cross
over trial)

• Correlated Proportions

Improvement 61 at low dose, 69 at high dose T24
discordant individuals (28), x0116 in favour
of high dose (n86 not used) McNemar(16-8)/v241.
633 so P0.0512
16
Jones and Kenward, 1987 (Analgesic dosage cross
over trial)

• Matched Binary Pairs

profile
Profile of estimated P-value
17

• Numerical Study

Numerical assessment of power (McNemar) Parameter
space 0??1. Powers evaluated at 10,000
equally spaced points. For 100 of points power
of EM at least at good as M and B.
18

• Computation

Depends on cardinality of sample space N
(eg3665) Estimated O(logN) times small
constant Partial maximisation O(logN) times
larger constant that increases with more nuisance
parameters Maximisation Slightly more than
partial. Estimate then maximise O(NlogN) but
more reliable Monte-Carlo methods needed for
large N
19

• Conclusion

Approx tests have poorer properties than you
think! E-Step leads to less sensitive
profile. EM give more pivotal and powerful
tests. M step is often unnecessary in practice
Further work Computation Alternatives to E
Second order asymptotic e.g. r R functions
available from authors
20

• Correlated Proportions

Common Test statistics Conditional Binomial
sign P-value McNemar (1947) (X01-X10)/vT
(CLT) LR statistic (CLT) Approximate tests
all free of n!? Maximised McNemar (Suissa and
Shuster,1994) Partial McNemar (Berger and Sidik,
2003)
21

• Numerical results

Dependence on sample size n? (x0111, t15)