Fully-frequentist handling of nuisance parameters in confidence limits

1
Fully-frequentist handling of nuisance parameters
in confidence limits
more details in http//www.samsi.info/200506/astr
o/workinggroup/phy/ and http//www.physics.ox.ac.
uk/phystat05/proceedings/files/Punzi_PHYSTAT05_fin
al.pdf

• Giovanni Punzi
• University and INFN-Pisa
• giovanni.punzi_at_pi.infn.it

2
Method Neyman constructionprojection

• Build a confidence band, by treating the nuisance
  parameter as any other parameter p( (x, e)
  (µ, ?) ) p(xµ, ?)p(e ?)
• Evaluate CR in (µ, ?) from the measurement (x0,
  e0)
• Project onto µ space to get rid of information on
  ?
• Properties
• Conceptually clean and simple.
• Guaranteed coverage.
• Well-behaved the intervals grow with increasing
  uncertainty
• So why doesnt everybody use it ?

Confidence band
observables
(x,e)
(x0,e0)
(m, ?)
parameters
(x0,e0)
e
overcoverage
m
Confidence interval on µ
3
Issues with the Neymanprojection

• Overcoverage
• Can be huge if you are not careful
• Behavior when ?? 0
• The limit may not be what you like it to be. This
  is an issue of other methods, too.
• Shape of intervals
• Hard to control the shape of the projections.
  e.g. not obvious how to obtain upper limits.
• CPU requirements
• Need clever algorithms
• Multidimensional complexity

4
Upper limits, naïve version (x ordering)
Coverage
?
µ
Max coverage
Not very exciting
Min coverage
5
Past experience with projection method

• No overwhelming enthusiasm in HEP. Most often
  replaced by some frequentist approximation
• Profile Likelihood
• Smearing/averaging on the nuisance parameter
  (Cousins-Highland)
• There were a few success stories
• Improved solution of the classical Poisson ratio
  problem coordinated projections. R. Cousins,
  Nucl. Instrum. Meth. A417 391 (1998)
• Final analysis of CHOOZ experiment limits with
  systematics, prof-LR orderingG.P. and
  G.Signorelli at PHYSTAT02 Eur. Phys.J.
  C27331(2003)
• Poissonbackground with uncertainty Hypothesis
  Test with prof-LR orderingK. Cranmer at
  PHYSTAT03, 05
• Poisson, uncertain efficiency confidence
  region/any ordering/unspecified error
  distribution G.P. at PHYSTAT05

6
My choice of Ordering Rule Phystat05

• I looked for an ordering rule in the space that
  would give me a desired ordering rule f0(x) in
  the interesting parameter space.
• I want the ordering to converge to the local
  ordering for each e, when ??0
• The ordering is made not to depend on ?, as I am
  trying to ignore that information. This is handy
  because is saves computation

A

• Order in such a way as to integrate the same
  conditional probability at each e
• This gives the ordering function
• Where f0(x) is the ordering function adopted for
  the 0-systematics case
• N.B. I also do some clipping, mostly for
  computational reasons
• when the tail probability for e becomes too small
  for given ?, I force it to low priority

7
The benchmark problem of group A

• Poisson(background), with a systematic
  uncertainty on efficiency

• e is a measurement of the unknown efficiency ?,
  with resolution ?
• ? is the efficiency (a normalization factor,
  can be larger than 1).
• NOTES
• The benchmark specifies G to be a Poisson, to
  avoid blow-up of Bayesian solutions, but the
  frequentist method allows much more freedom
• You can simply use a Gaussian G, as e0 or elt0 is
  not a problem. (Negative values can actually
  occur in practice, e.g., due to
  background-subtracted measurements)
• You can have a uniform G
• You can even have NO specified distribution, just
  a RANGE on ?
• You can choose 1-sided, Central intervals, or F-C
  at your pleasure.

8
Upper limits coverage, my ordering
Coverage
coverage
?
Max coverage
µ
Max coverage
Pretty good ! No overcoverage beyond
discretization ripples
Min coverage
9
Unified limits, my ordering rule
Coverage
?
Max/Min coverage
Max/Min coverage
Average coverage
10
Unified limits (F-C)

• Lets say we want Unified limits. The exact
  procedure, would be to evaluate the probability
  distribution of LR for each e, and order on it
  over the space.
• However, the LR theorem tells you that the
  distribution of the LR is independent of the
  parameters gt can use the LR itself as
  approximation of the ordering function.
• Re-discover profile-LR method as an approximation
  of the proposed ordering rule

11
Unified limits , LRprof ordering
Coverage
?
µ
Max/Min coverage
12
Coverage
LRprof ordering
proposed ordering
µ
Max coverage
average coverage
Max/Min coverage
Average coverage
Yes ! You can do even better than LRprof !
You do even better than LRprof !
Difference is unimportant in itself, but a good
hint that this is the right track !
13
Limit for 0-syst

• The problem gets solved in an easy and natural
  way - can have a perfect match built into the
  algorithm. The only requirement is not to let the
  grid step become too small

14
Great flexibility easy to work with unspecified
systematic distribution

• Very often, the only information we have about a
  nuisance parameter is a range eminlt e lt emax
• Tipical example theoretical parameters. But many
  others all cases where no actual experimental
  measurement is done of the nuisance parameter.
• Automatically handled in the current algorithm
  (at no point a subsidiary measurement is
  required, it just enters as part of the pdf). It
  is fast, and free from transition to continuum
  issues.
• It is actually much more convenient to treat
  error in this way.
• N.B. the result is NOT equivalent to varying the
  parameter within the range, and take the most
  conservative interval.

15
Unspecified error distribution

• Example 0.85lt ? lt1.15

16
Coverage for the no-distribution case
Coverage
17
Compare unknown distribution with flat
Assuming a uniform distribution provides you
with more information than just a range gt get
tighter limits as expected
Flat gt range
Flatlt range
18
Computational Aspects

• The ordering algorithm being independent of the
  nuisance parameters makes it much more convenient
  to perform the calculations
• one can order just one time for each value of mu,
  and then go through the same list every time,
  just skipping the elements that are clipped off
  for that particular value of the nuisance
  parameter.
• The clipping reduces the amount of space to be
  searched
• The chosen ordering naturally provides acceptance
  regions that are slowly varying with the
  subsidiary measurementgt need to sample only a
  very small number of points. (used 10, but 2
  would be enough)
• For the same reasons, when multiple nuisance
  parameters are present, large combinatorics is
  unnecessary sampling the corners of the
  hyperrectangle is sufficient (plus maybe
  additional random points).
• The discreteness limitation further reduces the
  time needed.
• Uncertainty given as ranges are extremely fast to
  calculate - one might willingly use this method
  for fast evaluation. They are also immune to the
  0-limit problem, so need no adjustments.

19
Conclusion

• waiting for THE MATRIX