G' Cowan

1
Report from the Statistics Forum
ATLAS Analysis Strategies Workshop CERN, 6
September, 2007
Glen Cowan Physics Department Royal Holloway,
University of London g.cowan_at_rhul.ac.uk www.pp.rhu
l.ac.uk/cowan
2
The ATLAS Statistics Forum
Initiated at Statistics Workshop in January
07 Statistics Forum started in March (Karl
Jakobs) Since then, meetings monthly (plus 3
joint meetings with CMS)
3
Current Membership
Conveners Eilam Gross, Glen Cowan Statistics
Experts GDC, Kyle Cranmer, Tom Junk, Bill
Murray, Alex Read Tool developers Bill Quayle,
Krzysztof Ciba Representatives from ATLAS
groups Higgs Eilam Gross Guillaume
Unal SUSY Dan Tovey SM Maarten
Boonekamp Exotics Samir Ferrag B-Physics
Eduard Burelo Top Yoram Rozen Heavy Ions,
... Ex officio Ian Hinchliffe Karl Jakobs
4
Mandate
Define ATLAS Statistics Standards in contact
with Physics Coordination and ATLAS
groups. Develop, validate and approve standard
statistical tools in close contact with the
software group. Represent ATLAS in discussions on
statistical issues with CMS. Discuss and make
recommendations on statistical issues such as
blind analyses, model independent searches, etc.
5
Web page
https//twiki.cern.ch/twiki/bin/view/Atlas/Statist
icsTools
6
Hypernews
https//hypernews.cern.ch/HyperNews/Atlas/get/phys
ics-statistics.html
7
cf. BaBar Statistics Group Hypernews
Detailed discussions
Heavy traffic
8
Some ongoing activities
ATLAS Statistics Book Higgs combination
(different channels ATLAS CMS) Software
development (RooStat) Sequential analysis
9
Twiki, i.e., a Living Document (thanks,
Krzysztof) Core authors assigned, detailed
outline discussed worked examples, particularly
related to systematics FAQ, recommendations Goal
first draft by October (need to work hard...)
10
Higgs Combination
Higgs suggested as a first major exercise for the
group, but we would like to undertake similar
efforts with other Physics Groups. Many people
involved see next talk by Eilam. In particular,
frequentist combination based on
profile likelihood very advanced (Kyle, Eilam,
Ofer, ...) But a brief comment on the Bayesian
methods Bayesian limits already widely used
e.g. in CDF (?Tom J.). We are also investigating
use of Bayes factors as a means of quantifying
significance of a discovery
Higgs
posterior odds
prior odds
Bayes factor B10
no Higgs
11
Higgs combination based on event counts
N independent channels, count ni events in search
regions
Constrain expected background bi with sideband
measurements
Expected number of signal events (m is global
parameter, m 1 for SM).
Consider a fixed Higgs mass and assume SM
branching ratios Bi. Suggested method
constrain m with limit mup consider mH excluded
if upper limit mup lt 1.0. For discovery, compute
Bayes factor for H0 m 0 vs. H1 m 1
12
Bayes factor ( ratio of integrated likelihoods)
B10 at m 1 gives change in odds of SM Higgs vs.
no Higgs in light of data, i.e., measures of
weight of evidence provided by data in support
of one hypothesis over another.
B10
Here B10 and profile likelihood ratio close
comparisons ongoing.
13
Statistical software
Bill Quayle, Kyle Cranmer (from Statistics Forum)
developers of TMVA (Andreas Hoecker et
al.), RooFit (Wouter Verkerke et al.),
MadTools/StatTools (Wisconsin group), ... (see
e.g. Bill Quayles 15 March 07 talk) Some
issues ATLAS specific (private) or HEP-wide
(public)? Use FrameworkXXX or, integrate
ComponentXXX into an existing framework? Distrib
ution (phystat.org, bundle with ROOT?)
14
RooFit/RooStats
Wouter Verkerke, Kyle Cranmer
Developments reported in ATLAS/CMS Forum, i.e.,
HEP-wide, not ATLAS-specific software.
E.g. Combination Toy Model (W. Verkerke -- 17
July 2007 ATLAS/CMS meeting)
Combined
Profile likelihood
Atlas
? talk by Wouter
CMS
15
(Renaud Bruneliere, 7 June and 3 July 07)
Sequential analysis
Looking at data as they are recorded and compute
a new p-value or likelihood ratio test every 10
pb-1. What is probability that p-value (or LR
test) lt 0.01 versus number of data looks ?
p-Value result versus L when H0 is true
P(at least 1 p-Value lt 0.01) versus L when H0 is
true
b 200 fb
Likelihood ratio test
p-value

• looking 1 time at data
  p(p-value lt 0.01H0) 0.01
• looking sequentially 100 times at data p(at
  least 1 p-value lt 0.01H0) 0.08 !

? Look-elsewhere effect in time. Need to
understand how to best quantify / deal with this!
16
Conclusions, thoughts
Statistics Forum working since March, several
projects ongoing Statistics Book Higgs
combination Software tools Sequential
analysis Want to increase interaction with
Physics Groups, If your group has a statistics
issue/problem, is developing a statistical
tool, etc. come tell us. (Also we will come to
you...)