Advanced Vision Science

1
Advanced Vision Science

• Signal detection continued models and measures

2
Recap
The ROC-(response operating characteristic) curve
connects points in a Hit/FA- plot, obtained from
different criteria at constant sensitivity
ROC-curve characterizes signal/detector
independently of criterion
important sensitivity and criterion
theoretically independent -may be correlated
empirically!
3
ROC-curve
Same sensitivity (for this signal), several
criteria
hits
false alarms
4
Greater sensitivity ROC-curve farther from
diagonal
(Perfection would be all hits and no false
alarms)
hits
false alarms
5
Suggests two kinds of measures sensitivity
measures
Distance (overlap) between distributions e.g. d'
Area under ROC-Curve A
6
(No Transcript)
7
(No Transcript)
8
Interpretaton of A
Area theorem A is equivalent to proportion of
correct answers in 2AFC-experiment
Given 1 noise stimulus 1 signal (noise)
stimulus, Which is which?
9
Makes sense
No distinction between signal and noise A .50
10
Perfect distinction between signal and noise A ?
1.
11
Here A .75
Next proof of the Area theorem
12
Recap In general
fn
fs
8 PH ? fs(x)dx ?
H(?)
8 PFA ? fn(x)dx
?
FA(?)
x
0 ?
PH
? FA-1(PFA)
ROC-curve PH H(?)
HFA-1(PFA)
Specific model depends on fn and fs
PFA
13
Reinterpretation for 2A FC experiment
fs
fn
x
0 ?
alternatives correspond with two points on
x-axis xs an xn. PC p(xsgtxn), if xn ?,
p(xsgtxn) H(?)
Summate all H(?) for every ? Weiging for density
of ? fn(?)
8 PC ?
H(?)fn(?)d? -8
14
Area under Roc-curve
fn
fs
8 PH ? fs(x)dx ?
H(?)
8 PFA ? fn(x)dx
?
FA(?)
x
0 ?
PH
ROC-curve PH H(?) as function
of PFA FA(?)
1 A ?
H(?)dFA(?) 0
8 PC ?
H(?)fn(?)d? -8
PFA
15
derivation (optional)
1 A ? H(?)dFA(?)
0
(-fn(?)d?)
Two small jobs Integration limits and minus sign
dFA(?) -fn(?)d?
8 PC ?
H(?)fn(?)d? -8
16
derivation (optional)
1 A ? H(?)dFA(?)
0
(-fn(?)d?)
Limits if FA(?)PFA 0 then ? 8 if
FA(?)PFA 1 than ? -8
Reverse -fn ? fn
8 PC ?
H(?)fn(?)d? -8
17
Criterion measures
Every point of ROC-curve gives criterion/bias
(given this sensitivity)
PH
Direction coefficient of tangent at that point as
measure of bias/criterion S
PFA
.49
ROC-curve PH as function of PFA
dPH direction coefficient ----- dPFA
18
Criterion measures
8 PH ? fs(x)dx ?
8 PFA ? fn(x)dx
?
dPH dPH dPFA
----- ----- ------.dx dPFA dx
dPH dPFA fs - ----- fn - ------- dx
dx
(chain rule)
dPH dPH/dx ----- ------------
dPFA dPFA/dx
- fs fs ----- ----- - fn
fn
ß S
19
Recap
Sensitivity/ Discriminative power 1. Distance
between distributions 2. Area under ROC-curve
(A)
f
h
Criterium/Bias 1. f/h ß 2. Direction
coeficient S
How does it work out in practice?
20
Three approaches
1. Really hard work Obtain a lot of ROC-curve
points by inducing several criteria (pay-off,
signal frequency)
Each point (proportions Hits en False alarms)
involves many trials In total an enormous number
of trials!
Then measure A graphically
21
Certainly 0 1 2 3 4 5 Certainly No signal
a signal
Using a certainty scale implies several criteria
but also lots of trials
22

• Rough approximation
• Area measure for one point A'

Mean of these two areas A'
h
f
23
if H 1, F?0, F?1, then B'' -1
F
A measure for criterion/bias Griers B''
H
if H -F 1 then B'' 0
if F 0, H? 0, H?1 than B'' 1
H(1 - H) F(1 F) B'' sign(H -
F)------------------------ H(1 - H)
F(1 F)
24

• Introducing assumptions
• Even when you do obtain many points they often
  do not lie on a neat curve

Then you have to fit a curve an make (implicit)
assumptions about distributions
And you can reduce effort more assumptions?
less measurement
25
Normal distributions are popular, but there are
others!
Simplest model noise and signal distributions
normal, equal variance.
One point of ROC-curve (PH, PFA pair) is enough
26
Gaussians models preliminary
Standard normal curve M0, sd 1
Transformations F(z) ? P F-1(P) of Z(P) ? z
see tabels and standard software
27
PH
Roc-curve PH f(PFA)
PFA
?
zH
-
Z-transformation ROC-curve P ? z zH
f(zFA)
zFA
Good way to plot several points
28
Equal variance model z-plot ROC 45
PFA 1- F(?), F(-?), zFA -? PH 1
F(-(d' - ?)) F(d' ?), zH d' ?
zH zFA d' d' zH - zFA
0 ?
zH
d'
d'
45
zFA
29
Distributions for several values of d' and
corresponding ROC-curves
30
Criterion/bias
ß h/f f(zH)/f(zF)
f
h
To obtain a symmetrical measure a log
transformation is often appied log ß log
h log f
S
31
ß h/f f(zH)/f(zF)
f
h
?
c
Alternative c (alias ?center), distance (in sd)
between middle (where hf) and criterion
c -(d'/2 ?)
zFA -? d' zH - zFA
zH zFA c - ---------- 2
32
ß
c
Isobiascurves for ß en c
33
Unequal variance Gaussian model
e.g. ss 2sn
Assymmetrical ROC- curves
34
PH
Unequal variance model sn1, ss
PFA
zH
PFA F(-?), zFA -?
µs/ss
?
-µs
zFA
tg(?) 1/ss
35
?m does not distinguish large and small ss
Distance to origin analogous to d'
Measures
zH
ZH -ZFA
e a
de Oev2 da Oav2
zFA
O
?m
(Pythagoras and similar triangles)
36
PH
Area under Gaussian ROC-curve Az
Area theorem!!!
PFA
Gaussian 2AFC
n
s
PC p(xsgtxn) p(xs-xngt0)
37
PC p(xsgtxn) p(xs- xngt0)
-µs
Az according to area theorem!
38
PH
Area under Gaussian ROC-curve Az
(shown already)
PFA
tg
Az F(da/v2) Equal variances Az
Ad' F(d'/v2)
39
Survey of signal detection measures
General Rough Gaussian many pts
one pt sn ? ss sn ss
Sensitivity
Criterion/bias
Useful measures for performance and criteria of
people, machines, and systems
40
Finite State models
H H m FA cr
PH a ?(1-a)
PFA ?
41
PH a ?(1-a)
PFA ?
Theoretical ROC curve
uncertain ? ?Yes 1-? ?No
Detect Yes
a
?
a
high threshold
Cf correction for guessing MC-questions
42
Analogously a low threshold model Signal leads
alwas to uncertain state noise leads with P ß
to nondetect state (always NO) and else to
uncertain state.
1-ß
Uncertain ? ?Yes 1-? ?No
Nondetect No
ß
43
A combined three state model
N O D
44
What are the costs of missing a wapon or
explosive at an airfield?
What are the costs of a false alarm?
What are the costs of screening (delays included)?
45
Costs and benefits Pay-off matrix
NB. Here C is a positive number one false alarm
costs 5 euro
no yes
S(N) N
EV p(Hit)VHit- p(Miss)CMiss p(CR)VCR -
p(FA)CFA
p(s)PH VHit (1-PH)CMiss
p(n)(1-PFA)VCR - PFACFA
Cf doing nothing EV p(n)VCR p(s)CMiss NB.
not in these formulas Screening is not for free!
46
An optimal decision in uncertainty
Set het criterion at a value of x (xc) at which
expected value/utility of Yes is equal to
expected value/utility of No
xc
x
EV(Yesxc) EV(Noxc)
47
Costs CFA positive!
EV(Yesxc) EV(Noxc)
VHit p(Hit) CFA p(FA) VCRp(CR) -
CMissp(Miss)
VHit p(signalxc) CFA p(noisexc)
VCRp(noisexc) - CMissp(signalxc)
p(signalxc) VCR CFA ----------------
--------------- p(noisexc) VHit CMiss
But how do we know these?
48
p(signalxc) VCR CFA ----------------
--------------- p(noisexc) VHit CMiss
We want these
p(xnoise)
We know these (in principle)
p(xsignal)
required a way to get from p(AB) to p(BA)
Bayes rule !
49
p(AB) p(BA) p(A)
--------- ----------
------- p(AB) p(BA) p(A) (odds
form)
Applied to signal detection
p(signalxc) --------------- p(noisexc)

50
p(signalxc) VCR CFA ----------------
--------------- p(noisexc) VHit CMiss
Bayes
p(xcsignal) p(signal) VCR CFA
-------------- --------- --- ---------------
p(xcnoise) p(noise) VHit CMiss
P(xcsignal) p(noise) VCRCFA
---------------- ----------- -----------
p(xcnoise) p(signal) VHitCMiss
ß prior odds payoff matrix
(of noise)
51
P(xcsignal) p(noise) VCRCFA
---------------- ------------ -----------
p(xcnoise) p(signal) VHitCMiss
So an ideal observer, knowing prior odds and
pay-off matrix, can compute an optimal criterion.
People are not very good at arithmatic, but adapt
reasonably well to pay-off matrices and prior odds