Visual Illusions: understanding why we see what we do

1
Visual Illusionsunderstanding why we see what
we do

• Jenny Read

Laboratory of Sensorimotor Research National Eye
Institute
2
How do we see?
3
How do we see?
4
Johannes Kepler (1604)

• Vision occurs when the image of the whole
  hemisphere of the world that is before the eye is
  fixed on the reddish white concave surface of the
  retina.

5

• How the image or picture is composed by the
  visual spirits that reside in the retina, and
  whether it is made to appear before the soul or
  the visual faculty by a spirit within the hollows
  of the brain, or whether the visual faculty goes
  forth from the brain into the retina to meet this
  image - this I leave to be disputed by the
  physicists.

Express in modern language not a picture on the
retina (hence not upside down)
Kepler, Ad Vitellionem paralipomena, 1604
6
How do we see?

• Value of illusions in understanding vision
• when we see what is out there we learn only that
  our system is good when we see what is not out
  there we learn how it works!
• Some depend on properties of retina
• others on properties of visual cortex
• so different illusions teach us about different
  levels of visual processing.

7
Outline

• I. Overview of visual processing
• whirlwind tour of some key stages
• retinal ganglion cells Hermann grid
• primary visual cortex tilt aftereffect
• binocular vision stereopsis
• middle temporal cortex motion aftereffect
• II. New understanding of an old illusion
• motion/disparity coding the Pulfrich effect

Less detailed. Optical illusions related to
neural subtrate Lack of bitmap picture Pulfrich
neural substrate not as had been thought
8
I. Overview of visual processing
9
cross-section of the eye
iris
retina
lens
cornea
fovea
pupil
blind spot
optic nerve
10
(No Transcript)
11
cross-section of the eye
retina
optic nerve
12
Hermann grid
13
Hermann grid
14
cross-section of the retina
15
cross-section of the retina
retinal ganglion cell axons form the optic nerve
light entering eye
16
receptive field

• The response of a retinal ganglion cell is
  influenced by light only within a small region of
  the retina.

17
receptive field

• The area of the retina where stimulation can
  alter the response of a cell is called its
  receptive field.
• The receptive fields of retinal ganglion cells
  are circular.

18
receptive field
Simplify whole section
19
receptive field

ON region

• - - -
• - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - -

- -
OFF region

20
receptive field

ON region

• - - -
• - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - -

- -
OFF region

21
receptive field

ON region

• - - -
• - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - -

- -
OFF region

22
receptive field

ON region

• - - -
• - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - - - -
• - - - -

- -
OFF region

23
optimal stimulus

ON region
- -
OFF region
24
Hermann grid
25
Hermann grid

• close to an optimal stimulus
• so you perceive the image here as bright.

26
Hermann grid

• here the stimulus is less good
• the retinal ganglion cell fires less
• so you perceive the image here as dimmer

27
Hermann grid

• The Hermann grid illusion arises from the
  properties of retinal ganglion cells.

Design principle not luminance but contrast
28
after the retina
29
after the retina
left eye
right eye
30
after the retina
Greek letter chi C
optic chiasm
left eye
right eye
visual cortex
thalamus
31
after the retina
optic chiasm
left eye
right eye
visual cortex
thalamus
32
binocular vision
left eye only
right eye only
both eyes
33
the optic chiasm
optic chiasm
thalamus
34
the optic chiasm
left visual cortex right visual hemi- field
optic chiasm
right visual cortex left visual hemifield
thalamus
35
Isaac Newton (1665)
There are a vast multitud of these flender pipes
wch flow from the braine the one halfe through
the right fide nerve till they come of the
juncture where they are each divided into two
branches the one passing to ye right fide of ye
right eye, the other halfe fhooting through ye
juncture soe passing to ye right fide of the
left eye.
36
loss of left eye
left eye only
right eye only
both eyes
37
loss of left visual cortex
optic chiasm
left eye
right eye
visual cortex
thalamus
38
loss of left visual cortex
optic chiasm
visual cortex
thalamus
39
loss of left visual cortex
40
different visual areas
optic chiasm
Make visual areas more obvious Move this slide
til after v1 section
V1
V3
MT
visual cortex
V2
thalamus
V4
41
fibres project first to V1
optic chiasm
V1
primary visual cortex
thalamus
42
thalamic architecture
left lateral geniculate nucleus
43
divided into six layers
left lateral geniculate nucleus
44
inputs from different eyes are segregated into
different layers
left lateral geniculate nucleus
Make transparent
inputs from left eye inputs from right eye
45
Diagram of cell receiving input from both
Left and right eye inputs are first combined in V1
46

• Stereopsis
• How we see depth

47
random-dot stereogram
put the glasses on red over left eye, blue over
right eye.
This is an illusion because all the dots are on
the screen.
48
how the glasses work
Dont need
49
how the glasses work
screen
50
how the glasses work
screen
screen
51
how the glasses work
screen
52
how the glasses work
screen
53
how the glasses work
screen
54
how the glasses work
55
stereograms

• The illusion of depth with 3d glasses arises
  because the brain combines inputs from the two
  eyes in V1.

56
tilt after-effect
57
tilt after-effect
Stress importance of fixation
58
tilt after-effect
59
David Hubel Torsten Wiesel
60
Hubel Wiesels experiment
61
receptive field
62
Historic video of Hubel Wiesels original
experiments
Acknowledgement By kind permission of David
Hubel, image supplied by Tony Movshon at the New
York University Center for Neural Science, USA
63
orientation tuning
64
orientation tuning
population of neurons tuned to different
orientations
preferred orientation
same tuning
65
vertical stimulus
neuronal firing
preferred orientation
66
tilted stimulus
neuronal firing
preferred orientation
67
adaptation
neuronal firing
preferred orientation
68
vertical stimulus after adaptation
neuronal firing
preferred orientation
69
vertical stimulus after adaptation is
perceived as tilted
neuronal firing
preferred orientation
70
selective neurons

• V1 neurons are tuned to lots of stimulus
  properties for example
• orientation
• stereoscopic disparity
• size
• In other visual areas, cells are tuned to more
  complex properties.

71
Most V1 neurons are not sensitive to direction of
stimulus motion
72
middle temporal cortex (MT)
optic chiasm
V1
V3
MT
V2
thalamus
the motion area
V4
73
MT neurons are sensitive to direction of stimulus
motion
74
MT neurons are sensitive to direction of stimulus
motion
75
motion after-effect

• Neurons in MT are believed to be responsible for
  motion after-effects like the waterfall illusion.

76
motion after-effect
And also when persons turn away from looking at
objects in motion, such as rivers, and especially
those which flow very rapidly, things really at
rest are seen moving
?a? ?p? t?? ?????µ???? d? µetaß?????s??, ????
?p? t?? p?taµ??, µa??sta d? ?p??t?? t???sta
?e?t???, ?a??eta? t? ??eµ???ta ?????µe?a.
Aristotle, On Dreams, ca. 350BC
77
The Falls of Foyers
Photo by Hiroshi Ashida
78
Robert Addams (1834)
"Having steadfastly looked for a few seconds at a
particular part of the cascade, admiring the
confluence and decussation of the currents
forming the liquid drapery of waters, and then
suddenly directing my eyes to the left, to
observe the face of the sombre age-worn rocks
immediately contiguous to the water-fall, I saw
the rocky surface as if in motion upwards."
79
Stress fixation
Acknowledgement Movie supplied by George Mather
at the University of Sussex
80
simple explanation
Waterfall is stationary.
81
simple explanation
Activity in up/down sensors is balanced.
82
simple explanation
Perception is that waterfall is stationary.
83
simple explanation
Perception is that waterfall is stationary.
Waterfall is moving downwards.
84
simple explanation
activity
sensors
Perception is that image is stationary.
Activity is highly unbalanced.
85
simple explanation
activity
sensors
Perception is that waterfall is stationary.
Perception is that waterfall moves downwards.
86
simple explanation
Animate adaptation
activity
sensors
During this period the down sensors adapt to
the stimulus.
Perception is that waterfall is stationary.
Perception is that waterfall moves downwards.
87
simple explanation
activity
sensors
Perception is that waterfall is stationary.
Waterfall is stationary.
Perception is that waterfall moves downwards.
88
simple explanation
activity
activity
sensors
sensors
Activity is unbalanced due to sensor adaptation.
Perception is that waterfall is stationary.
Perception is that waterfall moves downwards.
89
simple explanation
activity
activity
sensors
sensors
Perception is that waterfall is stationary.
Perception is that waterfall moves upwards.
Perception is that waterfall moves downwards.
90
Schouten disk
Acknowledgement Movie supplied by Wim van de
Grind at the Department of Comparative Psychology
at Utrecht University
91
blindness to motion
92
end of part I

• retina and retinal ganglion cells
• optic chiasm
• primary visual cortex V1
• binocular neurons
• orientation tuning
• example of a higher brain area MT
• motion sensors

93

• Backbone one thing at a time
• Motion, disparity, orientation.
• If going to rep separately, how does it bring
  them together?
• Pulfrich effect depends on motion AND disparity
  together

94
Part II The Pulfrich effectA new explanation of
an old illusion
95
demo of Pulfrich illusion

• Pendulum going back and forth on screen
• You see it rotate in depth
• Filter delay

96
The Pulfrich effect
Reality Perception

• Illusory perception of a moving object when one
  eyes image is delayed

97
outline

• The Pulfrich effect the traditional explanation
  for it.
• Why this was rejected, and the new explanation.
• Why THIS cant work and our NEW explanation
• How physiology, computer simulations
  psychophysics are combined to understand an old
  puzzle.

Modern expl both motn disparity at same
time Why people believe that Neural substrate How
our investigations of neural substrate lead to a
reevaluation of this idea
98
Other illusions Pulfrich lacks pysiology We did
phys Reevaluate psychophsics and modeling
Modeling
Psychophysics
Physiology
99
The old explanation

• Spatial disparity and temporal delay are
  geometrically equivalent.

100
spatial disparity and temporal delay are
equivalent
fovea
101
spatial disparity and temporal delay are
equivalent
left eye
F
F
right eye
102
zero spatial disparity
Here the left and right images fall at the same
distance from the fovea in both retinae.
left eye
F
F
right eye
103
moving object with zero spatial disparity
104
moving object with zero spatial disparity
left eye
F
F
right eye
105
moving object with zero spatial disparity
left eye
F
Flag up the importance of space-time diagrams
reitnal position
time
F
right eye
106
moving object with zero spatial disparity
left eye
F
reitnal position
time
F
right eye
107
near disparity
Now the image in the right eye is always one
slot below the image in the left eye.
left eye
F
F
right eye
108
moving object with near disparity
Now the image in the right eye is always one
slot below the image in the left eye.
left eye
F
F
right eye
109
moving object with near disparity
Now the image in the right eye is always one
slot below the image in the left eye.
left eye
F
position
Mark disparity
time
F
right eye
110

• Then show mvoing obj with delay images on
  retina have disparity
• Then intro spa-time diag

111
Spatial disparity
position
time
112
Spatial disparity
Temporal delay
position
position
time
time
113
Spatial disparity
Temporal delay
position
position
time
time
114
moving object with zero spatial disparity but
with temporal delay
115
moving object with zero spatial disparity but
with temporal delay
1
2
2
1
Image 1 from the right eye reaches the brain at
the same time as image 2 from the left.
116
moving object with zero spatial disparity but
with temporal delay
1
2
2
1
The brain doesnt know about the time delay
so it deduces the object is closer than it
really is.
117
moving object with zero spatial disparity but
with temporal delay
1
3
2
3
2
118
moving object with zero spatial disparity but
with temporal delay
1
3
2
3
2
119
moving object with zero spatial disparity but
with temporal delay
1
4
2
3
4
3
120
moving object with zero spatial disparity but
with temporal delay
1
4
2
3
4
3
121
An object moving in the plane of fixation but
with interocular delay
122
is perceived as moving in a plane closer to the
observer.
123
For motion in the opposite direction, the object
is seen as further away.
124
For motion in the opposite direction, the object
is seen as further away.
125
For motion in the opposite direction, the object
is seen as further away.
126
Spatial disparity is geometrically equivalent to
interocular delay.
position
position
time
time
127
The Pulfrich effect
Reality Perception

• Illusory perception of a moving object when one
  eyes image is delayed

128
demo of strobe Pulfrich
129
Stroboscopic Pulfrich effect
Flashing stimulus, one eye lagging the other.
Slide horizontally Then animate Show non-equiv
with spatial disparity slide vertical Then
animate
space
time
now
130
Stroboscopic Pulfrich effect
Flashing stimulus, one eye lagging the other.
No spatial disparity, purely temporal delay.
space
no spatial disparity
time
interocular delay
131
demo of strobe Pulfrich

• Reprise demon
• No dipsarity!

132
Stroboscopic Pulfrich effect

• How to explain the perception of depth with a
  stroboscopic stimulus?
• So people suggested that the visual system may
  have sensors which detect both motion and
  disparity.

133
Receptive fields

• Remember that neurons only respond to stimuli
  within their receptive field.
• We have seen how the stimulus can be represented
  in a space-time diagram

position
time
134
Receptive field
space
time
135
Receptive field
space
time
neuronal spiking
time
136
This receptive field responds equally well to
motion in either direction.
space
time
neuronal spiking
time
137
For a direction-sensitive cell, we need a tilted
receptive field.
space
time
neuronal spiking
time
138
For a direction-sensitive cell, we need a tilted
receptive field.
space
time
neuronal spiking
time
139
For a direction-sensitive cell, we need a tilted
receptive field.
space
time
neuronal spiking
time
140
Joint motion-disparity detectors

• For binocular neurons, we need a receptive field
  in each eye.
• The difference in position of the receptive field
  in each eye defines the stereo disparity to which
  the cell responds.

141
F
F
142
receptive fields tuned to near disparity
left-eye receptive field on the retina
right-eye receptive field
143
receptive fields tuned to near disparity
left-eye receptive field on the retina
right-eye receptive field
144
receptive fields tuned to near disparity
left-eye receptive field on the retina
right-eye receptive field
145
receptive fields tuned to near disparity
left-eye receptive field on the retina
right-eye receptive field
146
So, a cell which is sensitive to both motion and
disparity would have receptive fields like this
space
Explain the two eyes receptive fields
time
left-eye receptive field
147
These sensors would normally respond to a moving
object with disparity
space
time
now
148
They will also respond to a moving object with no
disparity but an interocular delay.
space
time
now
149
Stroboscopic Pulfrich effect

• No spatial disparity, purely interocular delay.

Stroboscopic stimulus activates tilted RFs.
space
time
now
150
electrophysiology

• Single-unit recordings in awake monkey.
• We looked for joint motion-disparity sensors in
  monkey primary visual cortex.
• We found very few!
• Most cells in V1 encode only disparity. Fewer
  encode motion, and fewer still encode motion and
  disparity together.

151
tilted receptive fields
space
time
predicted by joint-encoding models
152
Problem

• Straight receptive fields are not
  direction-selective.
• So how can they give a different perception of
  depth for a stimulus moving to the left versus
  one moving to the right?

153
The Pulfrich effect
Reality Perception

• Illusory perception of a moving object when one
  eyes image is delayed

154
electrophysiology

• Perhaps your perception of depth in the
  stroboscopic Pulfrich stimulus due to a tiny
  subset of cells in primary visual cortex?
• (those with tilted receptive fields)
• You are ignoring a huge number of neurons telling
  you it is in the screen!
• Or, maybe we dont need tilted receptive fields
  after all.

155
The accepted wisdom
Straight receptive fields are not
direction-selective.
space
space
time
time
156
But!
they are direction-selective for stimuli with an
interocular delay.
space
space
time
time
157

• For the Pulfrich stimulus, because of the
  interocular delay, straight receptive fields
  are direction-selective as well as
  disparity-selective.
• They respond differently to the two directions of
  motion.
• When the pendulum moves to the right, it is seen
  in front
• when it moves left, it is seen behind.

158

• Thus, the sort of cells which are common in
  monkey V1 can explain the Pulfrich effect.

159
Modeling

• If we include what we know about how cortical
  cells respond over time, we can predict the
  amount of depth we expect subjects to perceive in
  the stroboscopic Pulfrich effect.

160

• If this explanation is correct, it makes
  different predictions about
• Stress prediction

161
Prediction
1
0.5
amount of depth perceived (normalized units)
0
-0.5
-1
-1
-0.5
0
0.5
1
amount of interocular delay (normalized units)
162
Testing the model

• Find out how much depth the interocular delay
  really does cause.

163
back to the psychophysics lab
164
Data
1
0.5
amount of depth perceived (normalized units)
0
-0.5
-1
-1
-0.5
0
0.5
1
amount of interocular delay (normalized units)
165
Summary

• We do indeed find an S-shaped curve.
• Not expected but PREDICTED by our model!
• Model successfully explains the illusion based on
  the properties of real neurons.
• This suggests that joint-encoding is not
  required.
• We conclude that the great majority of cells in
  V1 contribute to the Pulfrich illusion.

166
Conclusions

• Neurons initially encode different aspects of the
  stimulus (motion, depth) separately.
• Subsequently, these stimulus attributes are
  unified into a single percept.

167
Modeling
Psychophysics
Overlap?
Physiology
168
Modeling
Psychophysics
the strobe Pulfrich illusion
Physiology
169
the strobe Pulfrich illusion
Modeling
Psychophysics
the strobe Pulfrich illusion
Physiology
170
the strobe Pulfrich illusion
Modeling
Psychophysics
joint encoding of motion and depth?
Physiology
171
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
joint encoding of motion and depth?
Physiology
172
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
joint encoding rarely occurs.
Physiology
173
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
joint encoding rarely occurs.
Physiology
joint encoding rarely occurs
174
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
The illusion can be explained using separate
encoding.
Physiology
joint encoding rarely occurs
175
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
The illusion can be explained using separate
encoding
The illusion can be explained using separate
encoding.
Physiology
joint encoding rarely occurs
176
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
The illusion can be explained using separate
encoding
Prediction how perception should quantitatively
vary with interocular delay.
Physiology
joint encoding rarely occurs
177
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
The illusion can be explained using separate
encoding
How perception varies with interocular delay.
Physiology
joint encoding rarely occurs
178
the strobe Pulfrich illusion
joint encoding of motion and depth?
Modeling
Psychophysics
?
The illusion can be explained using separate
encoding
How perception varies with interocular delay.
How perception varies with interocular delay.
Physiology
joint encoding rarely occurs
179
Comments

• This old illusion has shed new light on how we
  see.
• how the brain matches up left and right images
• how it represents different stimulus attributes
• Insight gained by a multidisciplinary effort
• physiology, modeling, psychophysics.

180
Comments

• Visual illusions are a powerful tool for
  understanding vision.
• We understand many aspects of why they occur
• but theres still much more work to be done
• understanding why we see what we do.

181
(No Transcript)
182
illusions discussed

• illusion that we see the entire image at once
• blind spot
• Hermann grid
• Tilt aftereffect
• Random-dot stereogram
• Waterfall illusion
• Schouten disk