CNS186Attention Shimojo

1
CNS186-Attention
(Shimojo) Attention Psychophysics
2
1. Why bother attention? Attention can be
defined as local and transient selection and
facilitation of sensory information processing,
by a stimulus/cue or voluntary effort. -It seems
to be an interface between (a) vision
other modalities, (b) vision motor,
(c) bottom-up top-down, and (d) conscious
subconscious processes. -In vision, it takes a
critical role in integration of various
attributes to form object representation.
(binding)
3
2. Classical concepts Classical concepts on
attention were initially developed in auditory
perception, and then generalized to
visual. -Filtering (Broadbent, 1958) cf.
dichotic listening technique (Cherry), cocktail
party effect -Selection late vs. early
(Treisman, 1964) -Limited capacity/processing
resource (Kahneman, 1973) -Spotlight, searchlight
metaphor (Jonides, 1980 Crick, 1984)
4
3. Tasks The main difficulty of studying
attention psychophysically is in that attention
itself is invisible. However, cleverly designed
tasks (as described below) have been employed to
make the effect of attention visible. -Detection
with pre-cue (Posner, 1980 Posner Cohen,
1984) cf. cost/benefit tradeoff
(Posner) -Visual search (Treisman Gelade, 1980
Treisman, 1982) -Texture segregation (Beck,
1967 Julesz, 1975) -Dual task (Posner Bodies,
1971) -Priming interference (Kahneman
Treisman, 1984)
5
Posners Cost-benefit Paradigm
Benefit of selective attention always comes with
cost.
6
Temporal development of costs and benefits
7
Treismans parallel (pop-out) and serial
(conjunction) search paradigm
8
Signature RT functions for parallel and
serial search
Feature integration theory
9
Search Asymmetry O vs. Q
Illusory Conjunction
10

• Treismans notion of illusory conjunction is
  problematic
• 1) Too rare to occur. 2) Lack of attention
  really necessary?
• lt-- Our own demo of illusory conjunction of
  motion color in the periphery.
• 
  (Wu, Kanai
  Shimojo, Nature 04)

11
Object File
Same Object Effect
12
4. Dichotomy The following is a list of
conventional dichotomy. These are not mutually
inconsistent, nor orthogonal to each
other. -Early vs. late selection (the
paradox) -Parallel vs. serial processing -Automati
c vs. controlled -Top-down vs. bottom-up cf.
Endogenous vs. exogenous Isolation of
endogenous attention a) RTs b)
Dual-task c) Eye movements (anti-saccade)
d) Motion perception (line-motion effect)
e) Schizophrenia
13
5. Neuropsychology Psychophysical study of
cognitive deficits in brain-damaged patients has
been called neuropsychology. There are abundant
evidence for brain localizations and network
responsible for attentional functions, as
described below. -For attention and maintenance
of arousal level, and for visual
orienting(Early, Posner, Reiman Raichle,
1989), a) the right parietal lobe is
important(eg. our TMS study) b)
NE(nonepinephrine) system is critical, and c)
maybe relevant to the central cognitive deficits
in schizophrenia -Posner's latest, three
different network arousal, saliency and
endogenous.
14
Three attention orienting mechanisms in the
brain (Posner Raichle, 94)
15
Right Parietal Dominance for Attention
16
Right parietal damage is more devastating than
left.
17
6. Line-motion effect (Hikosaka, et al.,
1993a, b Shimojo, et al., 1997) Saliency and
attention turned out to not just affect, but also
yield an illusory motion perception, which was
best demonstrated in the line-motion effect.
This indicates that the top-down and the
bottom-up modulating signals interact at
relatively early levels of visual pathways. -
stimulus driven line motion - cross-modal line
motion - endogenous line motion - line motion
related to motor programming, and
discrimination.
18
Cross-modal Line Motion A
transient prime (A, V, or T) at L or R--gt Line
probe
Sound
Flash
Electric pulse
19
Cross-modal Line Motion A
transient prime (A, V, or T) at L or R--gt Line
probe
Immediately after a transient priming stimulus, a
probe line presented simultaneously appears to
unfold from the primed side. (Shimojo, Miyauchi
Hikosaka, 1997)
20

• 7. IOR(Inhibition Of Return) vs. FOR(Facilitation
  Of
• Return) (Tanaka Shimojo, 1996)
• IOR refers to a delay of reaction time at a
  target location identical or
• close to that in the previous trial. FOR refers
  to the opposite effect,
• i.e. shortening of reaction time at the same
  location.
• They demonstrated that identical stimulus
  parameters and sequence
• can lead to IOR or FOR, entirely dependent on
  the task.
• IOR occurs in spatial orienting tasks, such as
  detection, and location
• discrimination, whereas FOR occurs in feature
  discrimination tasks
• including shape, size, orientation, color, and
  vernier.

21
IOR (Inhibition Of Return) vs. FOR (Facilitation
Of Return)
Target in trial 1
trial 2 (L or R 50-50)
Target in trial 2 Same location (L) or
Different (R) At which location RT is
faster? Findings detection or orienting tasks
--gt IOR. feature discrimination
tasks --gt FOR.
22
Method
A TMS following-up (Hayashi, Wu Shimojo, unp.)

• Measureing RT to a visual target
• after external cue presentation.

Fixation Cue Re-Fixation Target Response
FP at center
23
Method

• Cue was flashed at either left or right side of
  the FP.

The red square is presented at left side of the
FP as cue.
Fixation Cue Re-Fixation Target Response
24
Method

• The FP jumped to the left (right) periphery.

Fixation Cue Re-Fixation Target Response
25
Method

• Observers were asked to judge the location of the
  target by pressing left/right button.
• Target was identified as being the same color as
  the cue.

Red and blue squares are presented bilaterally to
the new FP as target and distractor.
Fixation Cue Re-Fixation Target Response
26
Results TMS topography
Red retinotopic, Blue environmental

• Difference of RT changes (Env Ret) Z value
• TMSs applied on right PPC at 100ms after saccade
  onset has selective facilitation effect on RT of
  Ret relative to Env.
• Indicating that TMS suppressed coordinate
  transformation in the right PPC.

27
Results Spatial tuning (x-t Heat map)

• Spatial representation of attention shifts from
  Env to Ret frame because of TMS over the right
  PPC.

28
Localization of Po2

• Posterior part of IPS.
• Known as a homologous area of the monkey LIP.

29
Summary

• The locus of externally cued-attention is
  represented in retinotopic coordinate and updated
  with eye movement to fit with environmental
  coordinate.
• TMS delivered to the right PPC at 100ms after the
  onset of saccade directed to leftward
  (contralateral to the stimulated hemisphere)
  effectively disrupts this updating process.
• There is laterality in that the right PPC is more
  critical than the left for coordinate
  transformation.

30
8. Collision (stream/bounce) perception
(Sekuler, et al., 1997 Watanabe Shimojo, 1998)
Two visual objects moving across each other in
X-shaped trajectories appear to stream across
each other, even though theoretically it may be
seen as bouncing.
31
Stream/bounce effect
A sound affects ambiguity-solving in visual
motion.
Physical stimulus
Percept
Bounce off
Time
or
Brief sound
Passing though
Metzger (1934)
Sekuler et al. (1997)
No sound --gt Passing through
Sound --gt Bouncing
(depending on timing)
A collision detector?
32
Delay0, i.e. The sound is synchronized with the
visual crossing.
33

• 8. Collision (stream/bounce) perception
• (Sekuler, et al., 1997 Watanabe Shimojo,
  1998)
• Two visual objects moving across each other in
  X-shaped trajectories
• appear to stream across each other, even though
  theoretically it may
• be seen as bouncing.
• Sekuler et al. demonstrated that a sound given at
  the visual
• coincidence makes the bouncing percept
  dominant.
• Watanabe Shimojo showed later that a visual
  flash or
• somatosensory vibration can do the same, as far
  as it is synchronized
• with the visual coincidence.

What does crossmodal mean?
34
Bounce vs. Stream
Perception Q Does it have to be an auditory
transient? A No! A transient sensory signal,
regardless of modality, synchronized with the
visual coincidence increases the chance of bounce
percept. ltSynchronized stimuligt ltResultsgt
ltAuthorsgt Auditory
Bounce Sekuler, Sekuler Lau (97)
Visual Bounce
Watanabe Shimojo (98) Cutaneous
Bounce Sheliga Miles (99),
Watanabe
(01)
Any transient, sharp-onset stimulus seems to have
the same bounce- inducing effect, either within
or outside of the visual modality.
35
Visual Flash
Sound
Tactile Vibration
Bouncing percept increase ()
Flash-Coincidence Asynchrony (ms)
36
Multiple Sounds Paradigm Embedded Sound
Visual Stimulus
Auditory Stimulus
Single sound
Embedded sound
Baseline pitch/intensity
Time
Deviant pitch/intensity
Baseline pitch/intensity
Manipulating the saliency of the synch. sound by
intensity or pitch.
37
(Watanabe Shimojo, 2001)
Effect of Pitch change
N9

1
0
0
8
0
Percent Bounce ()
6
0
Sound Omission
4
0
No sound
2
0
0
Lower
Baseline
higher
Pitch
The sound synchronized with visual crossing
-Saliency of the synch. sound is critical,
yet -different from mere startling response.
38
(Watanabe Shimojo, 2001)
-saliency of the synch. sound is critical, and
-the intensity as well as the context affect.
39

• They also showed that attentional distraction in
  general, regardless
• of whether it is caused in exogenous or
  endogenous way, can increase
• the chance of bouncing. They interpret this
  result in that attentional
• tracking enhance facilitation along the motion
  trajectory, thus
• enhancing streaming.
• A disruption of it has the opposite effect, i.e.
  enhancing the bouncing
• percept.
• The same group showed that in the human infants,
  only 5-month-olds
• and older showed categorically same perception
  as adult, suggesting
• that their cross-modal integration with
  attention become mature only
• at this age.

40
The effect of auditory context and
saliency Attention distraction in general
? Attention on motion --gt streaming Attention
driven away from motion --gt bouncing (1)
Exogenous (transient auditory and visual
stimuli) (stimulus-driven, autonomous,
transient) (2) Endogenous (concurrent
task) (voluntary, task-dependent, sustained)
41
Endogenous Distraction of Attention
Central Task
Time
Streaming or
Left or Right ?
Bouncing ?
42
Effect of central task (timing)
Bounce enhanced only when concurrent task going
on.
100
80
With Task (lower VF)
With Task (upper VF)
60
Without Task (lower VF)
"Bouncing" Judgment ()
40
Without Task (upper VF)
20
0
-150
-100
-50
0
50
100
150
Before
After
Timing of Central Task Presentation
(ms)
relative to Target Superimposition
43
Developmental issues

• Crossmodal integration Two theories
• Piaget integration is learned through
  interaction
• Gibson integration exists from birth
• Is bouncing illusion best accounted for in
  Piagetian or Gibsonian terms?
• Are there developmental changes in how sound
  influences visual motion perception?

44
Developmental aspects of
audio-visual integration
Method infant-controlled habituation
Trial 1
Trial 2
Trial N
Trial N1
Trial N2
A
A
A
A
A
.....
Habituation Phase
Test Phase
45
Example results
8 months (MW)
5 months (MW)
16
70
YES!
14
60
50
12
Looking Time (sec.)
10
40
30
8
NO.
20
6
10
4
2
0
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
10
11
Test Phase
Habituation Phase
Habituation Phase
Test Phase
(1) Pre. visual event (2) At (Familiar) (3) Post.
Dishabituation?
Habituation?
Dishabituation?
46
Overall results
20
20

18
18
NO. NO. YES! YES!
16
16
sound pre.
14
14
12
12
familiar (sound _at_)
Looking Time sec
10
10
8
8
sound post.
6
6
4
4
2
2
0
0
4
5
6
8
Age months
47
Eye movement recording another approach to the
bouncing illusion

• method corneal-reflection technique
• (tracking of IR light in the infants eye)
• main ideas
• - streaming perception continuous eye movements
• - bouncing perception tracking, and reverse
• results
• - main idea confirmed in some babies, but
• - oscillations between two objects in many babies
• next steps better eye recording techniques

48
(No Transcript)
49
Beating Heart Effect (Sheth Shimojo, 02)
1. Deadaptation Sound-induced recovery from
adaptation. 2. Resistanced to fade away
50
If luminance contrast is the same, color-based
saliency is critical.
Sound 100-0 ms before flash works best! --gt
Sound as attention cue.
SOA(Flash-Sound)
51

• 9. Some new phenomena related to representation
• Change Blindness (Bridgman, Rensink, ORegan, et
  al.))
• Inattentional Blindness (Rock Mack, et al.)
• Negative Priming (Yantis)
• d) Attentional Blink (Raymond, Shapiro, et al.)
  
• e) Repetition Blindness (Kanwisher, Potter, et
  al.)

52
(No Transcript)
53
(No Transcript)
54
(No Transcript)
55
Inattentional Blindness (Rock et al., 92)
A very salient stimulus near the fovea/attention
focus may not be
noticed, if task irrelevant.
56
Texture segregation requires attention.
57
No explicit memory of the ignored (recognition at
chance)
But a negative priming effect!
58
Negative priming even with unfamiliar shapes with
a long delay!
59
Attentional Blink
D
D
D
T1
D
T2
D
D
RSVP
Detection of T2
Time (frame)
60
Repetition Blindness
D
D
D
T
D
T
D
D
RSVP
Detection of 2nd T
Time (frame)
61

References Broadbent, D. E. (1958) Perception
and Communication. Pergamon Press. Early, T.,
Posner, M. I., Reiman, E. Raichle, M. E. (1989)
Left striato- pallidal hyperactivity in
schizophrenia. Psychiat. Develop., 7,
109-121. Hikosaka, O., Miyauchi, S. and Shimojo,
S. (1993) Voluntary and stimulus- induced
attention detected as motion sensation.
Perception, 22, 517-526. Hikosaka, O., Miyauchi,
S. and Shimojo, S. (1993) Focal visual
attention produces illusory temporal order
andmotion sensation. Vision Research, 33,
1219-1240. Kahneman, D. Treisman, A. M. (1984)
Changing views of attention and automaticity.
In R. Parasuraman D. R. Davies (eds.),
Varieties of Attention, 29-61, Acad.
Press. Posner, M. I. Cohen, Y. (1984)
Components of visual orienting. In H, Bouma D.
G. Bouwhis (eds.), Attention Performance X,
531-556, Erlbaum. Sekuler, R., Sekuler, A. B.,
and Lau, R. (1997) Sound alters visual motion
perception. Nature(London), 385, 308.
Tanaka, Y. and Shimojo, S. (1996) Location vs.
feature reaction time reveals dissociation
between two visual functions. Vision Research,
36, 2125-2140.