Tamara Nicol Medina

1
Using Extra-Linguistic Cues to Identify Good Word
Learning Instances

• Tamara Nicol Medina
• John Trueswell
• Lila Gleitman
• University Of Pennsylvania
• Jesse Snedeker
• Harvard University
• Society for Research in Child Development
• April 2, 2009, Denver, CO

2
(No Transcript)
3
Just look at the world!

• Observe physical and temporal contingencies
  between words and objects.
• (At least, for physically observable objects.)
• Experimental evidence supports ease of mapping
• Fast mapping (e.g., Carey, 1978 Mervis
  Bertrand, 1994 Behrend et al., 2001 Jaswal
  Markman, 2001)
• Cross-situational word learning (e.g., Yu
  Smith, 2007 Smith Yu, 2008 Vouloumanos, 2008
  Xu Tenenbaum, 2007)

4
(No Transcript)
5
Its Not that Easy! (Augustine, Locke, Quine,
Gleitman, Fodor, Siskind, etc.)

• Reference problem
• Book? Cat? Shoes? Chair? Cheerios? Cup?
  Rug? Pants? Head? Hand?
• Frame problem
• Dog or Puppy? Hand or Finger? Red or Ball?
• Naturalistic learning conditions
• Medina, Trueswell, Snedeker, Gleitman. (2008).
  When the shoe fits Cross-situational word
  learning in realistic learning environments.
  BUCLD.

6
How do learners narrow down the possibilities?

• Linguistic context (Landau Gleitman, 1985
  Gleitman, 1990 Gillette, Gleitman, Gleitman,
  Lederer, 1999)
• Learning biases
• Whole object constraint (Markman, 1989)
• Mutual exclusivity (Markman Wachtel, 1988
  Markman, Wasow, Hansen, 2003)
• Social-attentional cues (e.g., Baldwin 1991,
  1993 Tomasello Akhtar, 1995 Bloom, 2002
  Behne, Carpenter, Tomasello, 2005)

7
Social-Attentional Cues

• Nonverbal cues can reduce the range of possible
  interpretations.
• Direction of speaker eye-gaze (Baldwin, 1991,
  1993 Trueswell Gleitman, 2003 Nappa, Wessel,
  McEldoon, Gleitman, Trueswell, 2009)
• Joint attention Occurs naturally when parent
  and child are focused on the same thing at the
  same time (Baldwin, 1991 Bruner, 1978)
• 70 of mothers utterances (Collis, 1977
  Harris, Jones, Grant, 1983 Tomasello Todd,
  1983)
• Positively associated with early vocabulary
  acquisition (Tomasello Todd, 1983 Harris,
  Jones, Brookes, Grant, 1986 Tomasello, Mannle,
  Kruger, 1986 Akhtar, Dunham, Dunham, 1991)

8
Quality of Learning Instances(Baldwin, 1991)

• But what about the lack of perfect contingency
  between word and referent?

9
Follow-In vs Discrepant Labeling
Look! A dax!
10
Quality of Learning Instances(Baldwin, 1991)

• But what about the lack of perfect contingency
  between word and referent?
• Follow-in labeling eye gaze, voice direction,
  and body posture oriented at object child is
  currently focused on
• 16-19 mo-olds mapped correctly
• Discrepant labeling eye gaze, voice direction,
  and body posture directed at a hidden (but
  previously seen) object, while infant is focused
  on another object
• Infants did not map the word to the
  focused-object.

11
Social-attentional cues in interaction(Frank,
Goodman, Tenenbaum, In Press)

• Rollins corpus (CHILDES) mom and baby (6 mo)
• Social-attentional cues
• Infant Hands, Mouth (infant only), Touch,
  Looking (direction of eye gaze)
• Caregiver Hands, Touch, Looking (direction of
  eye gaze)
• Cross-situational word-learning model
  successfully discovered the mappings between
  words and objects.

12
Social-attentional cues in interaction(Frank,
Goodman, Tenenbaum, In Press)

• Rollins corpus (CHILDES) mom and baby (6 mo)
• Social-attentional cues
• Infant Hands, Mouth (infant only), Touch,
  Looking (direction of eye gaze)
• Caregiver Hands, Touch, Looking (direction of
  eye gaze)
• Cross-situational word-learning model
  successfully discovered the mappings between
  words and objects.

13
Social-attentional cues in interaction(Frank,
Goodman, Tenenbaum, In Press)

• Rollins corpus (CHILDES) mom and baby (6 mo)
• Social-attentional cues
• Infant Hands, Mouth (infant only), Touch,
  Looking (direction of eye gaze)
• Caregiver Hands, Touch, Looking (direction of
  eye gaze)
• Cross-situational word-learning model
  successfully discovered the mappings between
  words and objects.
• Joint attention? Follow-in?
• What would interaction look like if child were
  initiating actions?

14
Our Goals

• Look at a representative sample of parent-child
  interactions.
• Explore the conditions under which word meaning
  is transparent (or not) from extra-linguistic
  cues alone
• Presence (or absence) of cues
• Timing and coordination of cues
• Joint attention?
• Follow-in?

15
Selection of Stimuli

• Large video corpus of parent-child interactions
  in natural settings (home, outdoors, etc.)
• Snedeker, J. (2001). Interactions between
  infants (12-15 months) and their parents in four
  settings. Unpublished corpus.

16
Selection of Stimuli

• Word learning norming study
• Gertner, Y., Fisher, C., Gleitman, L., Joshi, A.,
  Snedeker, J. (In progress). Machine
  implementation of a verb learning algorithm.
• Adapation of Human Simulation Paradigm (Gillette,
  Gleitman, Gleitman, Lederer, 1999 Snedeker and
  Gleitman, 1999)
• Randomly selected six instances of highly
  frequent content words.
• Each instance was edited into a 40-second
  vignette.
• Sound turned off.
• Visual context only cue to word meaning, placing
  viewers in the situation of the early word
  learner.
• Utterance of target word (at 30 sec) indicated by
  a BEEP.
• Guess the mystery word in each vignette.

17
(No Transcript)
18
(silence)
(silence)
ltBEEPgt
30 sec
(silence)
10 sec
Drawings courtesy of Emily Trueswell
19
Selection of Stimuli

• Two types of vignettes
• High Informative vignettes guessed by gt50 of
  participants
• Low Informative vignettes guessed by lt33 of
  participants

20
Selection of Stimuli

• Two types of vignettes
• High Informative vignettes guessed by gt50 of
  participants
• Rare (only 7 of vignettes).
• All basic level objects.
• Low Informative vignettes guessed by lt33 of
  participants
• Stimuli for Current Study
• 8 nouns bag, ball, book, horse, necklace, nose,
  phone, shoe
• One HI vignette
• One LI vignette

21
Pilot Study Children

• N 12 (ages 31 to 54)
• Modified for fun!
• Shorter vignettes with funny noises
• What do you think the parent said?
• Celebratory animation

22
Pilot Study Children
?2 3.84

23
Extra Linguistic Cue Coding

• Is the target object visible in the scene (on
  screen)?
• Is the child moving or reaching towards the
  object?
• Is the child handling the object?
• Is the child looking at the object?
• Is the child looking at the parent?
• Is the parent moving or reaching towards the
  object?
• Is the parent handling the object?
• Is the parent looking at the object?
• Is the parent looking at the child?

24
Presence of Target Object
Error bars reflect Standard Error of the Mean.
25
Cue Occurrence at Word Onset
?2 7.27
?2 4.00
?2 4.27



26
Joint Attention at Word Onset
Child Looking at and/or Handling Object AND
Parent Looking at Object
?2 4.00

27
What is the timing of cues?

• Follow-in?
• Parent refers to object under childs focus of
  attention.
• First onset of cues relative to word onset.

28
First Onset of Cues
Error bars reflect Standard Error of the Mean.
Child Looking at Object t(1,12)1.56, p0.14
Child Moving/Reaching Toward Object t(1,12)2.05,
p0.06
Child Handling Object t(1,12)2.96, p0.01
Parent Handling Object t(1,8)1.09, p0.31
29
First Onset of Cues
Error bars reflect Standard Error of the Mean.
Parent Looking at Child t(1,13)0.54, p0.59
Parent Looking at Object t(1,12)0.08, p0.93
Child Looking at Parent t(1,6)0.02, p0.98
Parent Moving/Reaching Toward Object t(1,7)0.15,
p0.89
30
Differentiating HI and LI Vignettes

• High Informative
• Follow-In Utterance of target word immediately
  after first onset of childs shift in focus
  towards object.
• Joint Attention Co-occurring high rates of
  childs attention to object and parents
  attention to child and object.
• Low Informative
• Delayed follow-in.
• Low joint attention.

31
Implications

• Basic Level Object Terms provide a scaffold for
  further learning.
• Word order, syntax, abstract lexical items, etc.
• Vindication of Bruner/Baldwins social conditions
  for word learning found in natural parent-child
  interactions.
• Word learning is successful when cues align.