9.012 Brain and Cognitive Sciences II

1
9.012Brain andCognitive Sciences II
Part VIII Intro to Language Psycholinguistics
- Dr. Ted Gibson
2
Presented by Liu Lab
Fighting for Freedom with Cultured Neurons
3
Distributed Representations, Simple Recurrent
Networks, And Grammatical Structure Jeffrey L.
Elman (1991) Machine Learning

• Nathan Wilson

4
Distributed Representations/ Neural Networks

• are meant to capture the essence of neural
  computation
• many small, independent units calculating very
  simple functions in parallel.

5
Distributed Representations/ Neural Networks
EXPLICIT RULES?
6
Distributed Representations/ Neural Networks
EXPLICIT RULES?
7
Distributed Representations/ Neural Networks
EXPLICIT RULES?
EMERGENCE!
8
Distributed Representations/ Neural Networks

• are meant to capture the essence of neural
  computation
• many small, independent units calculating very
  simple functions in parallel.

9
FeedForward Neural Network (from Sebastians
Teaching)
10
Dont forget the nonlinearity!
11
FeedForward Neural Network (from Sebastians
Teaching)
12
Recurrent Network (also from Sebastian)
13
Why Apply Network / Connectionist Modeling to
Language Processing?

• Connectionist Modeling is Good at What it Does
• Language is a HARD problem

14
What We Are Going to Do
15
What We Are Going to Do

• Build a network

16
What We Are Going to Do

• Build a network
• Let it learn how to read

17
What We Are Going to Do

• Build a network
• Let it learn how to read
• Then test it!

18
What We Are Going to Do

• Build a network
• Let it learn how to read
• Then test it!
• Give it some words in a reasonably grammatical
  sentence
• Let it try to predict the next word,
• Based on what it knows about grammar

19
What We Are Going to Do

• Build a network
• Let it learn how to read
• Then test it!
• Give it some words in a reasonably grammatical
  sentence
• Let it try to predict the next word,
• Based on what it knows about grammar
• BUT Were not going to tell it any of the rules

20
What We Are Going to Do

• Build a network

21
FeedForward Neural Network (from Sebastians
Teaching)
22
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
23
What We Are Going to Do

• Build a network
• Let it learn how to read

24
Methods gt Network Implementation gt Training
Words Were going to Teach it

• - Nouns
• boy girl cat dog
• boys girls cats dogs
• - Proper Nouns
• John Mary
• Who
• - Verbs
• chase feed see hear walk live
• chases feeds sees hears walks lives
• End Sentence

25
Methods gt Network Implementation gt Training
1. Encode Each Word with Unique Activation Pattern
26
Methods gt Network Implementation gt Training
1. Encode Each Word with Unique Activation Pattern

• - boy gt 000000000000000000000001
• girl gt 000000000000000000000010
• feed gt 000000000000000000000100
• -sees gt 000000000000000000001000
• . . .
• who gt 010000000000000000000000
• End sentence gt
• 100000000000000000000000

27
Methods gt Network Implementation gt Training
1. Encode Each Word with Unique Activation Pattern

• - boy gt 000000000000000000000001
• girl gt 000000000000000000000010
• feed gt 000000000000000000000100
• -sees gt 000000000000000000001000
• . . .
• who gt 010000000000000000000000
• End sentence gt
• 100000000000000000000000

2. Feed these words sequentially to the
network (only feed words in sequences that make
good grammatical sense!)
28
Methods gt Network Implementation gt Structure
INPUT
29
Methods gt Network Implementation gt Structure
1000000000000
INPUT
30
Methods gt Network Implementation gt Structure
HIDDEN
1000000000000
INPUT
31
Methods gt Network Implementation gt Structure
100100100100100100100100
HIDDEN
1000000000000
INPUT
32
Methods gt Network Implementation gt Structure
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
33
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
34
Methods gt Network Implementation gt Training
1. Encode Each Word with Unique Activation Pattern

• - boy gt 000000000000000000000001
• girl gt 000000000000000000000010
• feed gt 000000000000000000000100
• -sees gt 000000000000000000001000
• . . .
• who gt 010000000000000000000000
• End sentence gt
• 100000000000000000000000

2. Feed these words sequentially to the
network (only feed words in sequences that make
good grammatical sense!)
35
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
36
What We Are Going to Do

• Build a network
• Let it learn how to read

37
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
38
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
If learning word relations, need some sort of
memory from word to word!
1000000000000
INPUT
39
FeedForward Neural Network (from Sebastians
Teaching)
40
Recurrent Network (also from Sebastian)
41
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
INPUT
42
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
43
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
44
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
45
Methods gt Network Implementation gt Structure
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
46
Methods gt Network Implementation gt Structure
BACKPROP!
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
47
What We Are Going to Do

• Build a network
• Let it learn how to read
• Then test it!
• Give it some words in a reasonably grammatical
  sentence
• Let it try to predict the next word,
• Based on what it knows about grammar
• BUT Were not going to tell it any of the rules

48
Results gt Emergent Properties of Network gt
Subject-Verb Agreement

• After Hearing
• boy.
• Network SHOULD predict next word is
• chases
• NOT
• chase
• Subject and verb should agree!

49
Results gt Emergent Properties of Network gt
Noun-Verb Agreement

• After Hearing
• boy.
• Network SHOULD predict next word is
• chases
• NOT
• chase
• Subject and verb should agree!

50
Results gt Emergent Properties of Network gt
Noun-Verb Agreement
boy..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
51
Results gt Emergent Properties of Network gt
Noun-Verb Agreement
boy..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
52
Results gt Emergent Properties of Network gt
Noun-Verb Agreement

• Likewise, after Hearing
• boys. (or boyz!)
• Network SHOULD predict next word is
• chase
• NOT
• chases
• Again, subject and verb should agree!

53
Results gt Emergent Properties of Network gt
Noun-Verb Agreement
boys..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
54
Results gt Emergent Properties of Network gt
Noun-Verb Agreement
boys..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
55
Results gt Emergent Properties of Network gt
Noun-Verb Agreement
boys..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Theres a difference between nouns and verbs.
There are even different kinds of nouns that
require different kinds of verbs.
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
56
Results gt Emergent Properties of Network gt
Verb-Argument Agreement

• After Hearing
• chase
• Network SHOULD predict next word is
• some direct object (like boys)
• NOT
• .
• Hey, if a verb needs an argument,
• it only makes sense to give it one!

57
Results gt Emergent Properties of Network gt
Verb-Argument Agreement

• Likewise, after hearing the verb
• lives
• Network SHOULD predict next word is
• .
• NOT
• dog
• If the verb doesnt make sense with an argument,
• It falls upon us to withhold one from it.

58
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boy chases..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
59
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boy chases..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
60
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boy lives..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
61
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boy lives..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
62
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boy lives..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
There are different kinds of verbs that require
different kinds of nouns.
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
63
Results gt Emergent Properties of Network gt
Longer-Range Dependence

• After hearing
• boy who mary chases
• Network might predict next word is
• boys
• Since it learned that boys follows mary
  chases
• But if its smart
• may realize that chases is linked to
  boys, not mary
• In which case you need a verb next, not a noun!
• A good lithmus test for some intermediate
  understanding?

64
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boys who Mary..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
65
Results gt Emergent Properties of Network gt
Verb-Argument Agreement
boys who Mary..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
66
Results gt Emergent Properties of Network gt
Subject-Verb Agreement
boys who mary chases..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
67
Results gt Emergent Properties of Network gt
Subject-Verb Agreement
boys who mary chases feed..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
68
Results gt Emergent Properties of Network gt
Subject-Verb Agreement
boys who mary chases feed cats..
End of Sentence
Who
Plural Verb, DO Impossible
Plural Verb, DO Required
Plural Verb, DO Optional
What Word Network Predicts is Next
Single Verb, DO Impossible
Single Verb, DO Required
Single Verb, DO Optional
Plural Noun
Single Noun
0.0 0.2 0.4 0.6 0.8 1.0
Activation
69
What We Are Going to Do

• Build a network
• Let it learn how to read
• Then test it!
• Give it some words in a reasonably grammatical
  sentence
• Let it try to predict the next word,
• Based on what it knows about grammar
• BUT Were not going to tell it any of the rules

70
Did Network Learn About Grammar?

• It learned there are different classes of nouns
  that need singular and plural verbs.
• It learned there are different classes of verbs
  that have diff. requirements in terms of direct
  objects.
• It learned that sometimes there are long-distance
  dependencies that dont follow from immediately
  preceding words
• gt relative clauses and constituent structure of
  sentences.

71
(No Transcript)
72
Once You Have a Successful Network, can Examine
its Properties with Controlled I/O Relationships

• Boys hear boys
• Boy hears boys.
• Boy who boys chase chases boys.
• Boys who boys chase chase boys.

73
Methods gt Network Implementation gt Structure
BACKPROP!
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
74
Distributed Representations/ Neural Networks
EXPLICIT RULES?
75
Distributed Representations/ Neural Networks
EXPLICIT RULES?
76
Methods gt Network Implementation gt Structure
BACKPROP!
0000000000001
OUTPUT
100100100100100100100100
HIDDEN
1000000000000
100100100100100100100100
INPUT
CONTEXT
77
What Does it Mean, No Explicit Rules?

• Does it just mean the mapping is too
  complicated?
• Too difficult to formulate?
• Unknown?
• Possibly just our own failure to understand the
  mechanism, rather than description of mechanism
  itself.

78
General Advantages of Distributed Models

• Distributed, which while not limitless, is less
  rigid than models where there is strict mapping
  from concept to node.
• Generalizations are captured at a higher level
  than input abstractly. So generalization to
  new input is possible.

79
FOUND / ISOLATED 4-CELL NEURAL NETWORKS
80
(No Transcript)
81
9.012Brain andCognitive Sciences II
Part VIII Intro to Language Psycholinguistics
- Dr. Ted Gibson
82
Presented by Liu Lab
Fighting for Freedom with Cultured Neurons
83
If you have built castles in the air, your work
need not be lost that is where they should be.
Now put the foundations under them.
-- Henry David Thoreau