Dynamical Insights into Structure in Connectionist Models

1
Dynamical Insights into Structure in
Connectionist Models
Whitney TaborDepartment of PsychologyUniversity
of ConnecticutJune 22, 2005
2
The Learning of Natural Language
Question How can a system without structure
acquire it?
3
The Learning of Natural Language
Question How can a system without structure
acquire it? Observation Certain very general
(and recurrent) connectionist networks, trained
on finite grammaticality data, behave as though
they have discovered the infinite state mechanism
from which the data were sampled.
4

Observation Connectionist models are a type of
dynamical system.
5
Insights
1. Dynamical analysis helps figure how to get
the network to succeed and to identify
success. 2. Dynamical analysis reveals stages
of development which correspond to phase
transitions. (Caveat I dont know if this
models stages correspond to human developmental
stages).
6
Timescales
Development babababa Except for the

Marabar Caves
t1
t2 Syntactic Joan! The cat is in the garage
t1 t2 t3 t4 Real
time Articulation, eye-movement
7
Elmans Paradigm for Connectionist Sentence
Structure Learning Simple Recurrent Network
8
Elmans Paradigm
Girl chases dogs. Boys chase dog. Girl sees
tiger. Dogs who girl chases see tiger. Tiger
eats boy
9
Elmans Paradigm boys
10
Elmans Paradigm who
11
Elmans Paradigm mary
12
Elmans Paradigm chases
13
Elmans Paradigm feed
14
Elmans Paradigm cats
15
Elmans Paradigm Puzzle
16
Elmans Paradigm Puzzle
Implication Even in the trained network, any
word can occur at any time.
17
Questions
What has the network learned? Has the network
discovered the structure of its language
environment?
18
Symbolic Characterization of the Environment
19
Probing what the network has discovered (Wiles
Elman, 1995)
Simple Case (called anbn) S ? a b S ? a S
b Possible Sentences a b, a a b b, a a a b b
b,
20
Hidden Unit Trajectory (aaa)
21
Hidden Unit Trajectory (bbb)
22
Vector flow for iterated as
23
Vector flow for iterated bs
24
Data check
The dog who the boy fed barked. The dog who the
boy fed barked. N1 N2
V2 V1 (Palindromic structure ---more
complex than anbn)
25
Rodriguez (2001) SRN on Palindrome Language
Learning success miserable Hidden unit space
noisy and complex Insightful analysis by
Rodriguez hand-built linear approximation of
weight matrix handles full palindrome language
26
The Beauty and the Horror
27
Barnsley (1988), Moore (1996), Tabor (2000)
Fractal sets for keeping track of embedded
structures in a bounded metric space.
28
Stack Map
29
Sentence Processing on Fractal
30
Fractal Analysis of Rodriguezs Linearization
31
Fractal Learning Neural Network
32
Learning Results (FLNN)
33
Emergence of an Infinite-State Grammar 0
34
Emergence of an Infinite-State Grammar 1
35
Emergence of an Infinite-State Grammar 2
36
Emergence of an Infinite-State Grammar 3
37
Emergence of an Infinite-State Grammar 4
38
Conclusions

• 1. Dynamical analysis helps figure how to get the
  network to succeed and to identify success.
• 2. Dynamical analysis reveals stages of
  development which correspond to phase
  transitions. There is a coherent sense in which
  this network has acquired a concept.

39
Conclusions
3. Timescales a. Attractor basin topology is
what emerges over developmental time. b.
Settling time within an attractor basin (measured
in number of steps) predicts number of words
needed to complete a sentence. (syntactic time)
c. Note In real-time connectionist sentence
processing models, settling time predicts
individual word reaction times.
40
Connectionist vs. Dynamical Field Theory Methods
Connectionism
DFT
Train a network on a challenging problem. Study
the model. Discover valuable new conceptual tools
for a domain.
Think carefully about the dynamics of a
domain. Build a dynamical system with plausible
topology.
BOTH Test predictions.
41
Connectionist vs. Dynamical Field Theory Methods
Connectionism
DFT
Train a network on a challenging problem. Study
the model. Discover valuable new conceptual tools
for a domain.
Think carefully about the dynamics of a
domain. Build a dynamical system with plausible
topology.
BOTH Test predictions.