8: Neural Networks II

1
8 Neural Networks II

• Outline
• Past tense acquisition
• Rumelhart McClellands model
• Criticisms
• Plunkett Marchmans model
• Fodor Pylyshyns critique of connectionism
• Responses
• Connectionism and folk psychology
• Can connectionism account for human cognition?

2
How do we learn language?

• Chomskyan view
• Innate knowledge of possible rules of language
• Children create hypotheses about how these rules
  apply to the language they are learning
• We have mental representations of these rules
• Alternate view
• No explicit representation of rules, although
  performance can be described in terms of rules

3
Past-tense acquisition (Brown, 1973)

• 1) specific forms learnt both regular and
  irregular
• 2) overgeneralisation of irregular verbs
• e.g. wented, goed, eated
• 3) correct pronunciation of both regular and
  irregular verbs

Correct production
time
4
Rumelhart McClellands (1986) Model
5
Rumelhart McClellands (1986) Model

• Architecture
• Single layer pattern associator
• Inputs present tense (460 units)
• Outputs past tense (460 units)
• Words represented as sets of Wickelfeatures
• Extra networks at back front of pattern
  associator to encode/decode Wickelfeatures from
  phonological representation

6
Rumelhart McClellands (1986) Model

• Word representation
• Phonological form /kAm/ came
• But, indistinguishable from /mAk/ or /Akm/
• Wickelphones
• Context sensitive kA, kAm, Am
• Can be analysed along 4 dimensions
• e.g. /A/ long, low, vowel, front
• Wickelfeatures

7
Training Results

• Training sets
• 10 high frequency words (8 irregular)
• 410 medium frequency words (76 irregular)
• 86 low frequency words (14 irregular)
• Trained on high frequency only then medium
  frequency added low frequency used later
• Results
• U-shaped curve
• Overgeneralisation
• Come ? comed, camed
• Eat ? eated

8
Implications

• Links between regular verb stems and past tense
  forms can be described using rules, but is
  governed by a mechanism which does not use
  explicit rules
• Knowledge of past-formation is distributed across
  the network
• Links between irregular verb stems and past tense
  forms are encoded in same set of weights
• In a rule-based account, there would need to be a
  rule for producing regular verbs and a list of
  exceptions (irregulars)

9
Pinker Prince (1988)

• The U-shaped curve is a result of the way in
  which the input was presented, not anything to do
  with the properties of the network
• The middle of the curve coincides with the
  addition of the medium frequency verbs
• Network is flooded by regular verbs forces
  network to generalise
• In real language input, there is no such
  discontinuity

10
Pinker Prince (1988)

• RM model does a poor job of generalizing to some
  novel verbs
• mail ? membled
• tour ? toureder
• Model doesnt conceive of stemsuffix
• Cannot encode the formula for creating a
  past-tense ending
• Task decomposition
• Past tense treated as autonomous
• Wickelphones Wickelfeatures

11
Plunkett Marchmans (1991) Response

• Model of past-tense acquisition using
  back-propagation network
• 3 layer network, 20 units per layer
• No discontinuity in input
• Didnt use wickelfeatures
• Parametric studies
• 74 of tokens irregular regular not learned
• 74 of tokens regular irregular not learned
• 50/50 (about the same as parental input) network
  performed well
• No global U-shaped curve
• Micro U-shaped curves corresponds better to
  child data as global U-shaped curve is a myth.

12
Fodor Pylyshyns (1988) critique of
connectionism

• Connectionism is inadequate as a representational
  system
• Symbolic system necessary for modelling cognitive
  processes
• Connectionist systems lack a combinatorial syntax
  and semantics
• Productivity
• Systematicity
• Coherence
• Connectionism just provides an implementation of
  symbolic processing

13
Response 1 Approximationist approach

• Symbolic models are approximately correct
• cf. Newtonian physics
• Subsymbolic models required for intuitive
  processing
• Language does not take place at a conscious level
• Cognitive behaviour not rule-governed, can merely
  be rule-described

14
Response 2 Externalist approach

• Symbols are human artifacts
• Symbol manipulation is learned
• Initially carried out on symbols in the external
  environment
• We are confronted with external symbols (e.g.
  written language and mathematical notation)
• We learn to manipulate them.
• Internalisation of these symbols occurs

15
Connectionism Folk Psychology

• Connectionism provides a conceptual foundation
  that might replace folk psychology.
• Neural nets show that simple cognitive tasks can
  be performed without employing features that
  could correspond to beliefs, desires and plans.
• Provides support for eliminativism

16
Can connectionism account for human cognition?

• Learning driven by examples
• Knowledge of rules is emergent
• Multitude of of sub-symbolic representations
• Complex interaction produces behaviour which is
  rule-like
• Knowledge of rules remains implicit
• Cannot analyse own activity
• Cannot form symbolic representations of rules

17
Clark Karmiloff-Smith (1993)

• To model human development adequately,
  connectionist systems must be able to
• Treat own representations as objects for further
  manipulation
• Do so independently of continual training input
• Retain copies of original networks
• Form new structured representations

18
Learning Outcomes Reading

• Understand connectionist models of past-tense
  acquisition
• Understand how rule-like behaviour can emerge
  from non-rule-based systems
• Be aware of some of the criticisms of the
  connectionist approach and the responses to those
  criticisms
• For next week, read
• Embodied Cognitive Science Basic Concepts from
  Pfeifer, R. Scheier, C. (1999). Understanding
  intelligence. Cambridge, MA MIT Press.
• Further Reading
• See lecture webpage or .pdf handout