Semantic Cognition: A Parallel Distributed Processing Approach

1
Semantic CognitionA Parallel Distributed
ProcessingApproach

• James L. McClelland
• Center for the Neural Basis of CognitionandDepar
  tments of Psychology and Computer Science,
  Carnegie Mellon Timothy T. Rogers
• Center for the Neural Basis of Cognition and
  nowMRC Cognition and Brain Sciences Unit, UK

CNBC A Joint Project of Carnegie Mellon and the
University of Pittsburgh
2
Approaches to Semantic Cognition

• Concepts and their Properties
• Is Socrates Mortal?
• Hierarchical Propositional Models
• Quillian, 1968 Collins and Quillian, 1969
• Theory-Theory and Related Approaches
• Murphy and Medin, 1985 Gopnik and Wellman, 1994
  Keil, 1991 Carey, 1985
• Parallel Distributed Processing
• Hinton, 1981 Rumelhart and Todd, 1993 McRae, De
  Sa, and Seidenberg, 1997

3
Plan for This Talk

• Compare a distributed, connectionist model that
  learns from exposure to information about the
  relations between concepts and their properties
  to the classical Hierarchical Propositional
  Approach.
• Show how the model accounts for a set of
  phenomena that have been introduced in support of
  Theory Theory
• Conclude with a brief consideration of where we
  are in the development of a theory of semantic
  cognition.

4
Initial Motivations for the Model

• Provide a connectionist alternative to
  traditional hierarchical propositional models of
  conceptual knowledge representation.
• Account for development of conceptual knowledge
  as a gradual process involving progressive
  differentiation.

5
QuilliansHierarchicalPropositional Model
6
The Parallel Distributed Processing Approach

• Processing occurs via propagation of activation
  among simple processing units.
• Knowledge is stored in the weights on connections
  between the simple processing units.
• Propositions are not stored directly.
• The ability to produce complete propositions from
  partial probes arises through the activation
  process, based on the knowledge stored in the
  weights.
• Learning occurs via adjustment of the
  connections.
• Semantic knowledge is gradually acquired through
  repeated exposure, mirroring the gradual nature
  of cognitive development.

7
Activation
The Rumelhart Model
8
The Training Data
All propositions true of items at the bottom
levelof the tree, e.g. Robin can grow, move,
fly
9

Error
10
Any Questions?
11
Differentiation in Development
12
(No Transcript)
13
The Rumelhart Model
14
(No Transcript)
15
(No Transcript)
16
Trajectories of Representations Through State
Space over Time
17
Any Questions?
18
Tenets of Theory Theory

• Intuitive domain knowledge of relations between
  items and their properties is use to decide
• which categories are good ones and which
  properties are central to particular concepts
• how properties should be generalized from one
  category to another
• Many proponents suggest that some theory-like
  knowledge (or constraints on acquiring such
  knowledge) must be available initially.
• Others emphasize reorganization of knowledge
  through experience, but provide very little
  discussion of how experience leads to
  reorganization.

19
Three Phenomena Supporting Theory Theory

• Category goodness and feature importance.
• Differential importance of properties in
  different concepts.
• Reorganization of conceptual knowledge.

20
Effects of Coherent Variation of Properties on
Learning

• Attributes that vary together create the concepts
  that populate the taxonomic hierarchy, and
  determine which properties are central to a given
  concept.
• Where sets of attributes vary together, they
  exert a strong effect on learning.
• Items with co-varying properties stay together
  through semantic space and form the clusters
  corresponding to super-ordinate concepts.
• Arbitrary properties (those that do not co-vary
  with others) are very difficult to learn, even
  when frequency is controlled.
• They control a late stage of differentiation in
  which individual items within clusters become
  conceptually distinct.

21
CoherenceTrainingEnvironment
Properties Coherent Incoherent
12345678910111213141516
Items
No Category Labels are Provided!
22
Effect of Coherence on Learning
23
Effect of Coherence on Representation
24
Extended modelfor remaining simulations
25
Progressive Differentiation of Category Structure
Without Names
300 Epochs
1200 Epochs
plants animals
plants animals
26
Any Questions?
27
Three Phenomena Supporting Theory Theory

• Category goodness and feature importance.
• Differential importance of properties in
  different concepts.
• Reorganization of conceptual knowledge.

28
Differential Importance (Marcario, 1991)

• 3-4 yr old children see a puppet and are told he
  likes to eat, or play with, a certain object
  (e.g., top object at right)
• Children then must choose another one that will
  be the same kind of thing to eat or that will
  be the same kind of thing to play with.
• In the first case they tend to choose the object
  with the same color.
• In the second case they will tend to choose the
  object with the same shape.

29
Adjustments to Training Environment

• To address this we added some new property units
  and created clear cases of feature-dependencies
  in the model
• Among the plants
• All trees are large
• All flowers are small
• Either can be bright or dull
• Among the animals
• All birds are bright
• All fish are dull
• Either can be small or large
• Though partially counter-factual, these
  assignments allow us to explore domain
  specificity of feature dependencies in the model.

30
Testing Feature Importance

• After partial learning, model is shown eight test
  objects
• Four Animals
• All have skin
• All combinations of bright/dull and large/small
• Four Plants
• All have roots
• All combinations of bright/dull and large/small
• Representations are generated by
  usingback-propagation, training the
  item-to-representation weights only.
• Representations are then compared to see which
  animals are treated as most similar, and also
  which plants are treated as most similar.

31
(One unit is addedfor each test object)
32
Similarities of Obtained Representations
Brightness is relevant for Animals
Size is relevant for Plants
33
Differential Feature Importance

• The simulation suggests that domain-general
  learning mechanisms can learn that different
  features are important for different concepts.
• The network has acquired domain-specific
  knowledge of just the sort theory theorists claim
  children know about concepts.
• It does so from the distributions of properties
  of concepts, without the aid of initial domain
  knowledge.

34
Phenomena Supporting Theory Theory

• Category goodness and feature importance.
• Differential importance of properties in
  different concepts.
• Reorganization of conceptual knowledge.

35
Conceptual Reorganization (Carey, 1985)

• Carey demonstrates that young children discover
  the unity of plants and animals as living things
  only around the age of 10.
• She suggests that the emergence of the concept of
  living thing coalesces from assimilation of
  different kinds of information, including
• Need for nutrients
• What it means to be dead vs. alive
• Reproductive properties

36
Conceptual Reorganization in the Model

• Our simulation model provides a vehicle for
  exploring how conceptual reorganization can
  occur.
• The model is capable of forming initial
  representations based on superficial appearances
• Later, it can discover shared structure that cuts
  across several different relational contexts, and
  use the emergent common structure as a basis for
  a deeper organization.

37
Reorganization Simulation

• We consider the coalescence of the superordinate
  categories plant and animal, in a situation where
  the training data initially supports a
  superficial organization based on appearance
  properties.
• In each training pattern, the input is an item
  and one of the three relations ISA, HAS, or CAN.
• The target includes all of the superficial
  appearance properties (IS properties) plus the
  properties appropriate for the relation.
• The model quickly learns representations that
  capture the superficial IS properties.
• Later, it reorganizes these representations as it
  learns the relation-dependent properties.

38
Organization of Conceptual Knowledge at Different
Points in Development
39
Phenomena Supporting Theory Theory

• Category goodness and feature importance.
• Differential importance of properties in
  different concepts.
• Reorganization of conceptual knowledge.

40
Summary

• The model exhibits several characteristics of
  human cognition that motivated the appeal to
  naïve domain theories.
• The model does these things simply by adjusting
  the weights on connections among simple
  processing units, and by propagating signals
  backward and forward through these weighted
  connections.

41
Relationship between the Model and Theory Theory

• There is a sense in which the knowledge in the
  connections plays the role of the informal domain
  theories advocated by theory theorists, and one
  might be tempted to suggest that the model is
  merely an implementation of the theory theory.
• However, it differs from the theory theory in
  several very important ways
• It provides explicit mechanisms indicating how
  domain knowledge influences semantic cognition.
• The PDP model avoids bringing in unwanted aspects
  of what we generally mean by theory
• It offers a learning process that provides a
  means for the acquisition of such knowledge.
• It demonstrates that some of the sorts of
  constraints theory-theorists have suggested might
  be innate can in fact be acquired from experience.

42
Conclusions

• In our view the theory theory should be viewed
  as more of a pre-theoretical heuristic than an
  actual theory of semantic cognition.
• Our own proposals, built on Hintons and
  Rumelharts, are far from the final word, and do
  not constitute a complete theory at this point.
• Our hope is that they will contribute, along with
  the work of many others, to the ongoing
  development of an adequate and complete theory of
  semantic cognition.