Computational%20Discovery%20of%20Communicable%20Knowledge

1
A Cognitive Architecture for Physical Agents
Pat Langley Computational Learning
Laboratory Center for the Study of Language and
Information Stanford University, Stanford,
California USA http//cll.stanford.edu/
Thanks to D. Choi, K. Cummings, N. Nejati, S.
Rogers, S. Sage, and D. Shapiro for their
contributions. This talk reports research. funded
by grants from DARPA IPTO and the National
Science Foundation, which are not responsible for
its contents.
2
Cognitive Systems

• The original goal of artificial intelligence was
  to design and implement computational artifacts
  that
• handled difficult tasks that require cognitive
  processing
• combined many capabilities into integrated
  systems
• provided insights into the nature of mind and
  intelligence.

Instead, modern AI has divided into many
subfields that care little about cognition,
systems, or intelligence. But the challenge
remains and we need far more research on
cognitive systems.
3
The Fragmentation of AI Research
?
4
The Domain of In-City Driving

• Consider driving a vehicle in a city, which
  requires
• selecting routes
• obeying traffic lights
• avoiding collisions
• being polite to others
• finding addresses
• staying in the lane
• parking safely
• stopping for pedestrians
• following other vehicles
• delivering packages
• These tasks range from low-level execution to
  high-level reasoning.

5
Newells Critique
In 1973, Allen Newell argued You cant play
twenty questions with nature and win. Instead,
he proposed that we

• move beyond isolated phenomena and capabilities
  to develop complete models of intelligent
  behavior
• demonstrate our systems intelligence on the same
  range of domains and tasks as humans can handle
• view artificial intelligence and cognitive
  psychology as close allies with distinct but
  related goals
• evaluate these systems in terms of generality and
  flexibility rather than success on a single class
  of tasks.
• However, there are different paths toward
  achieving such systems.

6
A System with Communicating Modules
software engineering / multi-agent systems
7
A System with Shared Short-Term Memory
short-term beliefs and goals
blackboard architectures
8
Integration vs. Unification

• Newells vision for research on theories of
  intelligence was that
• cognitive systems should make strong theoretical
  assumptions about the nature of the mind
• theories of intelligence should change only
  gradually, as new structures or processes are
  determined necessary
• later design choices should be constrained
  heavily by earlier ones, not made independently.

A successful framework is all about mutual
constraints, and it should provide a unified
theory of intelligent behavior. He associated
these aims with the idea of a cognitive
architecture.
9
A System with Shared Long-Term Memory
short-term beliefs and goals long-term
memory structures
cognitive architectures
10
A Constrained Cognitive Architecture
short-term beliefs and goals long-term
memory structures
11
The ICARUS Architecture
In this talk I will use one such framework ?
ICARUS ? to illustrate the advantages of
cognitive architectures. ICARUS incorporates a
variety of assumptions from psychological
theories the most basic are that

1. Short-term memories are distinct from long-term
   stores
2. Memories contain modular elements cast as list
   structures
3. Long-term structures are accessed through pattern
   matching
4. Cognition occurs in retrieval/selection/action
   cycles
5. Performance and learning compose elements in
   memory

These claims give ICARUS much in common with
other cognitive architectures like ACT-R, Soar,
and Prodigy.
12
Architectural Commitment to Memories

• A cognitive architecture makes a specific
  commitment to
• long-term memories that store knowledge and
  procedures
• short-term memories that store beliefs and
  goals
• sensori-motor memories that hold percepts and
  actions.

• For each memory, a cognitive architecture also
  commits to
• the encoding of contents in that memory
• the organization of structures within the
  memory
• the connections among structures across
  memories.

Each memory holds different content the agent
uses in activities.
13
Ideas about Representation
Cognitive psychology makes important
representational claims

• concepts and skills encode different aspects of
  knowledge that are stored as distinct cognitive
  structures
• cognition occurs in a physical context, with
  concepts and skills being grounded in perception
  and action
• many mental structures are relational in nature,
  in that they describe connections or interactions
  among objects
• long-term memories have hierarchical
  organizations that define complex structures in
  terms of simpler ones
• each element in a short-term memory is an active
  version of some structure in long-term memory.

ICARUS adopts these assumptions about the
contents of memory.
14
ICARUS Memories
Perceptual Buffer
Long-Term Conceptual Memory
Short-Term Belief Memory
Environment
Long-Term Skill Memory
Short-Term Goal Memory
Motor Buffer
15
Representing Long-Term Structures
ICARUS encodes two forms of general long-term
knowledge

• Conceptual clauses A set of relational inference
  rules with perceived objects or defined concepts
  in their antecedents
• Skill clauses A set of executable skills that
  specify
• a head that indicates a goal the skill achieves
• a single (typically defined) precondition
• a set of ordered subgoals or actions for
  achieving the goal.

These define a specialized class of hierarchical
task networks in which each task corresponds to a
goal concept. ICARUS syntax is very similar to
Nau et al.s SHOP2 formalism for hierarchical
task networks.
16
ICARUS Concepts for In-City Driving
((in-rightmost-lane ?self ?clane) percepts
( (self ?self) (segment ?seg) (line ?clane
segment ?seg)) relations ((driving-well-in-segme
nt ?self ?seg ?clane) (last-lane ?clane) (not
(lane-to-right ?clane ?anylane)))) ((driving-well
-in-segment ?self ?seg ?lane) percepts ((self
?self) (segment ?seg) (line ?lane segment ?seg))
relations ((in-segment ?self ?seg) (in-lane
?self ?lane) (aligned-with-lane-in-segment ?self
?seg ?lane) (centered-in-lane ?self ?seg
?lane) (steering-wheel-straight
?self))) ((in-lane ?self ?lane) percepts
( (self ?self segment ?seg) (line ?lane segment
?seg dist ?dist)) tests ( (gt ?dist -10)
(lt ?dist 0))) ((in-segment ?self ?seg)
percepts ( (self ?self segment ?seg) (segment
?seg)))
17
ICARUS Skills for In-City Driving
((in-rightmost-lane ?self ?line) percepts
((self ?self) (line ?line)) start
((last-lane ?line)) subgoals ((driving-well-in-s
egment ?self ?seg ?line))) ((driving-well-in-seg
ment ?self ?seg ?line) percepts ((segment
?seg) (line ?line) (self ?self)) start
((steering-wheel-straight ?self)) subgoals
((in-segment ?self ?seg) (centered-in-lane ?self
?seg ?line) (aligned-with-lane-in-segment ?self
?seg ?line) (steering-wheel-straight
?self))) ((in-segment ?self ?endsg) percepts
((self ?self speed ?speed) (intersection ?int
cross ?cross) (segment ?endsg street ?cross
angle ?angle)) start ((in-intersection-fo
r-right-turn ?self ?int)) actions ((?steer
1)))
18
Representing Short-Term Beliefs/Goals
(current-street me A) (current-segment me
g550) (lane-to-right g599 g601) (first-lane
g599) (last-lane g599) (last-lane
g601) (at-speed-for-u-turn me) (slow-for-right-tur
n me) (steering-wheel-not-straight
me) (centered-in-lane me g550 g599) (in-lane me
g599) (in-segment me g550) (on-right-side-in-segme
nt me) (intersection-behind g550
g522) (building-on-left g288) (building-on-left
g425) (building-on-left g427) (building-on-left
g429) (building-on-left g431) (building-on-left
g433) (building-on-right g287) (building-on-right
g279) (increasing-direction me) (buildings-on-righ
t g287 g279)
19
Encoding Perceived Objects
(self me speed 5 angle-of-road -0.5
steering-wheel-angle -0.1) (segment g562 street 1
dist -5.0 latdist 15.0) (line g564 length 100.0
width 0.5 dist 35.0 angle 1.1 color white segment
g562) (line g565 length 100.0 width 0.5 dist 15.0
angle 1.1 color white segment g562) (line g563
length 100.0 width 0.5 dist 25.0 angle 1.1 color
yellow segment g562) (segment g550 street A dist
oor latdist nil) (line g600 length 100.0 width
0.5 dist -15.0 angle -0.5 color white segment
g550) (line g601 length 100.0 width 0.5 dist 5.0
angle -0.5 color white segment g550) (line g599
length 100.0 width 0.5 dist -5.0 angle -0.5 color
yellow segment g550) (intersection g522 street A
cross 1 dist -5.0 latdist nil) (building g431
address 99 street A c1dist 38.2 c1angle -1.4
c2dist 57.4 c2angle -1.0) (building g425 address
25 street A c1dist 37.8 c1angle -2.8 c2dist 56.9
c2angle -3.1) (building g389 address 49 street 1
c1dist 49.2 c1angle 2.7 c2dist 53.0 c2angle
2.2) (sidewalk g471 dist 15.0 angle
-0.5) (sidewalk g474 dist 5.0 angle
1.07) (sidewalk g469 dist -25.0 angle
-0.5) (sidewalk g470 dist 45.0 angle
1.07) (stoplight g538 vcolor green hcolor red))
20
Hierarchical Structure of Long-Term Memory
ICARUS organizes both concepts and skills in a
hierarchical manner.
concepts
Each concept is defined in terms of other
concepts and/or percepts. Each skill is defined
in terms of other skills, concepts, and percepts.
skills
21
Hierarchical Structure of Long-Term Memory
ICARUS interleaves its long-term memories for
concepts and skills.
For example, the skill highlighted here refers
directly to the highlighted concepts.
22
Architectural Commitment to Processes

• In addition, a cognitive architecture makes
  commitments about
• performance processes for
• retrieval, matching, and selection
• inference and problem solving
• perception and motor control
• learning processes that
• generate new long-term knowledge structures
• refine and modulate existing structures

In most cognitive architectures, performance and
learning are tightly intertwined.
23
Ideas about Performance
Cognitive psychology makes clear claims about
performance

• humans can handle multiple goals with different
  priorities, which can interrupt tasks to which
  attention returns later
• conceptual inference, which typically occurs
  rapidly and unconsciously, is more basic than
  problem solving
• humans often resort to means-ends analysis to
  solve novel, unfamiliar problems
• mental problem solving requires greater cognitive
  resources than execution of automatized skills
• problem solving often occurs in a physical
  context, with mental processing being interleaved
  with execution.

ICARUS embodies these ideas in its performance
mechanisms.
24
ICARUS Functional Processes
Perceptual Buffer
Short-Term Belief Memory
Long-Term Conceptual Memory
Conceptual Inference
Perception
Environment
Skill Retrieval and Selection
Short-Term Goal Memory
Long-Term Skill Memory
Skill Execution
Problem Solving Skill Learning
Motor Buffer
25
ICARUS Inference-Execution Cycle
On each successive execution cycle, the ICARUS
architecture

1. places descriptions of sensed objects in the
   perceptual buffer
2. infers instances of concepts implied by the
   current situation
3. finds paths through the skill hierarchy from
   top-level goals
4. selects one or more applicable skill paths for
   execution
5. invokes the actions associated with each selected
   path.

ICARUS agents are teleoreactive (Nilsson, 1994)
in that they are executed reactively but in a
goal-directed manner.
26
Basic ICARUS Processes
ICARUS matches patterns to recognize concepts and
select skills.
concepts
Concepts are matched bottom up, starting from
percepts. Skill paths are matched top down,
starting from intentions.
skills
27
ICARUS Interleaves Execution and Problem Solving
Skill Hierarchy
Problem
Reactive Execution
?
no
impasse?
Primitive Skills
Executed plan
yes
Problem Solving
28
Interleaving Reactive Control and Problem Solving
Solve(G) Push the goal literal G onto the empty
goal stack GS. On each cycle, If the top
goal G of the goal stack GS is satisfied,
Then pop GS. Else if the goal stack GS does
not exceed the depth limit, Let S be
the skill instances whose heads unify with G.
If any applicable skill paths start from an
instance in S, Then select one of these
paths and execute it. Else let M be the
set of primitive skill instances that have not
already failed in which G is an effect.
If the set M is nonempty,
Then select a skill instance Q from M. Push
the start condition C of Q onto goal stack GS.
Else if G is a complex concept with
the unsatisfied subconcepts H and with satisfied
subconcepts F, Then if
there is a subconcept I in H that has not yet
failed, Then push
I onto the goal stack GS.
Else pop G from the goal stack GS and
store information about failure with G's parent.
Else pop G from the goal
stack GS. Store
information about failure with G's parent.
This is traditional means-ends analysis, with
three exceptions (1) conjunctive goals must be
defined concepts (2) chaining occurs over both
skills/operators and concepts/axioms and (3)
selected skills are executed whenever applicable.
29
A Successful Problem-Solving Trace
initial state
(clear C)
(hand-empty)
(unst. C B)
(clear B)
(unstack C B)
goal
(on C B)
(unst. B A)
(clear A)
(unstack B A)
(ontable A T)
(holding C)
(hand-empty)
(putdown C T)
(on B A)
(holding B)
30
Claims about Learning
Cognitive psychology has also developed ideas
about learning

• efforts to overcome impasses during problem
  solving can lead to the acquisition of new
  skills
• learning can transform backward-chaining
  heuristic search into more informed
  forward-chaining behavior
• learning is incremental and interleaved with
  performance
• structural learning involves monotonic addition
  of symbolic elements to long-term memory
• transfer to new tasks depends on the amount of
  structure shared with previously mastered tasks.

ICARUS incorporates these assumptions into its
basic operation.
31
ICARUS Learns Skills from Problem Solving
Reactive Execution
no
impasse?
Primitive Skills
Executed plan
yes
Problem Solving
Skill Learning
32
ICARUS Constraints on Skill Learning

• What determines the hierarchical structure of
  skill memory?
• The structure emerges the subproblems that arise
  during problem solving, which, because operator
  conditions and goals are single literals, form a
  semilattice.
• What determines the heads of the learned
  clauses/methods?
• The head of a learned clause is the goal literal
  that the planner achieved for the subproblem that
  produced it.
• What are the conditions on the learned
  clauses/methods?
• If the subproblem involved skill chaining, they
  are the conditions of the first subskill clause.
• If the subproblem involved concept chaining, they
  are the subconcepts that held at the subproblems
  outset.

33
Constructing Skills from a Trace
(clear C)
skill chaining
1
(hand-empty)
(unst. C B)
(clear B)
(unstack C B)
(on C B)
(unst. B A)
(clear A)
(unstack B A)
(ontable A T)
(holding C)
(hand-empty)
(putdown C T)
(on B A)
(holding B)
34
Constructing Skills from a Trace
(clear C)
1
(hand-empty)
(unst. C B)
(clear B)
(unstack C B)
(on C B)
(unst. B A)
(clear A)
(unstack B A)
skill chaining
2
(ontable A T)
(holding C)
(hand-empty)
(putdown C T)
(on B A)
(holding B)
35
Constructing Skills from a Trace
(clear C)
concept chaining
3
1
(hand-empty)
(unst. C B)
(clear B)
(unstack C B)
(on C B)
(unst. B A)
(clear A)
(unstack B A)
2
(ontable A T)
(holding C)
(hand-empty)
(putdown C T)
(on B A)
(holding B)
36
Constructing Skills from a Trace
skill chaining
(clear C)
4
3
1
(hand-empty)
(unst. C B)
(clear B)
(unstack C B)
(on C B)
(unst. B A)
(clear A)
(unstack B A)
2
(ontable A T)
(holding C)
(hand-empty)
(putdown C T)
(on B A)
(holding B)
37
Learned Skills in the Blocks World
(clear (?C) percepts ((block ?D) (block ?C))
start ((unstackable ?D ?C)) skills ((unstack
?D ?C)))(clear (?B) percepts ((block ?C)
(block ?B)) start ((on ?C ?B) (hand-empty))
skills ((unstackable ?C ?B) (unstack ?C
?B)))(unstackable (?C ?B) percepts ((block
?B) (block ?C)) start ((on ?C ?B)
(hand-empty)) skills ((clear ?C)
(hand-empty)))(hand-empty ( ) percepts
((block ?D) (table ?T1)) start ((putdownable
?D ?T1)) skills ((putdown ?D ?T1)))
Hierarchical skills are generalized traces of
successful means-ends problem solving
38
Cumulative Curves for Blocks World
39
Cumulative Curves for Blocks World
40
Cumulative Curves for FreeCell
41
Learning Skills for In-City Driving
We have also trained ICARUS to drive in our
in-city environment. We provide the system with
tasks of increasing complexity. Learning
transforms the problem-solving traces into
hierarchical skills. The agent uses these skills
to change lanes, turn, and park using only
reactive control.
42
Skill Clauses Learning for In-City Driving
((parked ?me ?g1152) percepts ( (lane-line
?g1152) (self ?me)) start ( )
subgoals ( (in-rightmost-lane ?me ?g1152)
(stopped ?me)) )((in-rightmost-lane
?me ?g1152) percepts ( (self ?me) (lane-line
?g1152)) start ( (last-lane ?g1152))
subgoals ( (driving-well-in-segment ?me ?g1101
?g1152)) ) ((driving-well-in-segment ?me ?g1101
?g1152) percepts ( (lane-line ?g1152)
(segment ?g1101) (self ?me)) start
( (steering-wheel-straight ?me))
subgoals ( (in-lane ?me ?g1152)
(centered-in-lane ?me ?g1101 ?g1152)
(aligned-with-lane-in-segment ?me ?g1101
?g1152) (steering-wheel-straight
?me)) )
43
Learning Curves for In-City Driving
44
Transfer of Skills in ICARUS

• The architecture also supports the transfer of
  knowledge in that
• skills acquired later can build on those learned
  earlier
• skill clauses are indexed by the goals they
  achieve
• conceptual inference supports mapping across
  domains.

We are exploring such effects in ICARUS as part
of a DARPA program on the transfer of learned
knowledge. Testbeds include first-person shooter
games, board games, and physics problem solving.
45
Transfer Effects in FreeCell
On 16-card FreeCell tasks, prior training aids
solution probability.
46
Transfer Effects in FreeCell
However, it also lets the system solve problems
with less effort.
47
Architectures as Programming Languages

• Cognitive architectures come with a programming
  language that
• includes a syntax linked to its representational
  assumptions
• inputs long-term knowledge and initial short-term
  elements
• provides an interpreter that runs the specified
  program
• incorporates tracing facilities to inspect system
  behavior

Such programming languages ease construction and
debugging of knowledge-based systems. For this
reason, cognitive architectures support far more
efficient development of software for intelligent
systems.
48
Programming in ICARUS

• The programming language associated with ICARUS
  comes with
• a syntax for concepts, skills, beliefs, and
  percepts
• the ability to load and parse such programs
• an interpreter for inference, execution,
  planning, and learning
• a trace package that displays system behavior
  over time

We have used this language to develop adaptive
intelligent agents in a variety of domains.
49
An ICARUS Agent for Urban Combat
50
Intellectual Precursors
ICARUS design has been influenced by many
previous efforts

• earlier research on integrated cognitive
  architectures
• especially ACT, Soar, and Prodigy
• earlier frameworks for reactive control of agents
• research on belief-desire-intention (BDI)
  architectures
• planning/execution with hierarchical transition
  networks
• work on learning macro-operators and
  search-control rules
• previous work on cumulative structure learning

However, the framework combines and extends ideas
from its various predecessors in novel ways.
51
Directions for Future Research
Future work on ICARUS should introduce additional
methods for

• forward chaining and mental simulation of skills
• learning expected utilities from skill execution
  histories
• learning new conceptual structures in addition to
  skills
• probabilistic encoding and matching of Boolean
  concepts
• flexible recognition of skills executed by other
  agents
• extension of short-term memory to store episodic
  traces.

Taken together, these features should make ICARUS
a more general and powerful cognitive
architecture.
52
Contributions of ICARUS
ICARUS is a cognitive architecture for physical
agents that

• includes separate memories for concepts and
  skills
• organizes both memories in a hierarchical
  fashion
• modulates reactive execution with goal seeking
• augments routine behavior with problem solving
  and
• learns hierarchical skills in a cumulative manner.

These ideas have their roots in cognitive
psychology, but they are also effective in
building flexible intelligent agents.
53
Concluding Remarks
We need more research on integrated intelligent
systems that

• are embedded within a unified cognitive
  architecture
• incorporate modules that provide mutual
  constraints
• demonstrate a wide range of intelligent behavior
  
• are evaluated on multiple tasks in challenging
  testbeds.

For more information about the ICARUS
architecture, see http//cll.stanford.edu
/research/ongoing/icarus/
54
End of Presentation