Development of a Cognitive Agent

1
Development ofa Cognitive Agent

• Stuart C. Shapiro
• Department of Computer Science and Engineering
• and Center for Cognitive Science
• State University of New York at Buffalo
• shapiro_at_cse.buffalo.edu

2
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

3
Goal

• A computational cognitive agent that can
• Understand and communicate in English
• Discuss specific, generic, and rule-like
  information
• Reason
• Discuss acts and plans
• Sense
• Act
• Remember and report what it has sensed and done.

4
Embodied Cassie

• A computational cognitive agent
• Embodied in hardware
• or Software-Simulated
• Based on SNePS and GLAIR.

5
SNePS

• Knowledge Representation and Reasoning
• Propositions as Terms
• SNIP SNePS Inference Package
• Bi-Directional Inference
• SNeBR SNePS Belief Revision
• SNeRE SNePS Rational Engine
• Interface Languages
• SNePSUL Lisp-Like
• SNePSLOG Logic-Like
• GATN for Fragments of English.

6
GLAIR Architecture
Grounded Layered Architecture with Integrated
Reasoning
Knowledge Level
NL
SNePS
Perceptuo-Motor Level
Sensory-Actuator Level
Vision
Sonar
Motion
Proprioception
7
Interaction with Cassie
(Current) Set of Beliefs SNePS
English (Statement, Question, Command)
Reasoning Clarification Dialogue Looking in World
GATN Parser
(Updated) Set of Beliefs SNePS
Actions SNeRE
(New Belief) SNePS
Answer SNIP
GATN Generator
Reasoning
English sentence expressing new belief
answering question reporting actions
8
Cassie, the FEVAHR
9
Cassie in the Lab
10
Garnet Simulation World
11
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

12
Entities, Terms, Symbols, Objects

• Cassies mental entity a person named Bill
• SNePS term B5
• Object in world

13
Intensional Representation

• Intensional entities are distinct even if
  coreferential.
• The morning star is the evening star.
• George IV wondered if Scott was the author of
  Waverly.

14
McCarthys Telephone Number Problem

• Mary's telephone number is Mike's telephone
  number.
• I understand that Mike's telephone number is
  Mary's telephone number.
• Pat knew Mike's telephone number.
• I understand that Pat knew Mike's telephone
  number.
• Pat dialed Mike's telephone number.
• I understand that Pat dialed Mike's telephone
  number.

15
Answering the Telephone Number Problem

• Did Pat dial Mary's telephone number?
• Yes, Pat dialed Mary's telephone number.
• Did Pat know Mary's telephone number?
• I don't know.

16
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

17
Logic for NLU Commonsense Reasoning

• Either Pat is a man or Pat is a woman or Pat is
  a robot.
• I understand that Pat is a robot or Pat is a
  woman or Pat is a man.
• Pat is a woman.
• I understand that Pat is a woman.
• What is Pat?
• Pat is a woman and Pat is not a robot and Pat
  is not a man.

18
Representation in FOPL?

• Man(Pat) ? Woman(Pat) ? Robot(Pat)
• but dont want inclusive or
• Man(Pat) Woman(Pat) Robot(Pat)

T
T
T
F
T
So dont want exclusive or either
19
andor

• andor(i, j)Pi, ..., Pn
• True iff at least i, and at most j of the Pi are
  True

20
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

21
Discussing Propositions

• That Bill is sweet is Mary's favorite
  proposition.
• I understand that Mary's favorite proposition is
  that Bill is sweet.
• Mike believes Mary's favorite proposition.
• I understand that Mike believes that Bill is
  sweet.

22
Discussing Sentences

• Hamlet said To be or not to be'.
• I understand that Hamlet said To be or not To
  be'.
• Polonius said Neither a borrower nor a lender
  be'.
• I understand that Polonius said Neither a
  borrower nor a lender be'.
• Who said To be or not to be'?
• Hamlet said To be or not To be'.

23
Representing Propositionsand Sentences

• Propositions and sentences must be
• First-class entities of the domain
• Represented by terms

24
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

25
Symbol Grounding byPerception and Action

• Agent can demonstrate understanding
• Of properties and descriptions
• By locating correct objects in the world
• Of verbs
• By acting properly in the world

26
Symbol-Grounding 1
Go to the green robot.
27
Symbol-Grounding 2
Go to the green robot.
I went to the green robot. I am near the green
robot.
28
Alignment

• Terms at GLAIR KR level
• Aligned with
• Descriptions and procedures
• At GLAIR PM level

29
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

30
Representation and Use of Indexicals

• Words whose meanings are determined by occasion
  of use
• E.g. I, you, now, then, here, there
• Deictic Center
• I SNePS term representing Cassie
• YOU person Cassie is talking with
• NOW current time.

31
Analysis of Indexicals(in input)

• First person pronouns YOU
• Second person pronouns I
• here location of YOU.

32
Generation of Indexicals

• I First person pronouns
• YOU Second person pronouns
• NOW used to determine tense.

33
Use of Indexicals 1
Come here.
34
Use of Indexicals 2
Come here.
I came to you, Stu. I am near you.
35
Use of Indexicals 3
Who am I?
Your name is Stu and you are a person.
Who have you talked to?
I am talking to you.
Talk to Bill.
I am talking to you, Bill.
Come here.
36
Use of Indexicals 4
Come here.
I found you. I am looking at you.
37
Use of Indexicals 5
Come here.
I found you. I am looking at you.
I came to you. I am near you.
38
Use of Indexicals 6
Who am I?
Your name is Bill and you are a person.
Who are you?
I am the FEVAHR and my name is Cassie.
Who have you talked to?
I talked to Stu and I am talking to you.
39
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

40
A Personal Sense of Time

• NOW contains SNePS term representing current
  time.
• NOW moves when Cassie acts.

41
Representation of Time
before
after
before
after
!
!
!
event
?????????????
time
agent
act
B1
action
object
B6
I
lex
NOW
42
Movement of Time
t1
43
Performing a Punctual Act
t1
44
Performing a Durative Act
t1
45
Keeping Track of Time 1
Who are you looking at? I am looking at you, Stu.
Come here.
46
Keeping Track of Time 2
Who are you looking at? I am looking at you.
Come here.
I came to you. I am near you.
Who are you looking at? I am looking at you.
47
Keeping Track of Time 3
Find Bill. I found Bill. I am looking at Bill.
Who are you looking at? I looked at you and I am
looking at Bill.
Who are you talking to? I am talking to you.
48
Keeping Track of Time 4
Follow a red robot. I found a red robot. I am
looking at a red robot.
49
Keeping Track of Time 5
Follow a red robot. I found a red robot. I am
looking at a red robot.
I went to a red robot. I am near a red robot. I
am following a red robot.
50
Keeping Track of Time 6
Who are you talking to? I am talking to you.
Who am I? Your name is Stu and you are a person.
Stop. I stopped.
51
Keeping Track of Time 7
Who are you looking at? I looked at you and I
looked at Bill and I looked at a red robot.
Who are you following? I followed a red robot.
Who are you talking to? I am talking to you.
52
Outline

• Introduction
• Intensional Representation
• Logic for NLU and Commonsense Reasoning
• Discussing Propositions and Sentences
• Symbol-Grounding by Perception and Action
• Representation and Use of Indexicals
• A Personal Sense of Time
• Summary

53
Goal

• A computational cognitive agent/robot
• That can communicate in natural language.

54
Intensional Representation

• SNePS terms represent mental entities.
• May assert that two entities are coreferential.
• Relations/acts may be declared transparent.

55
Logic for NLU and Commonsense Reasoning

• Designed logical connectives and rules of
  inference
• More appropriate than in standard FOPC.

56
Discussing Propositionsand Sentences

• Propositions and sentences are first-class
  entities.

57
Symbol-Grounding by Perception and Action

• Use of GLAIR architecture to connect entities
  with descriptions/functions used by sensors and
  effectors.

58
Representation and Useof Indexicals

• Use of Deictic Center for parser to interpret
  indexicals as current referents
• And for generator to generate indexicals from
  current referents.

59
A Personal Sense of Time

• NOW is current time.
• Updated when Cassie acts.

60
For More Information

• Personnel
• Manual
• Tutorial
• Bibliography
• ftpable SNePS source code
• etc.
• http//www.cse.buffalo.edu/sneps/