Knowledge Acquisition and Problem Solving

1
CS 785 Fall 2004
Knowledge Acquisition and Problem Solving
Knowledge engineering and manual knowledge
acquisition
Gheorghe Tecuci tecuci_at_gmu.eduhttp//lac.gmu.edu
/
Learning Agents Center and Computer Science
Department George Mason University
2
Overview
Typical scenario for knowledge engineeringand
manual knowledge acquisition
Basic ontology elicitation methods
Elicitation based on the personal construct theory
3
How are agents built Manual knowledge acquisition
Intelligent Agent
Problem Solving Engine
Subject Matter Expert
Knowledge
Engineer
Dialog
Programming
Knowledge Base
Results
A knowledge engineer attempts to understand how a
subject matter expert reasons and solves problems
and then encodes the acquired expertise into the
agent's knowledge base. The expert analyzes the
solutions generated by the agent (and often the
knowledge base itself) to identify errors, and
the knowledge engineer corrects the knowledge
base.
4
Why it is hard
The knowledge engineer has to become a kind of
subject matter expert in order to properly
understand experts problem solving knowledge.
This takes time and effort. Experts express
their knowledge informally, using natural
language, visual representations and common
sense, often omitting essential details that are
considered obvious. This form of knowledge is
very different from the one in which knowledge
has to be represented in the knowledge base
(which is formal, precise, and complete). This
transfer and transformation of knowledge, from
the domain expert through the knowledge engineer
to the agent, is long, painful and inefficient
(and is known as "the knowledge acquisition
bottleneck of the AI systems development
process).
5
Typical scenario for manual knowledge acquisition
Adapted from
B.G. Buchanan, D. Barstow, R. Bechtal, J.
Bennett, W. Clancey, C. Kulikowski, T. Mitchell,
D.A. Waterman, Constructing an Expert System,
in F. Hayes-Roth, D. Waterman and D. Lenat
(eds), Building Expert Systems, Addison-Wesley,
1983, pp.127-168.
6
Identification of a problem
The director of ORNL faces a problem. EPA
regulations forbid the discharge of quantities of
oil or hazardous chemicals into or upon waters of
the United States, when this discharge violates
specified quality standards. ORNL has
approximately 2000 buildings on a
200-square-mille government reservation, with 93
discharge sites entering White Oak Creek. Oil and
hazardous chemicals are stored and used
extensively at ORNL. The problem is to detect,
monitor, and contain spills of these materials.
7
Investigated solution
Develop a computer system that incorporates the
expertise of people familiar with spill detection
and containment (i.e. a knowledge-based system,
expert system or agent).
8
Participants
A knowledge engineer is assigned the job of
building the system.   The knowledge engineer
becomes familiar with the problem and the
domain.   The knowledge engineer finds an expert
on the subject who agrees to collaborate in
building the system.
What aspects may concern the expert?
9
Scope the problem to solve specify requirements
The knowledge engineer and the expert have a
series of meetings to better identify the
problem and to characterize it informally.
They decide to concentrate on identifying,
locating, and containing the spill.
10
Scope the problem to solve specify requirements

• When an accidental inland spill of an oil or
  chemical occurs,
• an emergency situation may exist, depending on
• the properties and quantity of the substance
  released,
• the location of the substance, and whether or not
  
• the substance enters a body of water.
• The observer of a spill should
• Characterize the spill and the probable hazards.
• Contain the spill material.
• Locate the source of the spill and stop any
  further release.
• Notify the Department of Environmental
  Management.

11
Understanding the expertise domain

• The knowledge engineer schedules numerous
  meetings with the expert to uncover
• basic concepts
• primitive relations
• definitions
• which are needed to talk about the addressed
  problem, and to understand it and its solutions.

12
Sample dialog knowledge engineer expert
KE Suppose you were told that a spill had been
detected in White Oak Creek one mile before it
enters White Oak Lake. What would you do to
contain the spill? SME That depends on a number
of factors. I would need to find the source in
order to prevent the possibility of further
contamination, probably by checking drains and
manholes for signs of the spill material. And it
helps to know what the spilled material is. KE
How can you tell what it is? SME Sometimes you
can tell what the substance is by its
smell. Sometimes you can tell by its color, but
that's not always reliable since dyes are used a
lot nowadays. Oil, however, floats on the surface
and forms a silvery film, while acids dissolves
completely in the water. Once you discover the
type of material spilled, you can eliminate any
building that either don't store the material at
all or don't store enough of it to account for
the spill.
13
What domain concepts can you identify in this
dialog?
KE Suppose you were told that a spill had been
detected in White Oak Creek one mile before it
enters White Oak Lake. What would you do to
contain the spill? SME That depends on a number
of factors. I would need to find the source in
order to prevent the possibility of further
contamination, probably by checking drains and
manholes for signs of the spill material. And it
helps to know what the spilled material is. KE
How can you tell what it is? SME Sometimes you
can tell what the substance is by its
smell. Sometimes you can tell by its color, but
that's not always reliable since dyes are used a
lot nowadays. Oil, however, floats on the surface
and forms a silvery film, while acids dissolves
completely in the water. Once you discover the
type of material spilled, you can eliminate any
building that either don't store the material at
all or don't store enough of it to account for
the spill.
14
Identify the basic concepts of the domain
KE Suppose you were told that a spill had been
detected in White Oak Creek one mile before it
enters White Oak Lake. What would you do to
contain the spill? SME That depends on a
number of factors. I would need to find the
source in order to prevent the possibility of
further contamination, probably by checking
drains and manholes for signs of the spill
material. And it helps to know what the spilled
material is. KE How can you tell what it
is? SME Sometimes you can tell what the
substance is by its smell. Sometimes you can tell
by its color, but that's not always reliable
since dyes are used a lot nowadays. Oil, however,
floats on the surface and forms a silvery film,
while acids dissolves completely in the water.
Once you discover the type of material spilled,
you can eliminate any building that either don't
store the material at all or don't store enough
of it to account for the spill.
15
Informally describe the identified concepts
KE Suppose you were told that a spill had been
detected in White Oak Creek one mile before it
enters White Oak Lake. What would you do to
contain the spill? SME That depends on a number
of factors. I would need to find the source in
order to prevent the possibility of further
contamination, probably by checking drains and
manholes for signs of the spill material. And it
helps to know what the spilled material is. KE
How can you tell what it is? SME Sometimes you
can tell what the substance is by its smell.
Sometimes you can tell by its color, but that's
not always reliable since dyes are used a lot
nowadays. Oil, however, floats on the surface and
forms a silvery film, while acids dissolves
completely in the water. Once you discover the
type of material spilled, you can eliminate any
building that either don't store the material at
all or don't store enough of it to account for
the spill.
Task
Identification of spill material
Spill
Attributes of spill
Source (Location) Material (Type) Volume
Material of spill
Attributes of material
Smell (Odor) Color Does it dissolve? Possible
locations Amount stored
16
Identify the basic concepts of the domain (cont.)
As a result of such dialogues, the knowledge
engineer identifies a set of concepts and
features used in this problem
Task Identification of spill material
Attributes of spill Type Oil, acid Location
ltA set of drains and manholesgt Volume ltA number
of litersgt Attributes of material Color
Silvery, clear, etc. Odor Pungent/choking,
etc. Does it dissolve? Possible locations ltA
set of buildingsgt Amount stored ltA number of
litersgt
17
Choosing the system-building language or tool
During conceptualization, the knowledge engineer
thinks also at a general system-building language
or tool for implementing the knowledge based
system. It was determined that the data are
well-structured and fairly reliable and that the
decision processes involve feedback and parallel
decisions. This suggests the use of a
rule-based language. Therefore the knowledge
engineer decides to use the rule-based language
ROSIE. ROSIE provides a general (rule-based)
inference engine, as well as a formalism for
representing the knowledge in the form of
assertions about objects and inference rules.
ROSIE could be regarded as a very general
expert system shell.
18
Represent the domain concepts object ontology
The knowledge engineer attempts to represent the
concepts in ROSIE's formalism
ASSERT each of BUILDING 3023 and BUILDING 3024 is
a building. ASSERT s6-1 is a source in BUILDING
3023. ASSERT s6-2 is a source in BUILDING
3024. ASSERT s6-1 does hold 2000 gallons of
gasoline. ASSERT s6-2 does hold 50 gallons of
acetic acid. ASSERT each of d6-1 and d6-2 is a
drain. ASSERT each of m6-1 and m6-2 is a
manhole. ASSERT any drain is a location and any
manhole is a location. ASSERT each of diesel oil,
hydraulic oil, transformer oil and gasoline is an
oil. ASSERT each of sulfuric acid, hydrochloric
acid and acetic acid is an acid. ASSERT every oil
is a possible-material of the spill and every
acid is a possible-material of the spill. ASSERT
the spill does smell of some material, e.g.
gasoline, vinegar, diesel oil. ASSERT the spill
does have some odor, e.g., a pungent/choking,
no odor. ASSERT the odor of the spill is, is
not known. ASSERT the spill does form some
appearance, e.g., a silvery film, no
film. ASSERT the spill does, does not dissolve
in water.
19
Define the problem solving rules
The knowledge engineer now uses the identified
concepts to represent the expert's method of
determining the spill material as a set of ROSIE
rules
To determine-spill-material 1 IF the spill
does not dissolve in water and the spill does
form a silvery film, THEN let the spill be
oil. 2 IF the spill does dissolve in
water and the spill does form no
film, THEN let the spill be acid. (continued on
next page)
20
Define the problem solving rules (cont.)
(continued from previous page) 3 IF the spill
oil and the odor of the spill is
known THEN choose situation IF the spill
does smell of gasoline THEN let the material of
the spill be gasoline with certainty
.9 IF the spill does smell of diesel
oil THEN let the material of the spill be
diesel oil with certainty .8. 4 IF the spill
acid and the odor of the spill is
known, THEN choose situation IF the spill
does have a pungent/choking odor THEN let the
material of the spill be hydrochloric acid
with certainty .7 IF the spill does smell of
vinegar THEN let the material of the spill be
acetic acid with certainty .8. End.
21
Verifying the problem solving rules
The knowledge engineer shows the rules to the
expert and asks for reactions
KE Here are some rules I think capture your
explanation about determining the type of
material spilled and eliminating possible spill
sources. What do you think? SME Uh-huh
(long pause). Yes, that begins to capture it. Of
course if the material is silver
nitrate it will dissolve only partially in the
water. KE I see. Well, let's add that
information to the knowledge base and see
what it looks like.
22
Refinement of the knowledge base
The knowledge engineer may now revise the
knowledge base by reformulating basic domain
concepts, and refining the rules.
Delete ASSERT the spill does, does not
dissolve in water. Add ASSERT the solubility of
the spill is some level - high, moderate,
low. Modify 1 IF the solubility of the
spill is low and the spill does form a silvery
film, THEN let the spill be oil. Add 1.5
IF the solubility of the spill is
moderate, THEN let the material of the spill be
silver-nitrate with certainty .6
23
Main phases of the agent development process
Defining the problem to solve and system to be
built requirements specification
Understanding the expertise domain
Choosing or building an agent building
tool Inference engine and representation
formalism
Development of the object ontology
Development of problem solving rules or methods
Refinement of the knowledge base
24
Main phases of the agent development process
Defining the problem to solve and system to be
built requirements specification
Research in Manual Knowledge Acquisition has
produced many systematic methods for
Understanding the expertise domain
Choosing or building an agent building
tool Inference engine and representation
formalism
Development of the object ontology
Development of problem solving rules or methods
Refinement of the knowledge base
25
Overview
Typical scenario for knowledge engineeringand
manual knowledge acquisition
Basic ontology elicitation methods
Elicitation based on the personal construct theory
26
Elicitation of the object ontology
Basic concept elicitation methods
Concept hierarchies elicitation
Relationships elicitation
27
Elicitation of the object ontology

• Elicitation of the object ontology of a domain
  means determining
• which concepts apply in the domain,
• what they mean,
• what is their relative place in the domain,
• what are the differentiating criteria
  distinguishing the similar concepts, and
• what is the organizational structure giving
  these concepts a coherence for the expert.
• In other words, this means the elicitation of the
  experts conception of his/her domain.

28
Basic concept elicitation methods
What are some natural ways of eliciting the basic
concepts of a domain?
Tutorial session delivered by the expert Ask the
expert to prepare an introductory talkoutlining
the whole domain, and to deliver it as a tutorial
session to the knowledge engineer. Then extract
concepts from the transcript of the talk.
29
Basic concept elicitation methods
What are some natural ways of eliciting the basic
concepts of a domain?
Ad-hoc list created by the expert Ask the expert
to generate a list of typical concepts and then
systematically probe for more relevant
information (e.g. using free association).
30
Basic concept elicitation methods
What are some natural ways of eliciting the basic
concepts of a domain?
Book index Extract concepts from the index of a
book describing the expertise domain.
31
Basic concept elicitation methods (cont.)
What are some natural ways of eliciting the basic
concepts of a domain?
Structured or unstructured interviewswith the
expert These are a goal-oriented method used
when the knowledge engineer wants to explore an
issue.
32
Unstructured interview with the expert
The questions and the alternative responses are
open-ended.
Example (the interview illustrated before in the
spill application)
KE Suppose you were told that a spill had been
detected in White Oak Creek one mile before it
enters White Oak Lake. What would you do to
contain the spill? SME That depends on a
number of factors. I would need to find the
source in order to prevent the possibility of
further contamination, probably by
Is this an easy to use method?
It is difficult to plan and conduct.
33
Structured interview with the expert
The questions are fixed in advance.

• Types of structured questions
• Multiple-choice questions
• Dichotomous (yes/no) questions
• Ranking scale questions

34
Types of structured questions multiple-choice
Example

• From Diabetic Foot Advisor (Awad, 1996)
• If a diabetic patient complains of foot problrms,
  who should he or she see first (check one)
• Podiatrist
• General practitioner
• Orthopedic surgeon
• Physical therapist

What are the main characteristics of this method?

• Offers specific choices.
• Faster tabulation.
• Less bias due to the way the answers are ordered.

35
Types of structured questions dichotomous
Example

• From Diabetic Foot Advisor (Awad, 1996)
• Do patients with neuropathy come for regular
  checkups?
• Yes
• No

36
Types of structured questions ranking scale
Ask the expert to arrange items in a list in
order of their importance or preference.
Example
(Awad, 1996)
37
Basic concept elicitation methods (cont.)
What are some natural ways of eliciting the basic
concepts of a domain?
Protocol analysis (think-aloud technique) Systema
tic collection and analysis of the thought
processes or problem-solving methods of an expert.
38
Protocol analysis (think-aloud technique)
Protocols (cases, scenarios) are collected by
asking experts to solve problems and to verbalize
what goes through their minds, stating directly
what they think. The solving process is carried
out in an automatic fashion while the expert
talks. The kowledge engineer does not interrupt
or ask questions. Structuring the information
elicited occurs later when the knowledge engineer
analyzes the protocol.
39
Example of a protocol Adapted from Awad (1996)
A doctor verbalizes the diagnosis of a diabetic
foot patient

• 1. This woman is in her mid to late forties.
• Being quite overweight and a diabetic, blisters
  are common occurrences.
• Pain is symptomatic of the blister.
• 4. Patient is experiencing this blister for the
  first time. She's probably more worried than
  being in pain.
• Being diabetic, blisters take a long time to
  heal. It is not likely to get worse.
• 40. I don't see broken skin or pus accumulating,
  which is a good sign.
• 41. I'm going to recommend NSD and soaking the
  foot in warm water before going to bed and after
  getting up.
• 42. Her husband will have to help.
• 43. I'm going to recommend that patient wear
  wide-toed shoes.
• So, for the moment, I am going to tell the
  patient to see me in two weeks.
• Right now, I wouldn't recommend any medical
  treatment. Surgery is the
• last thing on my mind.
• 66. I'll relay this diagnosis and decision to
  patient.

40
Elicitation experiment
Elicitation experiment with a subject matter
expert in the domain of domestic gas-fired hot
water and central heating system (Gammack, 1987).

• This experiment will be used to present basic
  methods
• Initial elicitation of the concepts
• Elicitation of a hierarchy of concepts
• Elicitation of the relationships between the
  concepts.

41
Initial elicitation of concepts
The first stage of concept elicitation is to ask
the expert to prepare an introductory talk
outlining the whole domain, and to deliver it as
a tutorial session to the knowledge engineer.
The important thing is that the expert
communicates fluently using the concepts of the
domain.
This resulted in about 90 nouns or compound
nouns, both concrete and abstract in nature.
The expert edited this list by removing
synonyms, slips of the tongue, and other aberrant
terms, which reduced the list to 75 familiar
concepts.
42
Initial elicitation of concepts (cont.)
The expert initially considered the dictionary
definition of these concepts to be adequate, but
since there is no guarantee that the expert's own
definition necessarily matches the dictionary
one, a personal definition of the concepts was
given. This produced a few new concepts, such as
"fluid", "safety", and "room". The definitions
indicated that sometimes a concept went beyond
the level of detail given in a general purpose
dictionary and sometimes it meant one very
specific idea in the context of the domain. This
illustrates an important issue Much human
expertise is likely to consist in the personal
and semantic associations (connotative meaning)
that an expert brings to domain concepts and may
result in the invention or appropriation of
personalized terms to describe esoteric or subtle
domain phenomena.
43
Initial elicitation of concepts (cont.)
The domain glossary obtained characterized the
component parts of a central heating system, such
as thermostats and radiators, but also included
general physical terms such as heat and
gravity. A second path through the transcript
yielded 42 relational concepts, usually verbs
(contains, heats, connects to, etc.). These
concepts will be used later to label
relationships between the discovered concepts.
44
Features of the basic concept elicitation methods
Which are the main strengths of this approach?
Strengths gives the knowledge engineer an
orientation to the domain. generates much
knowledge cheaply and naturally. not a
significant effort for the expert.
45
Features of the basic concept elicitation methods
Which are the main weaknesses of this approach?
Weaknesses incomplete and arbitrary
coverage the knowledge engineer needs
appropriate training and/or social skills
46
Elicitation of the object ontology
Basic concept elicitation methods
Concept hierarchies elicitation
Relationships elicitation
47
Concept hierarchy elicitation
The Card-Sort Method Type the concepts on
small individual index cards. Ask the expert to
group together the related concepts into as many
small groups as possible. Ask the expert to
label each of the groups. Ask the expert to
combine the groups into slightly larger groups,
and to label them. The result will be a
hierarchical organization of the concepts.
48
The Card-sort method illustration
Satchwell
Time Switch
Electric Time Controls
Programmer
Thermostat
Thermostat
Set Point
Rotary Control Knob
Gas Control Valve
Electricity
Control
Gas Control
Solenoid
Electrical System
Electrical Supply
Electrical Supply
Electrical Contact
Electrical Components
Fuse
Pump
Mechanical Components
Motorized Valve
Part of the hierarchy of concepts from the
card-sort method
49
Features of the Card-sort method
Which are the main strengths of this approach?
Strengths gives clusters of concepts and
hierarchical organization splits large domains
into manageable sub-areas easy to do and widely
applicable
50
Features of the Card-sort method
Which are the main weaknesses of this approach?
Weaknesses incomplete and unguided strict
hierarchy is usually too restrictive
How could we modify the method to build a tangled
hierarchy?
51
Elicitation of the object ontology
Basic concept elicitation methods
Concept hierarchies elicitation
Relationships elicitation
52
Relationships elicitation
Represents the acquired concepts into a semantic
network and acquires additional structural
knowledge Ask the expert to sort the concepts
by considering each concept C as a reference, and
identifying those related to it. Ask the
expert to order the concepts related to C along a
scale from 0 to 100, marked at the side of a
table. The values are read off the scale and
entered in a data matrix. Generate a network
from the matrix, where the nodes are the concepts
and the weighted links represent proximities.
For each pair of concepts identified as related
(e.g. value gt threshold), ask the expert what
that relationship is.
53
Relationships elicitation illustration
54
Developing the representation
For each pair of concepts identified by the
expert as relatable, ask what that relationship
was. This task produced 248 relationships. This
number was effectively reduced to around 124 due
to symmetry. Example of elicited
relationships part-of (radiator, primary
circuit ) feeds (water supply, header
tank) warms (radiator, air) Sometimes relations
were not so direct "boiler supplies heat
that causes water expansion that requires
header tank This suggests the relationship nece
ssitates (boiler, header tank)
55
Features of the relationships elicitation method
Which are the main strengths of this approach?
Strengths gives information on the domain
structure in the form of a network shows
which links are likely to be meaningful
organizes the elicitation of semantic
relationships
56
Features of the relationships elicitation method
Which are the main weaknesses of this approach?
Weaknesses results depend on various parameter
settings requires more time from the expert
combinatorial explosion limits its applicability
57
Overview
Typical scenario for knowledge engineeringand
manual knowledge acquisition
Basic ontology elicitation methods
Elicitation based on the personal construct theory
58
Elicitation based on the personal construct theory
The personal construct theory
What is a repertory grid
Elicitation of repertory grids
Grid analysis
Features of the repertory grid approach
59
The personal construct theory
A model of human thinking developed in 1955 by
the psychologist George Kelly, to study
psychiatry. Basic idea of the theory Each
person is a scientist with a personal model of
the world around him. He creates personal
constructs that classify his personal
observations or experience of the world,
developing theories that allow him to anticipate,
and to act in accordance with his
anticipation. A personal construct is therefore
an attribute whose values can distinguish a
subgroup of objects from another one. This
theory was used to develop techniques for
eliciting a subject matter experts personal
constructs with respect to his domain of
expertise.
60
Personal constructs illustration
Example of constructs (or dichotomous
distinctions) characterizing an employee for the
purpose of staff appraisal intelligent -
dim mild - abrasive ideas person - staid
Each person can be rated according to the above
constructs e.g. John is mild (i.e. John is not
abrasive)
How can we refine this characterization so that
we describes different persons more acurately?
61
Personal constructs illustration
How can we refine this characterization so that
we describes different persons more acurately?
The rating can be more refined (along the
mild-abrasive construct)
very mild
mild
very abrasive
abrasive
neutral





mild
abrasive
2
5
4
3
1





mild
abrasive
62
Elicitation based on the personal construct theory
The personal construct theory
What is a repertory grid
Elicitation of repertory grids
Grid analysis
Features of the repertory grid approach
63
What is a repertory grid
A repertory grid is a representation of a
persons (or experts) view of a particular
problem. It is a two-way classification of a set
of elements based on a set of constructs. Example
of a repertory grid for staff appraisal
1
5
64
Elicitation based on the personal construct theory
The personal construct theory
What is a repertory grid
Elicitation of repertory grids
Grid analysis
Features of the repertory grid approach
65
Elicitation of repertory grids Sample session
with the Pegasus system
Type in your purpose for doing this grid staff
appraisal Name some of the elements Dick, Liz,
Bob, Paul, Ann, Don, Mary
How can we elicit constructs, that is attributes
that differentiate between these persons?
66
Elicitation of repertory grids Sample session
How can we elicit constructs, that is attributes
that differentiate between these persons?
The triad method (or the minimal context
method) The elements are presented in groups of
three, three being the lowest number that will
produce both a similarity and a difference. The
subject is asked to say in what way two are alike
and thereby different from the third. This is the
emergent pole of the construct. The implicit
pole may be elicited by the difference method (in
what way does the singleton differ from the pair)
or by the opposite method (what would be the
opposite of the description of the pair).
67
Elicitation of repertory grids Sample session
Triad for elicitation of qualities Dick, Liz,
Bob Can you choose two of these elements which
are in some way alike and different from the
other one ? Yes Which is the different one ? Bob
Now I want you to think about what you have in
mind when you separate the pair from the other
one. How can you describe the two ends or poles
of the scale which discriminates Dick and Liz on
the left pole from Bob at the right pole ? left
pole rated 1 intelligent right pole rated
5 dim
68
Elicitation of repertory grids Sample session
Now that we have identified the intelligent-dim
construct, how can we elicit additional knowledge
using it?
By asking the expert to characterize the other
persons based on this construct
According to how you feel about the considered
persons, please assign to each of them a
provisional value from 1 (intelligent) to
(dim)5 Dick 1 Liz 1 Bob 5 Paul 5 Ann 3
Don 3 Mary 5 Ruth 4 Rob 5
69
Elicitation of repertory grids Sample session
1
5
70
Elicitation of repertory grids Sample session
How would the elicitation continue?
The session will continue, with Pegasus
presenting other triads for construct
elicitation, and the user defining the
corresponding constructs.
The current grid is
1
5
71
Elicitation of repertory grids Sample session
Consider the descriptions of Ann and Don. What do
you notice?
1
5
intelligent
L1 1 1 4 5 3 3 5 2
3 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 R5 less motivated
reliable
L6 3 2 2 5 1 1 5 1
2 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 R8 staid
R
A
M
R
B
D
D
P
L
o
n
a
u
o
i
o
a
i
b
n
r
t
b
c
n
u
z
y
h
k
l
How can we use the fact that Ann and Don have
similar descriptions to elicit additional
knowledge?
72
Elicitation of repertory grids Sample session
How can we use the fact that Ann and Don have
similar descriptions to elicit additional
knowledge?
Pegasus may direct the user in defining new
constructs that distinguish between the elements
that are very similar with respect to current
constructs.
1
5
intelligent
L1 1 1 4 5 3 3 5 2
3 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 R5 less motivated
reliable
L6 3 2 2 5 1 1 5 1
2 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 R8 staid
R
A
M
R
B
D
D
P
L
o
n
a
u
o
i
o
a
i
b
n
r
t
b
c
n
u
z
y
h
k
l
73
Elicitation of repertory grids Sample session
Ann and Don are matched at the 90 level. This
means that so far you have not distinguished
between Ann and Don. Do you want to split this ?
Yes
Think of a construct which separates Ann from
Don, with Ann on the left pole and Don on the
right pole. left pole rated 1 self
starters right pole rated 5 need a push
How can we use the new construct to elicit
additional knowledge?
74
Elicitation of repertory grids Sample session
How can we use the new construct to elicit
additional knowledge?
According to how you feel about the considered
persons, with respect to the self starters -
need a push construct, please assign to each of
them a provisional value from 1 (self starters)
to 5 (need a push) Dick 2 Liz 1 Bob 5 Paul
5 Ann 1 Don 5 Mary 5
75
Elicitation of repertory grids Sample session
Consider the descriptions of the constructs
intelligent - dim and little supervision
required need supervision. What do you notice?
1
5
How can we use the the similarity of these two
constructors to elicit additional knowledge?
76
Elicitation of repertory grids Sample session
Pegasus directs the user in defining new elements
that distinguish between the constructs that are
very similar with respect to current elements.
1
5
77
Elicitation of repertory grids Sample session
The two constructs you called intelligent -
dim little supervision-reqd - need
supervision are matched at 66 percent level. This
means that most of the time you are saying
intelligent you are also saying little
supervision required and most of the time you are
saying dim you are also saying need
supervision. Think of another element which is
either intelligent and needs supervision or
little supervision required and dim. Do you know
such a person ? John
How can we elicit additional knoiwledge about
John?
78
Elicitation of repertory grids Sample session
How can we elicit additional knoiwledge about
John?
Type in the ratings for John on each construct.
Left pole rated 1, right pole rated
5. intelligent - dim 5 willing -
unwilling 2 new boy - old sweats 3 little
supervision reqd - need supervision 3 motivated
- less motivated 2 ...
79
Elicitation of repertory grids Sample session
The final grid is
1
5
80
Grid analysis inferring new knowledge from grids
A repertory grid can be viewed a set of feature
vectors, each characterizing an element along the
dimensions indicated by the constructs.
81
Grid analysis inferring new knowledge from grids
How can we infer and elicit additional knowledge
from the grid? Hint Think of the card sort
method.
82
Hierarchical clustering of repertory grids
Clusters similar elements and attributes,
prompting the expert to name the clusters
intelligent
L1 1 1 4 5 3 3 5 2
3 5 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 2 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 3 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 3 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 2 R5 less motivated
researchoriented
reliable
L6 3 2 2 5 1 1 5 1
2 3 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 5 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 4 R8 staid
self-starters
L9 2 1 5 5 1 3 5 3
4 5 R9 need a push
creative
L10 1 1 5 5 2 3 4 3
4 5 R10 noncreative
L11 4 3 4 2 3 5 1 4
5 5 R11 unhelpful
helpful
professional
L12 1 2 3 3 2 1 5 2
4 4 R12 less professional
overall rating high
L13 2 1 3 4 1 2 5 2
3 4 R13 overall rating low
L14 2 2 5 4 3 5 1 5
3 1 R14 tidy
messers

J
D
R
D
A
M
R
B
P
L
o
i
o
o
n
a
u
o
a
i
h
c
b
n
n
r
t
b
u
z
n
k
y
h
l
83
Grid analysis inferring new knowledge from grids
How can we obtain inference rules from a grid?
84
Grid analysis inferring rules from grids
Consider the various individuals as positive
examples or as negative examples of an output
attribute (e.g. overal rating high)
Which individuals are positive examples of
overal rating high?
Which individuals are negative examples of
overal rating high?
85
Examples
We may consider that individuals with 4 or 5 are
positive examples of overall rating high, and
those with 1 or 2 are negative examples. We
ignore those with 3
intelligent
L1 1 1 4 5 3 3 5 2
3 5 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 2 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 3 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 3 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 2 R5 less motivated
reliable
L6 3 2 2 5 1 1 5 1
2 3 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 5 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 4 R8 staid
self-starters
L9 2 1 5 5 1 3 5 3
4 5 R9 need a push
creative
L10 1 1 5 5 2 3 4 3
4 5 R10 noncreative
L11 4 3 4 2 3 5 1 4
5 5 R11 unhelpful
helpful
professional
L12 1 2 3 3 2 1 5 2
4 4 R12 less professional
overall rating high
L13 2 1 3 4 1 2 5 2
3 4 R13 overall rating low
L14 2 2 5 4 3 5 1 5
3 1 R14 tidy
messers

J
D
R
D
A
M
R
B
P
L
o
i
o
o
n
a
u
Positive examples of overall rating high
o
a
i
Negative examples of overall rating high
h
c
b
n
n
r
t
b
u
z
n
k
y
h
l
86
Rule induction from repertory grids
To learn a rule from these examples we would need
to describe each of them. How can we describe a
person in terms of its attributes?
intelligent
L1 1 1 4 5 3 3 5 2
3 5 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 2 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 3 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 3 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 2 R5 less motivated
reliable
L6 3 2 2 5 1 1 5 1
2 3 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 5 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 4 R8 staid
self-starters
L9 2 1 5 5 1 3 5 3
4 5 R9 need a push
creative
L10 1 1 5 5 2 3 4 3
4 5 R10 noncreative
L11 4 3 4 2 3 5 1 4
5 5 R11 unhelpful
helpful
professional
L12 1 2 3 3 2 1 5 2
4 4 R12 less professional
overall rating high
L13 2 1 3 4 1 2 5 2
3 4 R13 overall rating low
L14 2 2 5 4 3 5 1 5
3 1 R14 tidy
messers

J
D
R
D
A
M
R
B
P
L
o
i
o
o
n
a
u
Positive examples of overall rating high
o
a
i
Negative examples of overall rating high
h
c
b
n
n
r
t
b
u
z
n
k
y
h
l
87
Examples description
We may consider the poles of the constructs as
being predicates (true for a person if the
corresonding value is 4 or 5) intelligent(Dick),
willing(Dick), new boy(Dick), motivated(Dick),
ideas man (Dick), unhelpful(Dick).
1
5
intelligent
L1 1 1 4 5 3 3 5 2
3 5 R1 dim
willing
L2 1 2 4 5 1 1 4 3
1 2 R2 unwilling
new boy
L3 1 2 3 5 4 4 4 1
4 3 R3 old sweats
little supervision reqd
L4 3 1 4 5 2 1 5 2
2 3 R4 need supervision
motivated
L5 1 1 4 5 2 2 5 3
3 2 R5 less motivated
reliable
L6 3 2 2 5 1 1 5 1
2 3 R6 not so reliable
mild
L7 3 4 5 2 2 3 1 5
4 5 R7 abrasive
ideas men
L8 1 1 5 4 2 3 1 3
4 4 R8 staid
self-starters
L9 2 1 5 5 1 3 5 3
4 5 R9 need a push
creative
L10 1 1 5 5 2 3 4 3
4 5 R10 noncreative
L11 4 3 4 2 3 5 1 4
5 5 R11 unhelpful
helpful
professional
L12 1 2 3 3 2 1 5 2
4 4 R12 less professional
overall rating high
L13 2 1 3 4 1 2 5 2
3 4 R13 overall rating low
L14 2 2 5 4 3 5 1 5
3 1 R14 tidy
messers

J
D
R
D
A
M
R
B
P
L
o
i
o
o
n
a
u
Positive examples of overall rating high
o
a
i
Negative examples of overall rating high
h
c
b
n
n
r
t
b
u
z
n
k
y
h
l
88
Rule induction from repertory grids (cont.)
The description of each element in the grid is a
positive or a negative example of an output
attribute (e.g. overall rating high) overall-rat
ing-high(Dick) Ü intelligent(Dick),
willing(Dick), new-boy(Dick),
little-sprv-req(Dick), motivated(Dick),
ideas-man(Dick), self-starters(Dick),
... overall-rating-high(Paul) Ü dim(Paul),
unwilling(Paul), experienced(Paul),
need-supervision(Paul), less-motivated(P
aul), not-so-reliable(Paul), mild(Paul),
... A rule for the output attribute is learned
through empirical induction from such
examples overall-rating-high(x) Ü
professional(x)
89
KSS0 An integrated knowledge elicitation and
inductive learning system (Gaines and Shaw, 1992)
Consists of the following modules ELICIT
elicits repertory grids from the expert FOCUS
hierarchically clusters elements and constructs
prompting the expert to add higher-level
constructs structuring the domain PRINCOM
spatially clusters elements and constructs
prompting the expert to add higher-level
constructs structuring the domain SOCIO compares
the structures for the same domain generated by
different experts INDUCT induces rules from the
repertory grid EXPORT transfers the results of
grid elicitation and analysis to an expert system
shell.
90
Features of the repertory grid approach
Which are some strengths of the repertory grids?
Strengths Repertory grids can be easily
elicited from a subject matter expert. Other
concepts and inference rules can be learned from
repertory grids, although the number of examples
is small.
91
Features of the repertory grid approach
Which are some weaknesses of the repertory grids?

• Weaknesses
• Other concepts and inference rules can be learned
  from repertory grids, but the number of examples
  is small and may not lead to accurate knowledge.
• More complex knowledge structures are difficult
  to generate from repertory grids since the grids
  are oriented toward representing declarative
  attribute-based knowledge

92
Exercises
Define a repertory grid for choosing a course to
enroll in.
Define a repertory grid for choosing a car.
Define a repertory grid for choosing a
dissertation director.
93
Recommended reading
G. Tecuci, Lecture Notes on Systematic
Elicitation of Expert Knowledge. B.G. Buchanan,
D. Barstow, R. Bechtal, J. Bennett, W. Clancey,
C. Kulikowski, T. Mitchell, D.A. Waterman,
Constructing an Expert System, in F. Hayes-Roth,
D. Waterman and D. Lenat (eds), Building Expert
Systems, Addison-Wesley, 1983, pp.127-168. John
G. Gammack, Different Techniques and Different
Aspects on Declarative Knowledge, in Alison L.
Kidd (ed), Knowledge Acquisition for Expert
Systems A Practical Handbook, Plenum Press,
1987. Shaw M.L.G. and Gaines B.R., An
interactive knowledge elicitation technique using
personal construct technology, in Alison L. Kidd
(ed), Knowledge Acquisition for Expert Systems A
Practical Handbook, Plenum Press, 1987.