Modeling Preferences

1
Modeling Preferences

• Farrokh Alemi, Ph.D.
• Sunday, February 06, 2005

2
Objectives

• Model decision makers preferences
• Learn how to interact with a policymaker in order
  to model their values
• Create a mathematical model that assigns higher
  numbers to more preferred options
• Test the accuracy/validity of the model
• Focused on one decision makers values

3
Historical Basis

• Bernoulli
• Von Neumann
• Edwards
• Von Winterfeldt Edwards

4
What is a model of values?

• Value models help us quantify a person's
  preferences
• Assign numbers to options so that higher numbers
  reflect more preferred options
• Assumptions
• Decision makers must select from several options
• The selection requires grading the preferences
  for the options.
• Preferences are quantified by examining the
  various attributes (characteristics, dimensions,
  or features) of the options.
• Overall preference is a function of decision
  makers preferences on each attribute
• Electronic Medical Record (EMR) system

5
Mathematical Model
An Additive Multi-Attribute Value Model
Overall Value V (A1) V (A2)   ... V (An)
Value on Attribute 1
Value on Attribute 2
Value on Attribute n
6
Why Model Preferences?

• To clarify and communicate
• To aid decision making in complex situations
• To replace decision maker in repetitive tasks
• To measure hard to measure concepts

7
Misleading Numbers

• Model scores are rough approximate measures of
  preference
• Nominal scale
• Ordinal scale
• Interval scale

8
Step 1. Would It Help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

9
Step1. Would it help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

• Administrator

10
Step1. Would it help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

• Administrator
• Judge efficiency of providers practice

11
Step1. Would it help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

• Administrator
• Judge efficiency of providers practice
• Severity of illness

12
Step1. Would it help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

• Administrator
• Judge efficiency of providers practice
• Severity of illness

• What plans would change if the judgment were
  different?
• What is being done now?
• If no one makes a judgment about the underlying
  concept, would it really matter, and who would
  complain?
• Would it be useful to tell how the judgment was
  made, or is it better to leave matters rather
  ambiguous?
• Must we choose among options, or should we let
  things unfold on their own?
• Is a subjective component critical to the
  judgment, or can it be based on objective
  standards?

13
Step 1.4 Would It Help?

• Who decides?
• What must be done?
• What judgments must be made?
• How can the model of the judgment be used?

• Administrator
• Judge efficiency of providers practice
• Severity of illness
• Model can be used repeatedly to analyze medical
  records

14
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

15
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

16
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

17
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

18
AnalystCan you recall a specific patient with a
very poor prognosis?
19
Expert I work in a referral center, and we see
a lot of severely ill patients. They seem to have
many illness and are unable to recover
completely, so they continue to worsen.
20
Analyst Tell me about a recent patient who was
severely ill.
21
Expert A 28-year-old homosexual male patient
deteriorated rapidly. He kept fighting recurrent
influenza and died from gastrointestinal cancer.
The real problem was that he couldn't tolerate
AZT, so we couldn't help him much. Once a person
has cancer, we can do little to maintain him.
22
Analyst Tell me about a patient with a good
prognosis, say close to five years.
23
Expert Well, let me think. A year ago we had a
32-year-old male patient diagnosed with AIDS who
has not had serious disease since--a few skin
infections but nothing serious. His spirit is up,
he continues working, and we have every reason to
expect he will survive four or five years.
24
Analyst What key difference between the two
patients made you realize that the first patient
had a poorer prognosis than the second?
25
 Expert That's a difficult question- patients
are so different from each other that it's tough
to point at one characteristic. But if you really
push me, I would say two characteristics the
history of illness and the ability to tolerate
AZT.
26
 Analyst What about the history is relevant?
27
Expert If I must predict a prognosis, I want to
know whether he has had serious illness in vital
organs.
28
 Analyst Which organs?
29
Expert Brain, heart, and lungs are more
important than, say, skin.
30
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

31
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

32
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

33
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

34
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

35
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes and do not interrupt

36
Step 2. Select Attributes

• Introduce yourself and your purpose
• Be judicious about pausing
• Ask the experts to introduce themselves
• Start with tangible examples
• Ask directly for additional attributes
• Arrange the attributes in a hierarchy
• Always use the expert's terminology
• Use prompts that feel most natural
• Take notes, and do not interrupt

37
Step 3. Do It Again

• Use two different perspectives
• Survival versus mortality prompts
• Check assumptions
• Attributes are exhaustive
• Attributes are not redundant
• Attributes are important to decision maker
• Attributes are not preferentially dependent

38
Attributes Judged Important in Measuring Severity
of AIDS

• Age
• Race
• Transmission mode
• Defining diagnosis 
• Time since defining diagnosis
• Diseases of nervous system
• Disseminated diseases
• Gastrointestinal diseases
• Skin diseases

• Lung diseases
• Heart diseases
•  Recurrence of a disease
• Functioning of the organs
• Co-morbidity
• Psychiatric co morbidity
• Nutritional status
• Drug markers
• Functional impairment

39
Step 4. Set Attribute Levels

• Decide on the best and worst
• Select a target population and ask the expert to
  describe the possible range of the attribute.
• Avoid adjectives, describe the levels
• Decide on levels in between
• Fill out the gaps to describe all possible
  situations

40
Analyst I understand that patients on total
parenteral treatment have the worst prognosis.
Can you think of other relatively common
conditions with a slightly better prognosis?
41
Expert Well, a host of things can happen. Pick
up any book . on nutritional diseases and you
find all kinds of things.
42
Analyst Right, but can you give me three or
four examples?
43
Expert Sure. The patient may be on antiemetics
or nutritional supplements.
44
Analyst Do these levels-include a level with a
moderately poor prognosis and one with a
relatively good prognosis?
45
Expert Not really. If you want a level
indicative of moderately poor prognosis, then you
should include whether the patient is receiving
Lomotil or Imodium. 
46
Step 4. Set Attribute Levels

• Decide on the best and worst
• Select a target population and ask the expert to
  describe the possible range of the attribute.
• Avoid adjectives, describe the levels
• Decide on levels in between
• Fill out the gaps to describe all possible
  situations

47
Levels for Skin Infection

• No skin disorder
• Kaposi's sarcoma
• Shingles
• Herpes complex
• Candida or mucus
• Thrush

48
Step 5. Assign Values to Single Attributes

• Double anchored assessment method
• 100 to best
• 0 to worst
• Rate the remaining levels
• Check against different anchors

49
Analyst Which among the skin disorders has the
worst prognosis?
50
Expert None is really that serious.
51
Analyst Yes, I understand that, but which is,
the most serious?
52
Expert Patients with thrush perhaps have a
worse prognosis than patients with other skin
infections.
53
Analyst Let's rate the severity of thrush at 100
and place the severity of no skin disorder at 0.
How would you rate shingles?
54
Expert Shingles is almost as serious as thrush.
55
Analyst This tells me that you might rate the
severity of shingles nearer 100 than 0. Where
exactly would you rate it?
56
Expert Maybe 90.
57
Analyst Can you now rate the remaining levels
for skin disorders?
58
Step 5. Assign Values to Single Attributes

• Double anchored assessment method
• 100 to best
• 0 to worst
• Rate the remaining levels
• Check against different anchors

59
Assigned Values to Skin Infection Attribute
60
Step 6. Choose an Aggregation Rule

• Additive multi-attribute utility model
• Used often
• Relatively robust
• Compensatory
• Multiplicative multi-attribute utility model
• Used less often

61
Step 7. Estimate Weights

• Assessing the ratio of the importance of two
  attributes
• Attributes are rank ordered
• Least important is assigned 10 points
• Decision maker is asked to rate next more
  important attributes by saying how many times
  more important it is

62
Analyst Which of the three attributes is most
important?
63
Expert Well, they are all important, but
patients with either lung infections or GI
diseases have worse prognoses than patients with
skin infections.
64
Analyst Do lung infections have a worse
prognosis than GI diseases?
65
Expert That's more difficult to answer. No. I
would say that for all practical purposes, they
have the same prognosis. Well, now that I think
about it, perhaps patients with GI diseases have
a slightly worse prognosis.
66
Step 7. Estimate Weights

• Assessing the ratio of the importance of two
  attributes
• Attributes are rank ordered
• Least important is assigned 10 points
• Decision maker is asked to rate next more
  important attributes by saying how many times
  more important it is

67
Analyst Let's say that we arbitrarily rate the
importance of skin infection in determining
prognosis at 10 points. GI diseases are how many
times more important than skin infections?
68
Analyst Let's say that we arbitrarily rate the
importance of skin infection in determining
prognosis at 10 points. GI diseases are how many
times more important than skin infections?
69
Expert Quite a bit. Maybe three times.
70
Expert Quite a bit. Maybe three times.
71
Analyst That is, if we assign 10 points to skin
infections, we should assign 30 points to the
importance of GI diseases?
72
Expert Yes, that sounds right.
73
Analyst How about lung infections? How many
more times important are they than GI diseases? 
74
Expert I would say about the same.
75
Analyst Would you consider lung infections
three times more serious than skin infections?
76
Expert Yes, I think that should be about right.
77
Step 7. Estimate Weights

• Assessing the ratio of the importance of two
  attributes
• Attributes are rank ordered
• Least important is assigned 10 points
• Decision maker is asked to rate next more
  important attributes by saying how many times
  more important it is

78
(No Transcript)
79
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

80
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

81
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

82
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

83
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

84
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

85
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

86
Step 8. Evaluate the Model

• Context dependence
• Face validity
• Based on available data
• Simple to use
• Inter-rater reliability
• Construct validity
• Simulates decision makers judgments
• Predicts behavior

87
Testing Accuracy of the Severity Index for AIDS

• Simulating decision makers judgments
• Generated scenarios of patients
• Asked experts to estimate prognosis of the
  patients
• Calculated model scores for each patient
• Compared model scores to average of experts
  estimates
• Test of predictive validity
• Followed patients over time
• Compared model scores to observed survival of
  patients

88
Results

• In 81 cases, The index developed from judgment of
  experts was more predictive of months of survival
  than Composite Laboratory Index.
• In 26 hospice cases, severity index developed
  from judgment of experts was able to correctly
  classify patients who had less than 6 months to
  live.

89
Take Home Lesson

• Spend most of your time on getting the right
  attributes. A rough scoring is usually
  sufficient.