General Issues in Scaling

1
General Issues in Scaling
2
What Is a Scale?
The assignment...
3
What Is a Scale?
The assignment...
...of objects...
Are you willing to permit immigrants to live in
your country? Are you willing to permit
immigrants to live in your neighborhood? Would
you let your child marry an immigrant?
4
What Is a Scale?
The assignment...
...of objects...
...to numbers...
Are you willing to permit immigrants to live in
your country? Are you willing to permit
immigrants to live in your neighborhood? Would
you let your child marry an immigrant?
5
What Is a Scale?
The assignment...
...of objects...
...to numbers...
Are you willing to permit immigrants to live in
your country? Are you willing to permit
immigrants to live in your neighborhood? Would
you let your child marry an immigrant?
...according to a rule...
6
What Is a Scale?
The assignment...
...of objects...
...to numbers...
Are you willing to permit immigrants to live in
your country? Are you willing to permit
immigrants to live in your neighborhood? Would
you let your child marry an immigrant?
Cumulative
...according to a rule...
7
Scale Versus Response Scale
Scale
Response Scale
Is used to collect the response for an item
Results from a process
8
Scale Versus Response Scale
Scale
Response Scale
9
Scale versus Response Scale
Scale
Response Scale
Results from a process
10
Scale Versus Response Scale
Scale
Response Scale
Is used to collect the response for an item.
Results from a process.
Each item on scale has a scale value.
11
Scale Versus Response Scale
Scale
Response Scale
Is used to collect the response for an item.
Results from a process.
Each item on scale has a scale value.
Item not associated with a scale value.
12
Scale Versus Response Scale
Scale
Response Scale
Is used to collect the response for an item.
Results from a process.
Each item on scale has a scale value.
Item not associated with a scale value.
Refers to a set of items.
13
Scale Versus Response Scale
Scale
Response Scale
Is used to collect the response for an item.
Results from a process.
Each item on scale has a scale value.
Item not associated with a scale value.
Refers to a set of items.
Used for a single item.
14
Purposes of Scaling

• Is the construct or concept a single dimensional
  one?
• What dimensions underlie some ratings?
• For assigning values to responses.

Hypothesis testing
Exploration
Scoring
15
Dimensionality
One-dimensional constructs
16
Dimensionality
One-dimensional constructs
Height
17
Dimensionality
One-dimensional constructs
Height
Taller
Shorter
18
Dimensionality
One-dimensional constructs
Height
Taller
Shorter
Thirst
19
Dimensionality
One-dimensional constructs
Height
Taller
Shorter
Thirst
More thirsty
Less thirsty
20
Dimensionality
One-dimensional constructs
Height
Taller
Shorter
Thirst
More thirsty
Less thirsty
Self-esteem
21
Dimensionality
One-dimensional constructs
Height
Taller
Shorter
Thirst
More thirsty
Less thirsty
Self-esteem
More
Less
22
Dimensionality
Two-dimensional construct
23
Dimensionality
Two-dimensional Construct
Academic ability
24
Dimensionality
Two-dimensional construct
Academic ability
Verbal
25
Dimensionality
Two-dimensional construct
Academic ability
Verbal
Better
Worse
26
Dimensionality
Quantitative
Two-dimensional construct
Academic ability
Verbal
Better
Worse
27
Dimensionality
Quantitative
Two-dimensional construct
Better
Academic ability
Verbal
Better
Worse
Worse
28
Dimensionality
Three-dimensional construct
29
Dimensionality
Three-dimensional construct
30
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
The meaning of a term can be described in 3
dimensions activity, evaluation, and potency
31
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Evaluation
32
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Evaluation
Unfavorable
Favorable
33
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Potency
Evaluation
Unfavorable
Favorable
34
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Potency
Powerful
Evaluation
Powerless
Unfavorable
Favorable
35
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Activity
Potency
Powerful
Evaluation
Powerless
Unfavorable
Favorable
36
Dimensionality
Semantic differential (meaning)
Three-dimensional construct
Active
Activity
Potency
Powerful
Evaluation
Passive
Powerless
Unfavorable
Favorable
37
Types of Scales

• Easier to understand
• Useful for understanding multidimensional
• Many concepts may be one-dimensional
• Reflect multiple properties simultaneously

One-dimensional
Multidimensional
38
Direct Estimation Methods

• Designed to have the subject give a direct
  quantitative estimate of the magnitude of an
  attribute
• Generally these are easy to fill out, with the
  subject responding along a scale from strongly
  agree to strongly disagree
• There are variations of this format

39
Types of Direct Estimation Methods

• Visual analogue scales uses a line of fixed
  length (100mm normally), with anchors on each end
  (no pain, extreme pain). Respondents place an X
  along the line. The measure is the length of the
  line from the end point to the X.
• Easy to use, useful in measuring change, however
  patients might not find it as easy to use.

40

• The precision (millimeters) is misleading since
  the real accuracy is not this good
• May be an age affect--this has been found to be
  more difficult for geriatric patients to use

41

• Adjectival Scales uses adjectives, can be either
  discrete or continuous
• Example What is the patients ability to
  understand instructions?
• much below average
• Below average
• average
• above average
• much above average

42
Likert Scaling Rater expresses his level of
agreement with a statement
Start with a large set of items that you think
all reflect the same construct.
43
Likert Scaling
A group of judges rates each item on a scale
where
1strongly disagree 2disagree 3undecided 4agree
5strongly agree
44
Deciding Which Items to Select

• Throw out items that have a low correlation with
  the total (summed) score across all items.
• For each item, get the average rating for the top
  quarter of judges and the bottom quarter.
• Do t-test higher t-values mean the items are
  better discriminators, so keep them.

Item-total correlations
Internal consistency
45
How Does Respondent Use?

• Gives a 1-to-5 (or sometimes a 0-to-4) rating for
  each item on the scale.
• Sum these across all items.
• This is why this is sometimes called a summated
  scale.
• dont forget to reverse any reversal items.

46
Semantic Differential Scale

• A characteristic is defined using a series of
  bipolar scales

Father Good________________________Bad Cruel______
__________________Kind Slow_______________________
_Fast Valuable_________________________Worthless T
ense_________________________Relaxed Strong_______
________________Weak
47
Criticism of Direct Estimation Methods

• Easy to use and design
• Easily understood by subjects (for the most part)
• Bias is more likely because the respondent can
  more easily see the obviously correct response
• Halo effect respondent rapidly rates all items
  similarly without looking at individual items

48

• The halo effect can be decreased by changing the
  order of ratings
• Another problem related to bias is that
  respondents often do not select the extreme
  categories, thereby reducing the precision of the
  measure

49
Comparative Methods of Scaling

• These methods were developed to circumvent some
  of the problems of the direct estimation methods.
  Most of these methods provide an interval scale
• Three methods most commonly seen
• Thurstone scaling
• Paired comparison technique
• Guttman scaling

50
Thurstone Scaling
51
Equal-Appearing Intervals
Start with a large set of items that you think
all reflect the same construct.
52
Equal-Appearing Intervals
Many judges individually sort these into 11 piles
where
1 least favorable to the concept
11 most favorable to the concept
1
2
3
4
5
6
7
8
9
10
11
53
Equal-Appearing Intervals
For each item, plot the distribution of pile
numbers.
54
Equal-Appearing Intervals
For each item, plot the distribution of pile
numbers.
55
Equal-Appearing Intervals
For each item, plot the distribution of pile
numbers.
Get the median.
56
Equal-Appearing Intervals
For each item, plot the distribution of pile
numbers.
and interquartile range.
Get the median.
57
Equal-Appearing Intervals
Choose those items that are at equal intervals
(from 1 to 11) and that have the smallest
interquartile range.
1
2
3
4
5
6
7
8
9
10
11
58
How Does Respondent Use Thurstone Scales?

• Respondent checks agree or disagree for each
  item.
• The respondents total score is the average scale
  value of all the items agreed to.

59
Paired Comparison Technique
60

• Uses a similar approach to Thurstone scaling, but
  is calibrated differently
• In this method, judges compare each item, one at
  a time, to each of the remaining items
• In each comparison they determine which of the
  two items has more of the property under study

61

• The percentage of times that each items was
  chosen over the other item is placed in a table
  (see table 4.1, p. 41)
• The normal curve is then used to convert these
  values to z-values
• For example, since 40 of the distribution in the
  normal curve falls to the left of -0.26, the
  value .40 is replaced by -0.26 (see p. 42)

62
Guttman Scaling
63
Guttman Scaling

• Also known as
• Scalogram analysis
• Cumulative scaling
• Purpose
• Establish a one-dimensional continuum
• Perfectly predict item responses from total score
• Seldom perfect in practice
• Most useful for developmental tests

64
Guttman Scaling
Start with a large set of items that you think
all reflect the same construct.
65
Guttman Scaling
Have many judges rate each statement with respect
to whether it is favorable with respect to the
construct -- a yes/no rating.
Rating Sheet
1 Manage time effectively
2 Manage resources effectively.
3 Scan a multitude of information and decide what
is important.
Yes No
4 Decide how to manage multiple tasks.
Yes No
5 Organize the work when directions are not
specific.
Yes No
Yes No
3 Scan a multitude of information and decide what
is important.
1 Manage time effectively
Yes No
Yes No
66
Guttman Scaling
Lay out a table
67
Guttman Scaling
Lay out a table
Item Item Item Item Item Item Respondent 2 7 5
3 8 ... 7 Y Y Y Y Y Y 15 Y Y Y -- Y -- 3 Y Y Y
Y -- -- 29 Y Y Y Y -- -- 19 Y Y Y -- -- -- 32 Y
Y -- Y -- -- 41 Y Y -- -- -- -- 6 Y Y -- -- --
-- 14 Y -- -- Y -- -- 33 -- -- -- -- -- --
68
Guttman Scaling
When sorted by row and column it will show
whether there is a cumulative scale.
Lay out a table
Item Item Item Item Item Item Respondent 2 7 5
3 8 ... 7 Y Y Y Y Y Y 15 Y Y Y -- Y -- 3 Y Y Y
Y -- -- 29 Y Y Y Y -- -- 19 Y Y Y -- -- -- 32 Y
Y -- Y -- -- 41 Y Y -- -- -- -- 6 Y Y -- -- --
-- 14 Y -- -- Y -- -- 33 -- -- -- -- -- --
69
Guttman Scaling
When sorted by row and column it will show
whether there is a cumulative scale.
Lay out a table
Item Item Item Item Item Item Respondent 2 7 5
3 8 ... 7 Y Y Y Y Y Y 15 Y Y Y -- Y -- 3 Y Y Y
Y -- -- 29 Y Y Y Y -- -- 19 Y Y Y -- -- -- 32 Y
Y -- Y -- -- 41 Y Y -- -- -- -- 6 Y Y -- -- --
-- 14 Y -- -- Y -- -- 33 -- -- -- -- -- --
Note the exceptions.
70
Guttman Scaling

• There are several statistical techniques for
  examining the table to find a cumulative scale.
• Because there is seldom a perfectly cumulative
  scale, we usually have to test how good it is.
• These statistics also estimate a scale score
  value for each item.

71
How Does Respondent Use?

• For selected items (the final scale), they simply
  answer whether they agree or not (yes/no).
• Since each item has a scale value, we can add
  these values.
• Total score is the sum of the scale items
  respondent agreed to.

72

• Guttman scales are evaluated in terms of
• The coefficient of reproducibility--the degree to
  which a persons score is a predictor of his
  response pattern (should be higher than 0.90)
• The coefficient of scalability whether the scale
  is really unidimensional and cumulative (should
  be at least 0.60).

73
Critique of Comparative Methods

• Require more development time than do the direct
  scaling methods
• This is worth the effort if we need to have
  interval responses
• The Guttman scale has only ordinal properties and
  is more difficult to obtain