Some (Simplified) Steps for Creating a Personality Questionnaire

1
Some (Simplified) Steps for Creating a
Personality Questionnaire

• Generate an item pool
• Administer the items to a sample of people
• Assess the uni-dimensionality of the item pool
• Assess the reliability of the measure
• Assess the validity of the measure

2
Generate an Item Pool

• Sample from the universe of possible items
• Rational/theoretical approach
• Observation (clinical, narrative, interviews,
  descriptions of others)

3
Administration

• Administer the questionnaire to a large sample of
  individuals.
• Can use full item set
• Can randomly sample items from full set

4
Dimensionality

• We want to ensure that there is only one major
  latent variable that the items have in common.
• The statistical tools that we use tend to assume
  uni-dimensionality.
• Multi-dimensional constructs are treated
  separately.
• Principal components analysis is sometimes used
  to determine whether one major variable underlies
  the item responses.

5
(No Transcript)
6
Choose items to factor
Choose Varimax under rotation options
Select Scree plot option in Extraction options
7
(No Transcript)
8
Reliability

• Reliability the extent to which measurements are
  free of random errors.
• Random error nonsystematic mistakes in
  measurement
• misreading a questionnaire item
• observer looks away when coding behavior
• nonsystematic misinterpretations of a behavior

9
Reliability

• What are the implications of random measurement
  errors for the quality of our measurements?

10
Reliability

• O T E S
• O a measured score (e.g., performance on an
  exam)
• T true score (e.g., the value we want)
• E random error
• S systematic error
• O T E
• (well ignore S for now, but well return to it
  later)

11
Reliability

• O T E
• The error becomes a part of what were measuring
• Once weve taken a measurement, we have an
  equation with two unknowns. We cant separate
  the relative contribution of T and E.
• 10 T E

12
Reliability Do random errors accumulate?

• Question If we sum or average multiple
  observations, will random errors accumulate?

13
Reliability Do random errors accumulate?

• Answer No. If E is truly random, we are just as
  likely to overestimate T as we are to
  underestimate T.

14
Reliability Do random errors accumulate?
Note The average of the seven Os is equal to T
15
Reliability Implications

• These demonstrations suggest that one important
  way to help eliminate the influence of random
  errors of measurement is to use multiple
  measurements.
• operationally define latent variables via
  multiple indicators
• use more than one observer when quantifying
  behaviors

16
Reliability Estimating reliability

• Question How can we estimate the reliability of
  our measurements?
• Answer Two common ways
• (a) test-retest reliability
• (b) internal consistency reliability

17
Reliability Estimating reliability

• Test-retest reliability Reliability assessed by
  measuring something at least twice at different
  time points.
• The logic is as follows If the errors of
  measurement are truly random, then the same
  errors are unlikely to be made more than once.
  Thus, to the degree that two measurements of the
  same thing agree, it is unlikely that those
  measurements contain a large proportion of random
  error.

18
Reliability Estimating reliability

• You didnt know it at the time, but when we
  conducted the subliminal recorded experiment, we
  assessed the test-retest reliability of a 6-item
  measure of self-esteem.
• The test-retest correlation was approximately
  .92, suggesting that there was very little random
  error present in our measurements.
• The test-retest correlation is an estimate of the
  proportion of true score variance present in a
  set of measurements.
• About 84 of the variation is true score
  variation the remaining 16 is error.

19
(No Transcript)
20
Reliability Estimating reliability

• Internal consistency Reliability assessed by
  measuring something at least twice within the
  same broad slice of time.
• Split-half based on an arbitrary split (e.g,
  comparing odd and even, first half and second
  half)
• Cronbachs alpha (?) based on the average of all
  possible split-halves

21
Alpha reliability

• Click on scale in the analyze menu
• Choose reliability analysis

22
(No Transcript)
23
Reliability Final notes

• An important implication As you increase the
  number of indicators, the amount of random error
  in the averaged measurement decreases.
• An important note Common indices of reliability
  range from 0 to 1 higher numbers indicate better
  reliability (i.e., less random error).