The Equivalence of Internet and Paperandpencil Personality Assessments

1
The Equivalence of Internet and Paper-and-pencil
Personality Assessments

• Chuah Siang Chee
• Fritz Drasgow
• Brent Roberts

Society for Industrial and Organizational
Psychology, Toronto, April 12-April 14, 2002
2
Internet Assessment

• Access from more locations and to more people
• Convenience for the user and test administrator
• User access from Internet enabled computers
• Electronic data entry
• Relative speed and low cost for large test
  administrations

3
Database Driven Website
4
Are Internet Assessments Equivalent to Other
Approaches?

• Measurement Equivalence
• Mean-Level
• Differential Item Functioning (DIF)
• Examinees of equal ability but from separate
  populations differ in their probability of giving
  a particular answer.

5
Previous research on mean-level differences
across modality

• Mean-level
• Social Desirability DistortionsMeta-analysis of
  non-cognitive tests across mediumsRichman,
  Kiesler, Weisband and Drasgow (1999)
• AnonymityLower Social Anxiety and Social
  Desirability with InternetJohnson (1999)
• Pasveer Ellard (1998)Self-Trust No
  differences
• Davis (1999)Ruminative Responses No differences

6
Problems with previous research on mean levels

• Comparison groups were not equivalent.
• Different recruiting techniques for groups
• Different samples sizes
• Unknown internet structure to assessment

7
Previous research on DIF

• No published research with DIF analysis

8
The Present Study

• Examined mean-level differences on the Big Five
  across testing medium
• Equivalent groups across modalities
• Paper-and-PencilTraditional administration under
  experimenter supervision.
• Computer LaboratoryGroup administration of WWW
  questionnaires in laboratory under experimenter
  supervision.
• InternetE-mail notification of WWW
  questionnaire. Administered over WWW for 1 week.

9
The Present Study

• Random Assignment to Conditions No
  Self-selection effects
• Identification of Subjects Unique Password
  security/integrity data Academic course
  credit Industry aberrant responses, linking
  information
• Multiple-access for Internet Questionnaire No
  Duplicate Responses

10
DIF Analyses

• SIBTEST (Stout Roussos, 1995)SIB Statistic ß
  Mantel-Haenszel
• Iterlink DIF (Steve Stark)Lords Chi-Square

11
Big 5 Personality Scale

• Trait Descriptive Adjective (Goldberg)
• Neuroticism, Extroversion, Openness,
  Agreeableness, Conscientiousness
• 5 point Likert Scale(Strongly Disagree
  Strongly Agree)How well do the adjectives
  describe you?

12
Big 5 Personality Items
Reverse Scored Items
Reverse Scored Items
13
Sample

• Undergraduates Psychology Students participating
  in the study for course credit

• Condition N
• Paper-and-pencil 266
• Computer Lab 222
• Internet 240

• Total
• 728

14
SIB Statistic p 15
Mantel-Haenszel p 16
Lords Chi-Square p 17
Expected of False Alarms

• 95 Confidence Interval
• 5 Comparisons (a X SD)
• a X SD 1.96 X (30 X .05 X .95)½ 2. 34 a X
  SD 1.96 X (123 X .05 X .95)½ 4.74
• Per Dimension 2.05 2.735 (0, 4.39)
• All 4 Dimensions 6.15 4.74 (1.41, 10.89)

18
Summary Analysis p 19
SIB Statistic Bonferroni Corr.
20
Mantel-Haenszel Bonferroni
21
Lords Chi-Square Bonferroni
22
Mean-Level Comparisons

• DIF analyses established that Personality Items
  are functioning equivalently across media
• Is there a shift in Trait Levels as a consequence
  of medium?
• mean-levels comparisons using ability/theta
  estimated by Bilog

23
Mean Trait Levels Effect Size
pEmotional Stability
Agreeableness
Extroversion
Conscientiousness

• Paper-and-Pencil Vs Both Computer
  AdministrationsEmotional Stability d .19

24
Conclusion

• Personality Tests are Equivalent
• Features of Internet Questionnaire Important

25
(No Transcript)