Application of an automated content analysis process to multisource comments

1
Application of an automated content analysis
process to multisource comments
Carrie Christianson DeMay, Anna Chandonnet,
Colleen Rasinowich, and Kristofer J.
Fenlason Data Recognition Corporation
A paper presented in Improving the use and
usefulness of multisource comments. Presented at
the Twenty-First Annual Meeting of the Society
for Industrial and Organizational Psychology,
Dallas, TX. May 2006.
2
Introduction

• Prevalence of multisource comments
• Most programs include this option (Timmreck
  Bracken, 1995 Rose Walsh, 2004)
• Comments regarded as useful
• Targeted individual work behaviors (Antonioni,
  1996)
• Recipients may attend more to than scaled
  responses (Ferstl Bruskiewicz, 2000)
• Possible uses for multisource comments
• Find training needs by helping recipient move
  from identifying to acting (Kulesa Bishop,
  2006)
• Individual development planning (Burke Gaylord,
  2005)
• Leadership development
• Categorize comments into topic areas (Rose
  Farrell, 2002)
• Tool to revise instrument
• Comment themes used as profile

3
Comment content coding

• Typically part of employee surveys
• Rose and Walsh (2004) found fewer than 1 of
  multisource feedback program managers use content
  coding
• Types of coding
• Manual
• Text analysis
• SPSS Text Analysis for Surveys 1.5

4
(No Transcript)
5
(No Transcript)
6
Research questions

• 1 How do the results differ when a pre-defined
  coding scheme is manually input into SPSS Text
  Analysis for Surveys 1.5 and also used for coding
  by manual coders?

7
Research questions

• 2 How do the results differ when SPSS Text
  Analysis for Surveys 1.5 automatically develops
  the coding scheme to be used by the software and
  by manual coders?

8
Research questions

• 3 - What differences exist between a coding
  scheme developed by SPSS Text Analysis for
  Surveys 1.5 when a sample of multisource comments
  are coded and a pre-defined coding scheme
  designed specifically for coding multisource
  comments?

9
Research questions

• 4 - What are the differences in cost and time
  using SPSS Text Analysis for Surveys 1.5 versus a
  manual coding process to code comments.

10
Method

• Sample
• 570 direct report comments from 2004
• Existing comment categories (Pre-defined scheme)
• Categories defined by Rose Farrell (2002)
• Developed for multisource comments
• SPSS Text Analysis categories (Automated scheme)
• Comment categories created via software

11
Category schemes

• Pre-defined scheme Rose Farrell (2002)
• Ability to Work with Others/Interpersonal Skills
• Communication
• Customer Service
• Honesty/Integrity
• Initiative
• Job Knowledge
• Job Skills/Personal Characteristics
• Leadership/management skills
• Miscellaneous
• Organizational Commitment
• Productivity
• Staff Development

• Automated scheme SPSS Text Analysis
• Goals
• Process
• Management
• Job/Work
• Skills
• Teams
• The Organization
• Demeanor
• People Interactions
• General Performance

12
Method
13
(No Transcript)
14
Results
1 - How do the results differ when a pre-defined
coding scheme is manually input into SPSS Text
Analysis for Surveys 1.5 and also used for coding
by manual coders?
15
Differences in comment coding

• Manual coding example
• Name has a gift of Leadership. She listens
  to other persons ideas and considers all view
  points. She is a motivator. Name challenges
  her people and encourages new ideas to better
  both the employee and the company. Her
  management skills surpass all others.
• SPSS Text Analysis for Surveys 1.5 coding example
• Name has a gift of Leadership. She listens
  to other persons ideas and considers all view
  points. She is a motivator. Name challenges
  her people and encourages new ideas to better
  both the employee and the company. Her
  management skills surpass all others.
• Pre-defined scheme
• Interpersonal Skills, Communication, Customer
  Service, Honesty/Integrity, Initiative, Job
  Knowledge, Job Skills, Leadership, Organizational
  Commitment, Productivity, Staff Development

16
Results

• 2 - How do results differ when SPSS Text Analysis
  for Surveys 1.5 automatically develops the coding
  scheme to be used by the software and by manual
  coders?

17
Results
3 - What differences exist between a coding
scheme developed by SPSS Text Analysis for
Surveys 1.5 when a sample of multisource comments
are coded and a pre-defined coding scheme
designed specifically for coding multisource
comments?

• Pre-defined scheme Rose Farrell (2002)
• Ability to Work with Others/Interpersonal Skills
• Communication
• Customer Service
• Honesty/Integrity
• Initiative
• Job Knowledge
• Job Skills/Personal Characteristics
• Leadership/management skills
• Miscellaneous
• Organizational Commitment
• Productivity
• Staff Development

• Automated scheme SPSS Text Analysis
• Goals
• Process
• Management
• Job/Work
• Skills
• Teams
• The Organization
• Demeanor
• People Interactions
• General Performance

18
Results

• 4 - What are the differences in cost and time
  using SPSS Text Analysis for Surveys 1.5 versus a
  manual coding process to code comments.
• Cost to code 570 comments
• Manual coding
• 40 hours labor
• Text analysis
• Software cost 3250.00
• Less than 1 hour labor

19
Ad Hoc Research

• 30,000 comments through SPSS Text Analysis for
  Surveys 1.5
• 3 hours processing
• Handled quantity

20
Discussion

• Lessons learned
• Human versus computer
• Sarcasm not detected by software
• Valence not assigned by software
• Very basic concepts identified by software much
  more complex concepts identified by manual coders
• Updating SPSS Text Analysis dictionaries
• Include all synonyms and word derivations as this
  does not occur by default

21
Future research

• Explore SPSS Text Analysis for Surveys 1.5
  features
• Develop a leadership development training plan
  using multisource comments
• Compare the results to an-EOS developed
  leadership development training plan
• Replicate with another sample of comments from
  same study

22
References

• Antonioni, D. (1996). Designing an effective
  360-degree appraisal feedback process.
  Organizational Dynamics, Autumn, 24-38.
• Burke, K. A., Gaylord, T. W. (2005, April).
  Using 360 comments to direct and impact
  development plans. Presented at the 19th annual
  conference of the Society for Industrial and
  Organizational Psychology, Los Angeles, CA.
• Dalessio, A. T. (1998). Using multisource
  feedback for employee development and personnel
  decisions. In J. W. Smither (Ed.), Performance
  Appraisal State of the art in practice (pp.
  278-330). San Francisco, CA Jossey-Bass.
• Ferstl, K. L., Bruskiewicz, K. T. (2000,
  April). Self-other agreement and cognitive
  reactions for multirater feedback. Presented at
  the 15th annual conference of the Society for
  Industrial and Organizational Psychology, New
  Orleans, LA.
• Kulesa, P., and Bishop, R.J. (2006). What did
  they really mean? New and emerging methods for
  analyzing themes in open-ended comments. In A.
  I. Kraut (Ed.), Getting Action from
  Organizational Surveys New Concepts,
  Technologies, and Applications (pp. 239-263).
  San Francisco, CA Jossey-Bass.
• Rose, D. S., Farrell, T. (2002, April). The
  use and abuse of comments in 360-degree feedback.
  Paper presented at the 18th Annual Conference of
  the Society for Industrial and Organizational
  Psychology, Toronto.
• Rose, D. S., Walsh, A. B. (2004). Current
  trends in 360º feedback (Technical report 8251).
  Data Driven Decisions, Inc.
• Smither, J. W., Walker, A. G. (2004). Are the
  characteristics of narrative comments related to
  improvement in multirater feedback ratings over
  time? Journal of Applied Psychology, 89 (3),
  575-581.
• SPSS, Inc. (2005). SPSS Text Analysis for
  Surveys 1.5 Users Guide. Chicago, IL SPSS,
  Inc.
• Timmreck, C. W., Bracken, D. W. (1995, May).
  Upward feedback in the trenches Challenges and
  realities. Presented at the 10th Annual
  Conference of the Society for Industrial and
  Organizational Psychology, Orlando, FL.