Developing a Metric for Evaluating Discussion Boards

1
Developing a Metric for Evaluating Discussion
Boards

• Dr. Robin Kay
• University of Ontario Institute of Technology
• 2 November 2004

2
Overview

• Background
• Data Collected
• 12 Areas of Evaluation
• Sample Results
• Summary

3
Background

• Discussion board use has grown extensively
  (e.g., Cooper, 2001)
• Some say tool is revolutionary ( e.g., Hara et.
  al., 1998, Li, 2003)
• Others say our understanding of discussion boards
  is minimal(e.g., Blignaut Trollip, 2003)
• One problem The metric used to evaluate
  effectiveness of discussion boards

4
Previous Metrics

• Most studies examine one or two aspects of online
  discussion
• A few researchers have attempted more complete
  analyses (e.g., Hara et. al., 1998 Zhu, 1998)
• Metrics are rarely theory-driven

5
Metric Needs

• Comprehensive
• Theory-Based
• Consistent

6
Data Collected (1 of 3)
7
Data Collected (2 of 3)
8
Data Collected (3 of 3)

• Subject Pool
• secondary, higher education
• Purpose of Discussion Board
• debate, posting resources, solving problems
• Individual Differences
• gender, typing speed, access
• Type of Course
• Online Only vs. Mixed

9
12 Areas of Evaluation

• Social Learning
• Cognitive Processing
• Discussion Quality
• Initial Question
• Role of Educator
• Navigation

• Challenges
• Types of Users
• Attitudes Toward DB
• Reponses Time
• Time of Learning
• Performance

10
Social Learning
11
Cognitive Processing
12
Discussion Quality
13
Initial Question
14
Role of Educator
15
Navigation
16
Challenges for Participants
17
Types of Users
18
Attitudes Toward Discussion Board
19
Response Time
20
Time of Learning
21
Learning Performance
22
Sample Results
Sample
Overview

• Challenges
• Types of Users
• Attitudes Toward DB
• Response Time
• Time of Learning
• Performance

• Social Learning
• Cognitive Processing
• Discussion Quality
• Initial Question
• Role of Educator
• Navigation

23
Sample

• 45 secondary students (2 classes)
• 13-15 years old
• Introductory computer science course
• HTML and Programming
• Private school
• All boys
• HTML (24 days)
• Programming (36 days)

24
Overview of Results
25
Social Learning
26
Cognitive Processing
27
Discussion Quality
28
Effect of Initial Question
29
Role of Educator
30
Navigation
31
Challenges
32
Types of Users (5 messages)
33
Attitude Toward Discussion Board
34
Response Time
35
Time of Learning
36
Performance
37
Summary

• Previous research uses limited metric
• 12 areas covered offer a relatively rich analysis
  of discussion
• If we want to understand the use if discussion
  boards we need metric that is
• Comprehensive (details are everything)
• Theory-Based

38
Contact Information

• Robin Kay
• robin.kay_at_uoit.ca
• Website for Paper Presentation
• faculty.uoit.ca/kay/elearn2004/elearn2004.htm