Evaluating Online Discussions: Four Different Frames of Analysis

1
Evaluating Online Discussions Four Different
Frames of Analysis

• Katrina A. Meyer
• Assistant Professor of Educational Leadership
• University of North Dakota
• 2003 WCET Conference

2
Four Frames

• King and Kitchener Reflective Judgment Model
• Perry Intellectual and Ethical Development Model
• Garrison et al. Critical Thinking Model
• Bloom Taxonomy of Educational Objectives

3
Findings

• Of 278 total postings, 74 were coded by all four
  frames (26 dropped).
• Of 206 postings, 63 or 130 postings had unique
  combinations of the four frames.
• Of the over 1500 possible permutations of the
  four frames, only 130 combinations were found,
  and 31 of these only appeared twice.

4
Findings

• 33.4 of postings captured the request for and
  response from students of personal experiences
  and beliefs (KK3 and P7).
• 32.5 of postings were related to requesting and
  responding with analyses that were either
  exploring or integrating ideas (Bloom
  Garrison).

5
Findings

• The type of triggering question influences the
  level of response from students.
• Questions created to trigger personal stories did
  so and questions targeted to elicit information
  or higher-level analysis did so.

6
Findings

• 45.3 were at levels 5-7 (King and Kitchener)
• 100 were at levels 5-9 (Perry)
• 52.2 were at the two highest levels (Garrison)
• 54.3 were at levels 4-6 (Bloom)
• Appropriate for doctoral-level students in a
  doctoral-level class.

7
Findings

• Each frame has value.
• Each frame focuses attention on a particular
  aspect or quality of the student and his/her
  thinking
• There is no one best frame, or perhaps one frame
  might be better suited for a particular
  discussion or a particularly set of learning
  objectives.
• Need to develop multiple frames for analyzing
  online discussions intended to address different
  learning situations.

8
Findings

• Perry was the most difficult framework.
• Distinctions between levels were difficult to
  discern.
• This may preclude it being used on a more
  frequent basis.

9
Findings

• King and Kitchener was less difficult to apply.
• Both King and Kitchener and Perry are
  classification schemas for an individual.
• Both might work better if the discussion were set
  up to generate a particular reflection or if the
  post was sufficiently long to give a sense of the
  students reasoning.
• The level of a specific post (or even several
  postings) may or may not indicate the students
  predominant developmental stage.

10
Next Level of Analysis

• A stage may indicate the students primary level
  of response or
• A stage the group must pass through (i.e.,
  information) in an effort to develop higher-level
  analyses.
• Use frames to analyze the ebb and flow of online
  discussions as a group effort, rather than
  focusing on the individual postings as a
  reflection of the students level of thought.

11
Cautions

• Danger that a posting might become colored by the
  frame.
• Each frame focused the analysis at the same time
  it eliminated other perspectives.
• It is a lens that filters out AND focuses.

12
More Frames Needed

• Assessment of social presence the ability of
  students to contribute in such a way as to make
  their personalities come to life in their
  postings.
• Assessment of group functioning how online
  groups work together to develop an understanding
  of and solutions to a problem.