Measuring Learning Effectiveness:

1
Measuring Learning Effectiveness

• A New Look at No Significant Difference Findings
  in The Literature
• by
• Ernest H. Joy II, Ph.D.

2
Abstract

• Hundreds of media comparison studies claim no
  significant difference
• Much of this research is seriously flawed
• No significant difference findings can result
  simply due to faulty research designs
• Misinterpretation can easily occur
• Pedagogy is the key factor in learning
  effectiveness regardless of delivery medium

3
Background

• Media comparison studies are difficult to design
  due to many behavioral variables
• Nevertheless, the literature is replete with
  media comparison studies
• Current focus is to compare distributed learning
  systems, i.e., ALN to traditional classroom
  learning

4
The Great DebateDo Media Alone Influence
Learning Effectiveness? Clark vs Kozma

• Media are merely conveyances
• Instructional methods, not media, influence
  learning effectiveness
• Predicated on widely accepted learning theory
  (Gagne, Briggs, and Wager)

• Media methods are interconnected
• technology interacts with learners who are
  similarly engaged
• learning from media is a complementary process
  between the learner and the media

5
Pedagogy is Key to Learning

• Events necessary for learning comprise both
  internal, or cognitive information processing
  such as temporary storage, encoding, and
  retrieval and external events such as gaining
  attention, drill and practice, and feedback .
  (Gagne, Briggs, and Wager, 1992.)
• Instructional method is any way to shape
  information that activates, supplants, or
  compensates for the cognitive processes necessary
  for learning to occur. (Saloman cited in Clark,
  1994.)

6
Clarks Replaceability Challenge

• As long as there is another medium capable of
  conveying similar methods of instruction
  resulting in similar learning outcomes, the cause
  must be something other than the media itself.
• Until such time as a single medium becomes
  necessary for learning, media will continue to
  serve only as a conveyor of processes which
  contribute to learning.

7
Common Flaws in Media
Comparison StudiesA. Specification
Error

• Lack of control group
• Unspecified size of control group
• Non-random selection/assignment of participants
• Assessment instrument bias

8
Common Flaws in Media
Comparison StudiesB. Failure to
control for key variables

• Instructional methods
• Ability
• Prior knowledge
• Learning style
• Teacher effects
• Time on task
• Familiarity with media

9
Consequences
It is less likely that significant differences
will be found since there is no single difference
affecting only the treatment group. On the
other hand, when a significant difference is
found, that difference is most likely the result
of some uncontrolled treatment (s) affecting one
group more so than the other.
10
Conclusions

• Exercise caution in interpreting results of media
  comparison studies
• Improvement in design of media comparison studies
  is problematic.
• Pedagogy is widely accepted in the literature as
  key to learning effectiveness regardless of the
  delivery medium.

11
Recommendations

• Do not rely on media comparison research alone to
  determine the effectiveness of an instructional
  delivery system
• Focus on pedagogy to improve learning
  effectiveness
• Shift the research inquiry How best to employ
  the most appropriate delivery medium?