Assessing the Impact of Technology on Learning Outcomes

1
Assessing the Impact of Technology on Learning
Outcomes

• Geoff Norman, PhD
• norman_at_mcmaster.ca

2

• If the new educational technologies are so
  incredibly
• Learner - centered
• Accessible
• Efficient
• Individualizable
• Active
• why is it so hard to show that students learn
  more from them and like them more

3

• The Cost - Effectiveness Equation
• Cost / electronic lecture hour
• 1992 Planalyzer (hematology) 50,000
• 2004 Temple/Hopkins 10,000
• ca. 2000 face to face 500-1000?
• Is electronic is 20-100 times as effective?

4
Agenda

• Researching Effectiveness of Instructional
  Technology
• Designing for the active ingredient
• Technology as compared to what?
• Measured with what?

5
Some Bad Design Strategies

• Compare Computer to Conventional
• or
• Conventional Computer to Conventional
• 1) What exactly is the conventional group
  getting?
• 2) What exactly is the computer adding?
• Any additional time additional learning

6

• Specify precisely what is added by the technology
• Then control for the technology
• At instruction
• At test
• i.e. Avoid media-comparative studies

7
An Example - Anatomy

• In learning 3-D materials (e.g. anatomy),
  presentation of dynamic images (virtual
  reality) should enhance anatomy learning
• Experimental -- multiple views
• Control -- key views
• All interventions and tests on the computer

8
Intervention
0
Key Views (2) KV
Multiple Views (36) MV
80
0
220
180
180
90
270
310
0
9
Test (50 items)
10
M.V.
KV
MV
11
SPECIFY THE ACTIVE INGREDIENTEffect of lo/hi
simulator on surgical skills (Anastakis, 1999
Matsumoto, 2002)

• Anastakis
• 6 procedures n 23
• Text only,
• Bench model,
• Cadaver
• Matsumoto
• Cystoscopy, ureteroscopy n 40
• Text only
• Lo fidelity (straws coffee cup) 20
• Hi fidelity (plastic ) 3700
• Test on hi-fi model

12
Anastakis
13
Matsumoto
14
More Design Problems

• Use long term followup (weeks or months)
• 4) How do you measure the dose?
• 5) Hawthorne effect
• The fact that theyre being studied makes
  intervention group do better
• 6) John Henry effect
• The fact that the other guys are being studied
  makes the control group work harder

15
Measurement Problems

• 1) Measurement based on satisfaction
• Computer wins against text (almost always)
• Computer loses against teacher (almost always)
• No relation between satisfaction and learning
  anyway (almost always)
• 2) Measurement based on performance on the
  computer
• Learning confounded with familiarity
• (NBME CBX)

16
Surgical simulator for anatomy(Harari, Rawn, et
al., 2004)

• Intervention Shoulder anatomy
• virtual reality simulator vs.
• Text
• Measurement
• Identification of structures
• Satisfaction

17
Hariri
18
CAI vs. Tutorial for Phobia Inst.(McDonough,
2002)

• Intervention
• Tutorial group vs. CAI
• Measurement
• Satisfaction rating
• Knowledge gain

19
McDonough
20
Testing on the TechnologyAuscultation of Heart
Sounds

• High-fidelity simulator (Harvey)
• (and high price --100,000)
• - plastic body
• - heart sounds distributed through chest
• - coordinated wrist pulse, JVP
• - chest moves (but no breath sounds)
• (Ewy et al, 1987 Issenberg et al., 1999)

21
Harvey as test (Issenberg, 1999)
22
Real Patients as test (Ewy et al., 1987)
23
SOLUTION

• Clear specification of the value added by
  e-learning
• Control for other aspects (e.g. familiarity with
  software)
• Use medium-neutral measurement
• Control time to avoid co-intervention

24

• Specification of value-added
• Make the experimental manipulation specific to
  this component (e.g. rotation of wrist for
  anatomy pulse and auscultation for Harvey)
• Control for medium, computer experience
• Do experimental manipulation within medium

25

• Use medium-neutral (or bias -) measurement
• Test in real world transfer
• e.g.real patients for heart sounds
• or
• Test in a medium that is biased against
  intervention
• e.g final exam on paper
• or
• Do both arms of study on same medium
• (everybody gets the computer)
• and manipulate the active ingredient
• e.g. brain anatomy, wrist anatomy

26
Conclusions

• A new medium is often not necessarily better than
  the old, and is not often better
• Active learning is a description of what the
  brain is doing, not what the mouse is doing
• Decent studies of new media are hard to do

27
Conclusions

• Virtual simulation has as much chance of
  replacing face-to-face interaction for helping
  people achieve conceptual understanding
• as
• In vitro fertilization has of replacing
  face-to-face intercourse for helping people
  achieve conception