In-vivo research on learning

1
In-vivo research on learning

• Charles Perfetti
• PSLC Summer School 2009

2
In-vivo experiments

• In Vitro
• In Vivo

3
Features of in-vivo experiments in learning

• On-Line course?
• An Intelligent Tutoring System?
• A real class real students an intervention that
  counts.

4
The value of in-vivo experiments in learning

• Noisy, uncontrolled environment
• Content of intervention is validated by course
  goals
• So Built in generalization to classroom learning

5
Problems faced by an in-vivo researcher

• Noisy, uncontrolled environment
• As for your experiment
• Students have other things to do
• Instructors have other things to do

6
Examples of in-vivo studies

• Algebra, Physics, Chemistry, Geometry, French,
  Chinese,English
• Some with computer tutors in major role
• ITS
• Practice tutors
• Some without tutors or tutors in minor role

7
Pre-requisites for an in-vivo experiment

• Knowledge components analysis
• Mapping of KCA to a learning or instructional
  hypothesis
• Theory based
• Empirical precedent
• Mapping instructional hypothesis to specific
  intervention

8
Knowledge Components vs. curriculum topic
Single Topic (Area) as unit 12 separate KCs as
units
Enabled by Data Shop
9
Mapping a KCA onto an instructional hypothesis

• The case of Chinese characters

10
Mapping an instructional hypothesis to an
instructional intervention

• Learning event space

11
Instructional Event Space
Learning Events
Instructional Events
Assessment Events
Explicit or implicit Focus on Valid Features Make
Knowledge Accessible Promote Active
Processing Schedule events effectively Coordinate
multiple events
Performance
12
1. Learning meanings of Chinese characters

• Knowledge Components Analysis
• 2 (2) Knowledge Components
• the character as a whole (plus its meaning)
• the radical that is part of the character (plus
  its meaning)
• Two approaches based on this analysis
• (1) Dunlap, Liu, Perfetti (2) Pavlik
• Two different Instructional Events manipulations
• Illustrate 1 here Feature focus

13
Instructional Event Space
Associate character form with meaning
Assessment Events
Whole Character means x
Default (typical) Instructional event
Performance
14
Instructional Event Space
Associate radical with x and whole character
with x
Part of character means x
Assessment Events
Highlighted radical sun/day
Performance
Dunlap et al Instructional event manipulation
semantic radical instruction
15
Learning English Spelling (Background knowledge
and feature focusing themes)
Dunlap, Juffs, Friedline, Perfetti
16
KC analysis of English spelling

• phonologyorthography
• /breit/--brate
• /hiyl/--heel
• /hiyl/--heal
• So phonology-semantics-orthography

17
Feature focusing interventions

• 130 students in levels 3 4, 5
• Interventions
• Pure feature focus form only
  (pronunciation-spelling pairs)
• Meaning mediated focus form meaning
  (pronunciation-meaning-spelling triads)
• 7 sessions, 30 minutes per session over 7 weeks

18
Dunlap, Juffs, Friedline, Perfetti
19
Learning Measures

• Across-session error rates (transfer to new
  items)
• Post-test tone judgments presented by tutor
• Two successive syllables heard. Are they same or
  different in tone? (transfer to different task)
• Nature of syllable pairs
• Tone same, segments different /duan/3
  /liang/3
• Same onset and rime, shi2 -- shi3
• Share rime only, e.g. dao2 kao3
• Share neither onset nor rime, e.g., duo2 --
  gong3.

20
Studies with major role for a computer tutor

• Formative evaluation. How can the tutor be
  improved?
• Summative evaluation. Is the tutor effective?
• Both of these apply to all instructional
  interventions, whether tutor based or not

21
Formative Evaluation Examples

• User interface testing
• Early, before the rest of the tutor is built
• Engage students and instructors
• Get detailed response from students viewing tutor
  with talk-aloud procedures
• Wizard of Oz
• Human (the Wizard) in the next room watches a
  copy of screen
• Responds when student presses Hint button or
  makes an error
• User interface evaluation
• Does the wizard have enough information?
• Can the wizard intervene early enough?
• Tutor tactics evaluation. What did the Wiz do
  when?

22
Formative Example 3 Snapshot critiques

• Procedure ITS log file
• Select student help events from log file
• Experts examine context leading up to the help
  message noting the help they would provide
• Examine match between help from experts and that
  from ITS.
• Compare with match between two experts.
• Modify ITS help messages according to reliable
  expert input.

23
Summative evaluations

• Question Is the tutor (or other instructional
  intervention) more effective than a control?
• Typical design
• Experimental group gets the instructional
  intervention (the tutor).
• Control group learns via the traditional or
  current practice method
• Pre post tests
• Data analysis
• Did the tutor group do better than the control?

24
Control conditions for in-vivo experiments

• Typical control conditions
• Existing classroom instruction
• Textbook exercise problems
• For cog tutors
• Another tutoring system
• Human tutoring
• A control intervention
• 2 plausible interventionswhich is more effective

25
Learning Assessments

• Immediate Learning
• Long-term retention
• Transfer
• Over content, form, testing situations
• Accelerated Future Learning
• New content learning measure

26
Instructional Event Space
Learning Events
Instructional Events
Assessment Events
Explicit or implicit Focus on Valid Features Make
Knowledge Accessible Promote Active
Processing Schedule events effectively Coordinate
multiple events

• Learning
• Long term retention
• Transfer
• Accelerated future learning

Performance
27
Transfer illustrated Liu, Wang, Perfetti
Chinese tone perception study

• In-vivo study
• Traditional classroom (not online)
• Materials from students textbook
• New materials each week for 8 weeks of term 1
• Term 2 continued this, and added novel syllables
  unfamiliar to the student
• 3 instructional conditions
• tone number pin yin, contour pin-yin contour
  only
• Hint system
• (CTAT) Tutors presented materials in 3 different
  instructional interfaces, according to the 3
  conditions
• Data shop logged individual student data

28
Illustration of 2 conditions from Liu et al
shi
29
Data from Liu et al tone study
Learning curves week-by-week
30
Multiple kinds of transfer

• Liu et al shows 2 kinds of materials transfer
• Within term 1, learning sessions, each syllable
  to be learned was different but familiar. So
  transfer of learning to familiar items
• At second term, there were unfamiliar syllables.
  So transfer of learning to unfamiliar items. (Not
  so good.)

31
Example of acceleration of future learning (Min
Chi VanLehn)

• First probability, then physics. During
  probability only,
• Half students taught an explicit strategy
• Half not taught a strategy (normal instruction)

Score
Pre
Post
Probability Training
32
Creating assessments

• General strategy
• Guided by cognitive task analysis (pre-test as
  well) including learning goals and specific
  knowledge components
• Include some items from the pre-test
• Check for basic learning
• Some items similar to training items
• Measures near-transfer
• Some problems dissimilar to training problems
• Measures far-transfer

33
Mistakes to avoid in test design

• Tests that are
• Too difficult
• Too easy
• Too long
• Tests that
• Fail to represent instructed content
• Missing content over sampling from some content
• Depend too much on background knolwedge

Notice problems in test means
Notice variances
34
Interpreting test results as learning

• Post-test in relation to pre-test. 2 strategies
• ANOVA on
• gain scores
• First check pre-test equivalence
• Not recommended if pre-tests not equivalent
• Pre-test, post test as within-subjects variable
  (t-tests for non-independent samples)
• ANCOVA. Post-tests scores are dependent variable
  pre-test scores are co-variate

35
Plot learning results

• Bar graphs for instructional conditions
• Differences due to conditions
• Learning Curves
• Growth over time/instruction

36
Bar graphs (with error bars!)
37
Learning Curves
Error rate
Weekly sessions over 2 terms
38
Learning Curves
Error rate
Weekly sessions over 2 terms
PSLC summer school 2009
38
39
A final word on experiments

• In-vivo limitations
• The role of (in-vitro) laboratory studies

40
(No Transcript)
41
The end