Self-diagnosis,%20Scaffolding%20and%20Transfer:%20A%20tale%20of%20two%20Problems

1
Self-diagnosis, Scaffolding and Transfer A tale
of two Problems

• A. Mason1, E. Cohen2, C. Singh1 and E.
  Yerushalmi2
• 1University of Pittsburgh
• 2Weizmann Institute of Science, Rehovot, Israel
• PERC 2009

2
Research Design
Goal recitation sections deliberately prompted
to self-diagnose, given varying levels of support
Who will perform the best?
1st stage Students attempt a quiz problem, 2nd stage Intervention-Self-diagnosis with alternative supports 1st stage Students attempt a quiz problem, 2nd stage Intervention-Self-diagnosis with alternative supports 1st stage Students attempt a quiz problem, 2nd stage Intervention-Self-diagnosis with alternative supports 1st stage Students attempt a quiz problem, 2nd stage Intervention-Self-diagnosis with alternative supports
Text note Book (D) Sample Solution (C) TA outline diagnosis Rubric (B) Control-no SD TA discuss solution, (A)
3rd stage Post Solving paired quiz problem 3rd stage Post Solving paired quiz problem 3rd stage Post Solving paired quiz problem 3rd stage Post Solving paired quiz problem
24 students 28 students 31 students 100 students
First-semester algebra-based introductory physics 1 instructor, 2 TAs First-semester algebra-based introductory physics 1 instructor, 2 TAs First-semester algebra-based introductory physics 1 instructor, 2 TAs First-semester algebra-based introductory physics 1 instructor, 2 TAs
3
Research Rationale
1) What are students able to diagnose if
deliberately prompted to self-diagnose given
alternate supports? 2) What is the effect of SD
on consecutive problem solving? Assumption
high-performers are the ones which learn from
their mistakes. If we can get low-performers to
learn from their mistakes, 3) Do we reduce the
gap between low- and high-performers?
4
1st study 2nd study
Non-Conventional problem, Paired midterm problem given a few days after self-diagnosis Conventional problem, Paired midterm problem given a month after self-diagnosis
Quiz 7 and its Post (midterm III)
Quiz 6 and its Post (midterm II)
Boy jumping on skateboard going over a inclined
plane what height will he reach? (i.e. solve for
max height)
Girl on rollercoaster going over a circular bump
at peak of bump, how does weight change on a
scale?
Both pairs exhibit similar general procedure
(principles/intermediate variables), different
surface features
5
Analysis Tools-Analysis Rubric
I-quiz (grade of quiz solution given by researcher) S-quiz (grade of quiz solution as diagnosed by student) I-SD (researchers grade of the diagnosis done by the student)
Invoking Appropriate principles EC, MC, Newton second law
Invoking Justification Justify EC and MC
Invoking Compactness
Applying EC, MC, Newton second law
Desc-ription Drawing Knowns FBD, masses velocities before and after collision, velocities and heights or Ek and Ep before and after going up an incline
plan Target intermediate variables v a h
checking Units, limit case
Total Scores (Phy. Score, Pres. Score) Total Scores (Phy. Score, Pres. Score) Total Scores (Phy. Score, Pres. Score) Total Scores (Phy. Score, Pres. Score)
Generic
Specific
Valid focus on students diagnostic
ability Versatile generic/specific Reliable
Inter-rater reliability gt 80
Physics
Presenting reasoning
6
Research questions and expectations
Research Questions Inter group comparison Given alternative supports, how will external support affect self-diagnosis? how will it affect consecutive problem-solving Intra group comparison How does the self-diagnosis affect the gap between low and high achieving students? Do low achievers succeed in diagnosing themselves? Do they perform better in consecutive problem-solving?
Expectations More external support better self-diagnosis better performance on consecutive problem-solving The self-diagnosis task will be successful in reducing the gap between low and high achieving students
Reservations If self-diagnosis is NOT MEANINGFUL Then between-group differences in self-diagnosis performance wont be reflected in consecutive problem-solving The gap between low and high achievers will remain the same in consecutive problem-solving
7
What is a MEANINGFUL self-diagnosis?
THEORY (Chi, 2000) Students learn from solved
examples by providing self-explanations. The
student is expected
a) to compare and realize significant
differences between the two solutions b) to
acknowledge conflicts between their mental model
and the sample solution, leading to self-repair
of a flawed mental model.
8
1. Meaningful intervention Students scoring
on the diagnosis of their mistakes is independent
of their prior knowledge, and involves
self-repairing their mental modelOnly in this
case do we expect better self-diagnosis to lead
to better performance on consecutive
problem-solving, thus reducing the gap between
low and high achievers. VS.2.
Superficial intervention Students scoring on
the diagnosis of their mistakes is independent of
their prior knowledge, yet students do not
involve self-repairing their mental model3.
Weak intervention Students scoring on the
diagnosis of their mistakes depends on their
prior knowledge.
How does a NON-MEANINGFUL self diagnosis differ
from a MEANINGFUL one?
9
Expectations - intra group comparison
Correlation Control Intervention Intervention Intervention
Correlation Control Meaningful (Supports weak students in improving their SD performance. Involves meaningful learning leading to transfer.) Superficial (Supports weak students in improving their SD performance, However, does not involve meaningful learning leading to transfer.) Weak (Does not support weak students to improve their SD performance)
Pre vs. SD N/A N/S N/S Positive
SD vs. Post N/A Positive N/S Positive
Pre vs. Post Positive N/S Positive Positive
10
Inter group findings
Intervention Intervention Intervention
Outline Rubric (B) Sample solution (C) Minimal guidance (D)
1st study (non conventional problem) SD mean (Std. Err) 0.73 (0.05) 0.57 (0.05) 0.24 (0.06)
1st study (non conventional problem) P value BgtCgtD (ANCOVA based on quiz physics) BgtCgtD (ANCOVA based on quiz physics) BgtCgtD (ANCOVA based on quiz physics)
1st study (non conventional problem) Post mean (Std. Err) 0.53 (0.05) 0.33 (0.05) 0.47 (0.06)
1st study (non conventional problem) P value C differs from B and D (ANCOVA based on quiz physics) C differs from B and D (ANCOVA based on quiz physics) C differs from B and D (ANCOVA based on quiz physics)
2nd study (conventional problem) SD mean (Std. Err) 0.56 (0.06) 0.62 (0.06) 0.61 (0.05
2nd study (conventional problem) P value No difference between groups (ANCOVA based on quiz physics) No difference between groups (ANCOVA based on quiz physics) No difference between groups (ANCOVA based on quiz physics)
2nd study (conventional problem) post mean (Std. Err) 0.66 (0.04) 0.72 (0.06) 0.76 (0.04)
2nd study (conventional problem) P value No difference between groups (ANCOVA based on quiz physics) No difference between groups (ANCOVA based on quiz physics) No difference between groups (ANCOVA based on quiz physics)
11
The inter group findings suggest

• How do alternative external supports affect
  self-diagnosis?
• External support makes a difference in a non
  conventional problem, but not in a conventional
  one.
• e.g. Text and notes are enough for group (D)
  to perform SD in quiz 7, but not
• in quiz 6, where to self-diagnose students
  needed sample solution and rubric.
• How do alternative external supports affect
  consecutive problem-solving?
• It seems it doesnt
• e.g A) SD average grade for group B in quiz 6
  was much better than that for
• the other groups, but their post average was
  comparable to group D.
• B) SD average grade for group D in quiz 7 was
  the same as group C, and
• slightly better that that for group B (not
  significantly). D got the highest
• average in their post, although this average
  was comparable to groups B and C.

12
Intra group findings, 2nd study group D
Control A, A intervention intervention intervention
Control A, A Weak Superficial Meaningful
Correlation Pre-SD N/A N/S
Correlation SD-Post N/A Positive corr0.53, p valuelt0.05)
Correlation Pre-Post Positive 0.44/0.35 plt0.05 N/S
13
Intra group findings, 1st study all groups, 2nd
study all groups but D
Control A, A intervention intervention intervention
Control A, A Weak Superficial Meaningful
Correlation Pre-SD N/A N/S N/S
Correlation SD-Post N/A N/S
Correlation Pre-Post Positive 0.44/0.35 plt0.05 N/S
?
14
The intra group findings suggest
Group D, quiz 7 All other attempts, all groups
D performs a meaningful diagnosis Pre/Post correlation is not significant (midterm performance is independent of prior knowledge), while it is positive for control - SD/Post correlation is positive (midterm performance depends on SD performance) PUZZLE no match to expectations! - On one hand, the intervention seems to be meaningful (Pre-post positive for control none for intervention groups) - On the other hand, it seems superficial, as for most groups there is no correlation SD-Post.
15
Possible resolution for puzzle group D
Self diagnosing with minimal help (D, 2nd study)

Meaningful SD
Meaningful SD
D had to struggle to find a related sample
solution in the text, which they could in the 2nd
study (conventional) ? meaningful diagnosis
16
Possible resolution for puzzle groups B and C
Self diagnosing with maximal help (B,C)
?
Meaningful SD
Superficial SD ???
It is hard to differentiate between meaningful
and superficial diagnosis as the sample solution
allows to diagnose without struggling
17
Possible resolution for puzzleWhat is unique for
group D?

• Meaningful self diagnosis involves Chi
• Stage a) comparing and realizing significant
  differences
• between the two solutions
• Stage b) acknowledging conflicts between a flawed
  mental
• model and the sample solution
• Student rarely make explicit remarks reflecting
  stage b,
• thus, the question is whether we can conclude
  from remarks
• reflecting stage a - difference between the two
  solutions -
• that a meaningful self diagnosis took place,
• We claim that we can do so only for students who
  received
• minimal support for SD.

18
Possible resolution for puzzleWhat is unique for
group D?

• Students in groups B and C could compare their
  solutions
• to the sample solution provided and with minimal
  cognitive
• engagement state "I did not do this equation".
• Yet, in group D, students who were able to
  diagnose must have
• been cognitively engaged as they had to search on
  their own
• for a solution related to the problem they were
  trying to
• Diagnose.
• Therefore, for group D the grade indeed indicates
  how
• meaningful the diagnosis is.