Assessment of Student Problem Solving Processes

1
Assessment of Student Problem Solving Processes

• Jennifer L. Docktor
• Ken Heller
• Physics Education Research Development Group
• http//groups.physics.umn.edu/physed

DUE-0715615
2
Problem Solving Measure

• Problem solving is an important part of learning
  physics.
• Unfortunately, there is no standard way to
  measure problem solving so that student progress
  can be assessed.
• The goal is to develop a robust instrument to
  assess students written solutions to physics
  problems, and obtain evidence for reliability,
  validity, and utility of scores.
• The instrument should be general
• not specific to instructor practices or
  techniques
• applicable to a range of problem topics and types

3
Reliability, Validity, Utility

• Reliability score agreement
• Validity evidence from multiple sources
• Content
• Response processes
• Internal external structure
• Generalizability
• Consequences of testing
• Utility - usefulness of scores

AERA, APA, NCME (1999). Standards for educational
and psychological testing. Washington, DC
American Educational Research Association. Messick
, S. (1995). Validity of psychological
assessment. American Psychologist, 50(9),
741-749.
4
Overview of Study

1. Drafting the instrument (rubric)
2. Preliminary tests with two raters (final exams
   and instructor solutions)
3. Training exercise with graduate students
4. Analysis of tests from an introductory mechanics
   course
5. Student problem-solving interviews (in progress)

5
What is problem solving?

• Problem solving is the process of moving toward
  a goal when the path to that goal is uncertain
  (Martinez, 1998, p. 605)
• What is a problem for one person might not be a
  problem for another person.
• Problem solving involves decision-making.
• If the steps to reach a solution are immediately
  known, this is an exercise for the solver.

Martinez, M. E. (1998). What is Problem Solving?
Phi Delta Kappan, 79, 605-609. Hayes, J.R.
(1989). The complete problem solver (2nd ed.).
Hillsdale, NJ Lawrence Erlbaum
Associates. Schoenfeld, A.H. (1985). Mathematical
problem solving. Orlando, FL Academic Press, Inc.
6
Problem Solving Process

• Organize problem information
• Introduce symbolic notation
• Identify key concepts

Understand / Describe the Problem

• Use concepts to relate target to known information

Devise a Plan

• appropriate math procedures

Carry Out the Plan
Look Back

• check answer

P?lya, G. (1957). How to solve it (2nd ed.).
Princeton, NJ Princeton University Press. Reif,
F. Heller, J.I. (1982). Knowledge structure and
problem solving in physics. Educational
Psychologist, 17(2), 102-127.
7
Problem Solver Characteristics

• Experienced solvers
• Hierarchical knowledge organization or chunks
• Low-detail overview / description of the problem
  before equations
• qualitative analysis
• Principle-based approaches
• Solve in symbols first
• Monitor progress, evaluate the solution

• Inexperienced solvers
• Knowledge disconnected
• Little representation (jump to equations)
• Inefficient approaches (formula-seeking
  solution pattern matching)
• Early number crunching
• Do not evaluate solution

Chi, M. T., Feltovich, P. J., Glaser, R.
(1980). Categorization and representation of
physics problems by experts and novices.
Cognitive Science, 5, 121-152. Larkin, J.,
McDermott, J., Simon, D.P., Simon, H.A. (1980).
Expert and novice performance in solving physics
problems. Science, 208(4450), 1335-1342.
8
Instrument at a glance (Rubric)
SCORE
5 4 3 2 1 0 NA (P) NA (S)





CATEGORY (based on literature)
Useful Description
Physics Approach
Specific Application
Math Procedures
Logical Progression

• Minimum number of categories that include
  relevant aspects of problem solving
• Minimum number of scores that give enough
  information to improve instruction

Want
9
Rubric Category Descriptions

• Useful Description
• organize information from the problem statement
  symbolically, visually, and/or in writing.
• Physics Approach
• select appropriate physics concepts and
  principles
• Specific Application of Physics
• apply physics approach to the specific conditions
  in problem
• Mathematical Procedures
• follow appropriate correct math
  rules/procedures
• Logical Progression
• (overall) solution progresses logically it is
  coherent, focused toward a goal, and consistent

10
Rubric Scores (in general)
5 4 3 2 1 0
Complete appro-priate Minor omissionor errors Parts missing and/or contain errors Most missing and/or contain errors All inappro-priate No evidence of category
NOT APPLICABLE (NA)
NA - Problem NA - Solver
Not necessary for this problem (i.e. visualization or physics principles given) Not necessary for this solver (i.e. able to solve without explicit statement)
11
(No Transcript)
12
Early Tests of the Rubric

• Preliminary testing (two raters)
• Distinguishes instructor student solutions
• Score agreement between two raters good
• Training Exercise (8 Graduate Students)
• Half scored a mechanics problem, half EM
• Scored 8 student solutions with the rubric,
  received example scores rationale for first 3,
  then re-scored 5 and scored 5 new solutions
• Answered survey questions about the rubric

13
Written Training Exercise

• Minimal written training was insufficient
• confusion about NA scores (want more examples)
• perfect score agreement was 34 before training
  and improved only slightly with training to 44
  (agreement within one score 77 ? 80)
• difficulty distinguishing physics approach
  application
• Math Logical progression most affected by
  training
• multi-part problems more difficult to score
• Grad students influenced by traditional grading
  experience

14
Grad Student Comments

• Influenced by traditional grading experiences
• Unwilling to score math logic if physics
  incorrect
• Desire to weight categories

• I don't think credit should be given for a
  clear, focused, consistent solution with correct
  math that uses a totally wrong physics approach
  (GS1)

• The student didn't do any math that was wrong,
  but it seems like too many points for such simple
  mathI would weigh the points for math depending
  on how difficult it was. In this problem the math
  was very simple (GS8)

15
Grad Student Comments

• Difficulty distinguishing categories
• Physics approach application
• Description logical progression

Specific application of physics was most
difficult. I find this difficult to untangle from
physics approach. Also, how should I score it
when the approach is wrong? (GS1)
I think description organization are in some
respect very correlated, could perhaps be
combined (GS5)
16
Analysis of Tests

• Calculus-based introductory physics course for
  Science Engineering students (mechanics)
• Fall gt900 students split into 4 lecture sections
• 4 Tests during the semester
• Problems graded in the usual way by teaching
  assistants
• After they were graded, I used the rubric to
  evaluate 8 problems spaced throughout the
  semester
• Approximately 300 student solutions per problem
  (copies made by TAs from 2 sections)

17
Exam 3 Question
Show all work! The system of three blocks shown
is released from rest. The connecting strings are
massless, the pulleys ideal and massless, and
there is no friction between the 3 kg block and
the table.
(A) At the instant M3 is moving at speed v, how
far (d) has it moved from the point where it was
released from rest? (answer in terms of M1, M2,
M3, g and v.) 10 points (B) At the instant the
3 kg block is moving with a speed of 0.8 m/s, how
far, d, has it moved from the point where it was
released from rest? 5 pts (C). (D).
SYMBOLIC
CUES ON MASS 3
How would you solve part A?
18
Grader Scores
Excludes part c) multiple choice
question. Average score the same (9 points or
half).
19
Rubric Scores

• Useful Description Free-body diagram (not
  necessary for energy approach)
• Physics Approach Deciding to use Newtons 2nd
  Law or Energy Conservation
• Specific Application Correctly using Newtons
  2nd Law or Energy Cons.
• Math Procedures solving for target
• Logical Progression clear, focused, consistent

20
Common Responses
Statements in red suggest students focused on M3,
which was cued in the problem statement
21
Example Student Solution
22
Example Student Solution
Only consider kinetic energy of mass M3. ? Was
cued in problem statement.
23
Example Student Solution

• (E1E2E3)

24
Example Student Solutions
Considers forces on M3, and uses Tmg (incorrect)
25
Example Student Solution
Answer is correct, but reasoning for F unclear
26
Findings

• The rubric indicates areas of student difficulty
  for a given problem
• i.e. the most common difficulty is specific
  application of physics whereas other categories
  are adequate
• Focus instruction to coach physics, math, clear
  and logical reasoning processes, etc.
• The rubric responds to different problem features
• For example, in this problem visualization skills
  were not generally measured.
• Modify problems to elicit / practice processes

27
Problem Characteristics that could Bias Problem
Solving

• Description
• Picture given
• Familiarity of context
• Prompts symbols for quantities
• Prompt procedures (i.e. Draw a FBD)
• Physics
• Prompts physics
• Cue focuses on a specific objects

• Math
• Symbolic vs. numeric question
• Mathematics too simple (i.e. one-step problem)
• Excessively lengthy or detailed math

28
Summary

• A rubric has been developed from descriptions of
  problem solving process, expert-novice research
  studies, and past studies at UMN
• Focus on written solutions to physics problems
• Training revised to improve score agreement
• Rubric provides useful information that can be
  used for research instruction
• Rubric works for standard range of physics topics
  in an introductory mechanics course
• There are some problem characteristics that make
  score interpretation difficult
• Interviews will provide information about
  response processes

29
docktor_at_physics.umn.edu http//groups.physics.umn.
edu/physed

• Additional Slides
• (if time permits)

30
Exam 2 Question (Different)
A block of mass m 3 kg and a block of unknown
mass M are connected by a massless rope over a
frictionless pulley, as shown below. The kinetic
frictional coefficient between the block m and
the inclined plane is µk 0.17. The plane makes
an angle 30o with horizontal. The acceleration,
a, of the block M is 1 m/s2 downward. (A) Draw
free-body diagrams for both masses. 5
points (B) Find the tension in the rope. 5
points (C) If the block M drops by 0.5 m, how
much work, W, is done on the block m by the
tension in the rope? 15 points
NUMERIC

• A block of known mass m and a block of unknown
  mass M are connected by a massless rope over a
  frictionless pulley, as shown. The kinetic
  frictional coefficient between the block m and
  the inclined plane is µk. The acceleration, a, of
  the block M points downward.
• (A) If the block M drops by a distance h, how
  much work, W, is done on the block m by the
  tension in the rope? Answer in terms of known
  quantities. 15 points

SYMBOLIC
31
Grader Scores
AVERAGE 15 points
Numeric, prompted Several people received the
full number of points, some about half.
AVERAGE 16 points
Symbolic Fewer students could follow through to
get the correct answer.
32
Rubric Scores
prompted

• Useful Description Free-body diagram
• Physics Approach Deciding to use Newtons 2nd
  Law
• Specific Application Correctly using Newtons
  2nd Law
• Math Procedures solving for target
• Logical Progression clear, focused, consistent

33
Solution Examples

• (numeric question w/FBD prompted)

Could draw FBD, but didnt seem to use it to
solve the problem
34
Solution Example

• (numeric question w/FBD prompted)

NUMBERS
NOTE received full credit from the grader
35

• (numeric question w/FBD prompted)

36

• Symbolic form of question

37

• Symbolic form of question

Left answer in terms of unknown mass M
38
Findings about the Problem Statement

• Both questions exhibited similar problem solving
  characteristics shown by the rubric.
• However
• prompting appears to mask a students inclination
  to draw a free-body diagram
• the symbolic problem statement might interfere
  with the students ability to construct a logical
  path to a solution
• the numerical problem statement might interfere
  with the students ability to correctly apply
  Newtons second law
• In addition, the numerical problem statement
  causes students to manipulate numbers rather than
  symbols

39
Findings about the Rubric

• The rubric provides significantly more
  information than grading that can be used for
  coaching students
• Focus instruction on physics, math, clear and
  logical reasoning processes, etc.
• The rubric provides instructors information about
  how the problem statement affects students
  problem solving performance
• Could be used to modify problems

40
References

• http//groups.physics.umn.edu/physed
• docktor_at_physics.umn.edu

• P. Heller, R. Keith, and S. Anderson, Teaching
  problem solving through cooperative grouping.
  Part 1 Group versus individual problem solving,
  Am. J. Phys., 60(7), 627-636 (1992).
• J.M. Blue, Sex differences in physics learning
  and evaluations in an introductory course.
  Unpublished doctoral dissertation, University of
  Minnesota, Twin Cities (1997).
• T. Foster, The development of students'
  problem-solving skills from instruction
  emphasizing qualitative problem-solving.
  Unpublished doctoral dissertation, University of
  Minnesota, Twin Cities (2000).
• J.H. Larkin, J. McDermott, D.P. Simon, and H.A.
  Simon, Expert and novice performance in solving
  physics problems, Science 208 (4450), 1335-1342.
• F. Reif and J.I. Heller, Knowledge structure
  and problem solving in physics, Educational
  Psychologist, 17(2), 102-127 (1982).

41

• Additional Slides

42
Independent scoring of student solutions by a PER
graduate student and a high school physics
teacher (N160)
Inter-rater Reliability
Kappa lt0 No agreement 0-0.19 Poor 0.20-0.39
Fair 0.40-0.59 Moderate 0.60-0.79
Substantial 0.80-1 Almost perfect
Category agree (exact) agree (within 1) Cohens kappa
Physics Approach 71.3 97.1 0.62
Useful Description 75.0 99.2 0.63
Specific Application 61.3 96.9 0.48
Math Procedures 65.6 99.4 0.51
Logical Progression 63.1 96.9 0.49
OVERALL 67.3 98.5 0.55
43
Inter-rater Agreement
BEFORE TRAINING BEFORE TRAINING AFTER TRAINING AFTER TRAINING
Perfect Agreement Agreement Within One Perfect Agreement Agreement Within One
Useful Description 38 75 38 80
Physics Approach 37 82 47 90
Specific Application 45 95 48 93
Math Procedures 20 63 39 76
Logical Progression 28 70 50 88
OVERALL 34 77 44 85
Weighted kappa 0.270.03 0.270.03 0.420.03 0.420.03
Fair agreement
Moderate agreement
44
All Training in Writing Example
Training includes the actual student solution
CATEGORY
RATIONALE
SCORE
45
Exam 1 Question 1

• A block of mass m2.5 kg starts from rest
  and slides down a frictionless ramp that makes an
  angle of ?25o with respect to the horizontal
  floor. The block slides a distance d down the
  ramp to reach the bottom. At the bottom of the
  ramp, the speed of the block is measured to be
  v12 m/s.
• Draw a diagram, labeling ? and d. 5 points
• b) What is the acceleration of the block, in
  terms of g? 5 points
• c) What is the distance, d, in meters? 15 points

INSTRUCTOR SOLUTION
46
Grader Scores
gt40 of students received the full points on this
question Was this an exercise or a problem?
47
Rubric Scores
Scores shifted to high end (5s) or NA
48
Problem Solving Process
1
2
3

1. Identify define the problem
2. Analyze the situation
3. Generate possible solutions/approaches
4. Select approach devise a plan
5. Carry out the plan
6. Evaluate the solution

4
5
6
http//www.hc-sc.gc.ca/fniah-spnia/images/fnihb-dg
spni/pubs/services/toolbox-outils/78-eng.gif
49
Developing Testing the Rubric
1. Draft instrument based on literature
archived exam data
3. Pilot with graduate students (brief training)
2. Test with two raters (consistency of scores)
4. Analyze pilot data (feedback scores)
Spring 2007
Fall 2007
Fall 2008
Spring 2008
Summer 2007
Summer 2008
Spring 2009
Summer 2009
5. Revise rubric and training materials. Retest.
8. Final data analysis reporting
6. Collect score exam problems from fall
semester of 1301 course.
7. (Interviews) Video audio recordings of
students solving problems.