CS1 Assessment Using Memory Diagrams

1
CS1 Assessment Using Memory Diagrams

• Mark A. Holliday
• David R. Luginbuhl
• Dept of Mathematics and Computer Science
• Western Carolina University

SIGCSE Technical Symposium on Computer Science
Education Norfolk, VA March 5, 2004
2
Overview

• Background
• Introduction to Memory Diagrams
• A simple example
• Qualitative Assessment
• Noting and correcting common errors
• Quantitative Assessment
• Results so far
• Conclusion

3
Motivation

• Problem
• How to introduce and reinforce object-oriented
  concepts in an introductory CS course
• How to help CS1 students visualize the state of
  the computer as the Java program executes

4
Motivation

• Solution
• Develop a visual representation of each change in
  the state of memory (abstractly) due to the
  object-oriented effects of executing a sequence
  of Java statements
• Memory diagrams are more detailed than the code
  since there is at least one change in the state
  of memory per statement.
• Often there are several changes in the state of
  memory for a single statement.

5
Motivation

• Solution continued
• Write the fragment in both Java and the visual
  language
• The visual language must be detailed and precise
• The visual representation accesses an alternative
  and supplemental mode of thinking

6
Relationship to Psychology

• In other application domains psychologists have
  studied the effect on text comprehension by
  supplementing with visual representations
• P. Goolkasian, Picture-Word Differences in a
  Sentence Verification Task, Memory and Cognition,
  1996.
• B. Henderson, personal communication. 2003.

7
An Elementary Analogy
The white rabbit eats green lettuce.
rabbit eats lettuce.

• In elementary school, many of us were taught the
  rules of good grammar through sentence
  diagramming
• The teacher would show students the parts of a
  sentence with diagrams
• Students would also be required to diagram
  sentences to better understand the concepts of
  grammar

8
Method Invocation

• otherColor red
• aColor blue // a)
• aColor aColor.concat(otherColor)
• // rhs call is b)
• // rhs return is c)
• // lhs is d)

9
Language Design Principles

• Use shape to reinforce when concepts are the same
• e.g., all kinds of variables (local variables,
  parameters, and fields) are rectangles
• Use shape to reinforce when concepts differ
• e.g., variables, objects, classes all have
  different shapes (rectangles, circles, and
  diamonds)
• Use position to reinforce concepts
• e.g., the reference contained in a reference
  variable starts inside the variable, not on the
  border
• e.g., the rectangle representing an instance
  variable is placed inside the circle representing
  the object

10
Student Exposure to Diagrams

• We introduce these diagrams to students on the
  first day of CS1 class
• We ask students to produce diagrams of their own
• In in-class groupwork
• In weekly closed labs
• On quizzes and tests

11
Student Exposure to Diagrams

• Example in-class groupwork
• all groups at blackboards so that everyone can
  see the diagrams and multiple group members can
  be drawing simultaneously
• everyone must be able to explain during the
  demonstration
• no chalk for the strongest group member
• at least a third of lecture time is spent doing
  this (with an upperclass student helper assisting)

12
Student Exposure to Diagrams

• How do we have the time for all the groupwork and
  diagramming?
• Ours is a minimalist CS1
• no exception handling
• no inheritance, abstract classes, or interfaces
• no recursion
• no inner classes or event-driven code
• no graphics
• but we do real console I/O (not hidden)
• Bottom line reinforcement of concepts in a
  number of contexts
• But not just for learning

13
Student Assessment

• By having students use diagrams themselves, we
  have them demonstrate their comprehension of
  object-oriented concepts
• We believe these diagrams have potential for
  measuring programming comprehension

14
Qualitative Student Assessment

• Test problem from last year
• Diagram the program fragment at the right
• Note last line particularly
• Static public variable (System.out)
• Method composition
• PrintStream object
• Dog object
• Creation of a String object

Dog spot // fig a) spot new Dog("spot") //
right hand side is fig b) // left hand side and
equal sign is fig c) System.out.println(spot.toStr
ing()) // fig d)
Student attempts to produce figure a
15
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure a
• creating the object as well as the variable
• putting variable name inside variable rectangle

Solution
Student attempts to produce figure b
16
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure b
• not creating an object putting spot inside
  spot rectangle
• name field problems
• not showing fields rectangle
• putting fields rectangle on object boundary
• not showing String object for spot and that the
  name field holds a reference to that String
  object
• reference problems
• arrow in wrong direction
• arrow starts on the boundary of the rectangle
  instead of inside the rectangle

17
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure b
• already placing the reference in the spot variable

Student attempts to produce figure c
18
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure c
• placing the reference on the boundary of the spot
  rectangle instead of inside

Student attempts to produce figure d
19
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure d

PrintStream out
Solution
name
String namespot
20
Common Mistakes
Dog spot // fig a) spot new Dog("spot")
// right hand side is fig b) // left hand
side and equal sign is fig c) System.out.println(s
pot.toString()) // fig d)

• Figure d
• not show toString method invocation correctly
• not show the String object created correctly
• not show the reference to the new String object
  being passed as the argument to the println
  method call
• not show that System.out is a static public field
  of the System class
• now show println method invocation correctly

21
Quantitative Student Assessment

• Previous example illustrates how the diagrams are
  used qualitatively to assess student
  comprehension
• to pinpoint problems and provide proper
  reinforcement of specific concepts
• Can we also use them for quantitative assessment?
• Can we also evaluate how effective they are in
  assessment?

22
Quantitative Assessment (cont.)

• Completed Study Correlation of score on memory
  diagram question with rest of test and with
  course score
• Three test questions from two CS1 classes
• Two studies (correlation with rest of test and
  correlation with course score) for each of the
  three questions (three experiments)

23
Quantitative Assessment (cont.)

• An example quantitative comparison using the
  third test question (experiment three)
• Histogram is sorted by memory diagram question
  score
• The three series of scores appear to track each
  other
• Mean score of the memory diagram question is
  significantly lower
• not surprising since a correct memory diagram
  requires a deeper understanding of the effect of
  a program fragment

24
Conclusion Memory Diagrams

• A relatively low-tech approach for teaching OO
  concepts
• Well-suited for classroom, labs, exams
• Write a program fragment in both Java and in
  the visual language
• Importance of shape and placement for reinforcing
  concepts
• Having students make their own diagrams adds to
  this reinforcement

25
Conclusion Memory Diagrams

• Promise of diagrams for measuring comprehension
• If students can diagram what is happening in
  memory, they are probably understanding the
  deeper meaning of the program
• Measured assessment effectiveness relative to
  rest of test and course score
• Currently measuring assessment effectiveness
  compared to corresponding Java program fragment

26
Conclusion Memory Diagrams

• To learn more
• Mark A. Holliday and David Luginbuhl, Using
  Memory Diagrams When Teaching a Java-Based CS1,
  Proc. of the 41st ACM Southeast Conference,
  Savannah, GA, March 2003.
• Mark A. Holliday and David Luginbuhl, CS1
  Assessment Using Memory Diagrams, Proc. Of
  SIGCSE 2004, Norfolk, VA, March, 2004
• Mark A. Holliday, Introducing Java Using Memory
  Diagrams, (include short guide to the notation)
  http//cs.wcu.edu/holliday/LectureNotes/150/lectu
  res.html
• Presentation slides http//cs.wcu.edu/holliday/s
  igcse04.ppt