Drawing Diagrams in an Online Examination

1
Drawing Diagrams in an Online Examination

• Pete Thomas
• with help from
• Neil Smith Kevin Waugh

2
Online Examinations
3
Issues

• What features should a drawing tool provide?
• How familiar should students be with the tool
  prior to the examination?
• How should the tool be provided to students in
  order to be used under examination conditions?
• How should a diagram be represented for
  transmission to the server?
• How should a diagram be represented for grading
  purposes?
• How to grade a diagram?
• How to provide feedback on a students diagram?

4
A typical exam question
Use the drawing tool to draw a diagram that
illustrates how the data hazard inherent in the
execution of the pair of instructions ADD
R2, R3, R1 SUB R1, R5, R4 by a 4-stage
pipeline, can be overcome.
5
A simple drawing tool
6
A typical student drawing
7
Opinions of the Drawing Tool
8
Student comments

• The scroll bars in the boxes I found offputting,
  I started off drawing boxes that were too small -
  I nearly didnt do this question because of
  that.
• Text boxes needed to be bigger so could see
  whole answer at once without scrolling
• The text boxes could be a bit bigger at least
  big enough to contain the expected answers. It
  was a bit disconcerting trying to scroll up and
  down the boxes.
• Very easy once I got the hang of it

9
Opinions of the Online Examination

• Drawing tool not a drawback

10
The second trial

• 15 students
• 12 attempted the drawing question
• 9 answered the question successfully (but not
  necessarily correctly)
• 3 tried to draw a diagram but gave up
• 3 did not attempt the question

11
Discussion

• There is a definite reluctance on the part of
  some students to attempt to use a software tool
  under examination conditions, primarily due to
  the anticipated length of time it would take to
  complete a drawing.
• The exam question, whilst in the compulsory part
  of the paper, was of a low stakes variety.
• Only 3 out of 15 students did not attempt the
  question - but this could be because they did not
  know the answer!
• If the students had had experience of the tool
  prior to the examination, this hesitancy would
  have been reduced.
• One would normally expect students to be exposed
  to the tool during their normal studies.

12
Imprecise diagrams

• Incomplete
• Contains extraneous material
• Malformed
• Typical student diagrams!

13
A model answer
14
A typical student drawing
15
Understanding diagrams

• Segmentation and Assimilation
• Translate a raster-based image into a set of
  diagrammatic primitives such as boxes, lines and
  text
• Identification
• Use domain knowledge to identify minimal
  meaningful units (MMUs)
• Aggregation
• Combines MMUs into higher level abstract features
• Interpretation
• Look for meaning in the diagram (e.g. comparing a
  student diagram with a specimen solution)

16
Marking diagrams
17
Automatic marking (pipelines)

• 12 student answers to a drawing question
• Tutor mark x 2.78 (out of 5) s 1.05
• Auto mark x 2.73, s 1.09
• Spearmans rho 0.529, significant at the 0.05
  level (2-tailed), N 12
• Pearsons r 0.763, significant at the 0.01
  level (2-tailed), N 12
• But, N 12

-
-
18
Current work

• Automatic marking of tutor marked assignments
• Entity-Relationship (ER) diagrams
• Feedback
• Building a tool for revision purposes

19
The revised tool for E-R diagrams
20
Automatic marking (ERDs)

• 26 student answers to a drawing question
• Tutor mark x 21.35, s 3.43 (range 13-25)
• Auto mark x 22.42, s 2.12 (range 18-25)
• Spearmans rho 0.957, significant at the 0.01
  level (2-tailed), N 26
• Pearsons r 0.939, significant at the 0.01
  level (2-tailed), N 26

-
-
21
Marking ERDs
22
Feedback student answer
23
Feedback compare with specimen solution
24
Conclusions

• Students seem to like online exams.
• Using a drawing tool under online exam conditions
  does not seem to pose major problems,
  particularly if the tool has been used prior to
  the exam.
• A simple drawing tool can be an effective
  addition to an online exam.
• Having some success in automatically marking
  diagrams.
• Making progress towards meaningful feedback.