Programming Aptitude Testing Materials, Data analysis

1
Programming Aptitude Testing Materials, Data
analysis

• Saeed Dehnadi
• Middlesex University

2
I talk about

• Test materials
• Questions
• Mental Models
• Answer Sheet
• Mark Sheet
• Interpretation Algorithms
• Results
• Middlesex University (initial test) - 2005
• University of Newcastle (Australia) - 2006
• York - 2006
• Strathclyde - 2006
• Middlesex University (second test) - 2006
• Collaborators
• Summary

3
Questions 1 (single assignment)
4
(M1) Mental model move value

• a b
• The value of b is given to a and b changes its
  value to zero.
• a ? b // b ? 0
• Ans ( a 20 , b 0 ) 8th Answer

5
(M2) Mental modelcopy value

• a b
• The value of b is given to a and b keeps its
  original value.
• a ? b // b unchanged
• Correct conception of LHS ? RHS
• Ans ( a 20 , b 20 ) 4th Answer

6
(M3) Mental model move value

• a b
• The value of a is given to b and a changes its
  value to zero.
• b ? a // a ? 0
• Ans ( a 0 , b 10 ) 3rd Answer

7
(M4) Mental modelcopy value

• a b
• The value of a is given to b and a keeps its
  original value.
• b ? a // a unchanged
• Ans ( a 10 , b 10 ) 1st Answer

8
Mental models (M5) (copy add) value

• a b
• The sum of a and b is given to a, and b keeps its
  original value.
• a ? (a b) // b unchanged
• Ans ( a 30 , b 20 ) 2nd Answer

9
Mental models (M6) (move add) value

• a b
• The sum of a and b is given to a, and b changes
  its value to zero.
• a ? (a b) // b ? 0
• Ans ( a 30 , b 0 ) 10th Answer

10
Mental models (M7) (copy add) value

• a b
• The sum of a and b is given to b, and a keeps its
  original value.
• b ? (a b) // a unchanged
• Ans ( a 10 , b 30 ) 9th Answer

11
Mental models (M8) (move add) value

• a b
• The sum of a and b is given to b, and a changes
  its value to zero.
• b ? (a b) // a ? 0
• Ans ( a 0 , b 30 ) 5th Answer

12
Mental models (M9) no changes

• a b
• a and b keep their original values.
• a unchanged // b unchanged
• Ans ( a 10 , b 20 ) 6th Answer

13
Mental models (M10) Equality

• a b
• Assignment is a simple equation, and then all
  equal values of a and b are acceptable.
• Ans ( a 10 , b 10 ) and
• Ans ( a 20 , b 20 )
• both 1st and 4th answers

14
Mental models (M11) Swap values

• a b
• a and b swap their values simultaneously.
• a ? b a gets bs value
• b ? a b gets as value
• Ans ( a 20 , b 10 ) 7th Answer

15
Models for a single assignment int a 10
int b 20a b
16
Answer Sheet (Question1)
17
Questions 2 (single assignment)
18
Questions 3 (single assignment)
19
Question 4 (two assignments)
20
Mental modelM1 Sequence

• a b
• b a
• Model is M1 that applies sequentially through
  both statements
• L1) a ? b and b ? 0 then
• a 20 and b 0
• L2) b ? a and a ? 0 then
• b 20 and a 0
• Single answer
• a 0 , b 20

21
Mental modelM1 Independent (M1 I)

• a b
• b a
• Model is M1 that applies independently for each
  individual line.
• L1) a ? b and b ? 0
• L2) b ? a and a ? 0
• Multiple answers
• a 20 and b 0
• a 0 and b 10

22
Mental modelM1 Simultaneous-single (M1 Ss)

• a b
• b a
• M1 Ss Same as (M1 I), subjects only interested on
  Left-hand-side values of statements and ignores
  the right-hand-side values.
• L1) a ? b and b ? ignores
• L2) b ? a and a ? ignores
• Single answer
• a 20 , b 10

23
Answer Sheet (question4)
24
Question 5 (two assignments)
25
Question 6 (two assignments)
26
Answer Sheet (question6)
27
Question 7 (three assignments)
28
Answer Sheet (question7)
29
Mark Sheet
30
Mark Sheet Interpretation(Tree Structure)
31
Mark Sheet Interpretation (Mode Structure)
32
Data sample
33

• Middlesex University
• Barnet College
• First Experiment
• 2005

Data provided by Saeed DehnadiMiddlesex
University
34
Experiment

• Subjects
• 30 students from Middlesex University
  Introductory to programming (Java)
• 31 students from Barnet College Further Education
  Programming Course
• Time
• Week 0 of the course
• Questions
• 12 questions

35
CategoriesWeek 0

• Consistent (45)
• Inconsistent (39)
• Blank (16)

36
CategoriesWeek 3

• Consistent (67)
• Inconsistent (33)
• Blank (0)

37
Stable group membership(Week3)
Data provided by Saeed Dehnadi Middlesex
University
38
Pass/Fail statistics(Initial Result)
P lt 0.01
Data provided by Saeed Dehnadi Middlesex
University
39
Two populations(Initial Result)
40

• Australian Data
• University of Newcastle
• 2006

Data provided by SimonSchool of DCIT (Design,
Communication, and Information Technology)
41
Pass/Fail statistics(First glance)
P lt 0.01
Data provided by Simon The University of
Newcastle
42
Pass/Fail statistics Exclude m2
P lt 0.025
Data provided by Simon The University of
Newcastle
43
Pass/Fail statistics Exclude m2
P lt 0.001
Data provided by Simon The University of
Newcastle
44

• Strathclyde Data
• The University of Strathclyde
• 2006

Data provided by Linxiao Mu
45
Mental models in week 20
Value assignment
Reference assignment
Data provided by Linxiao Mu The University of
Strathclyde
46
Value assignment mental models in final exam
P lt 0.05
Data provided by Linxiao Mu The University of
Strathclyde
47
Value assignment mental models in 4 in-course
tests
P lt 0.01
Data provided by Linxiao Mu The University of
Strathclyde
48
Reference assignment mental models in final exam
P lt 0.025
Data provided by Linxiao Mu The University of
Strathclyde
49
Reference assignment mental models in 4
in-course tests
P lt 0.05
Data provided by Linxiao Mu The University of
Strathclyde
50

• York
• The University of York
• 2006

Data provided by Dimitar Kazakov
51
Pass/Fail statistics
P lt 0.001
Data provided by Dimitar Kazakov The University
of York
52

• Middlesex University
• Second Experiment
• 2006

Data provided by Saeed Dehnadi Middlesex
University
53
Pass/Fail (Quiz 1)
P lt 0.20
Data provided by Saeed Dehnadi Middlesex
University
54
Pass/Fail (Quiz 2)
P lt 0.01
Data provided by Saeed Dehnadi Middlesex
University
55
Collaborators

• Ireland - School of Computing in DCU
• Australia - Charles Stuart University
• Mozambique - Instituto Superior de Transportes e
  Comunicações, Maputo
• UK- Bournemouth University (Royal School of
  Signals in Blandford)
• UK University of Birmingham
• UK University of Sheffield
• UK Kings College, London
• Canada - University of Toronto
• Denmark University of Aarhus
• USA A-level in Computing, High school
• UK - A-level in Computing, North West College
• Germany Computing study, teacher training
  Institution

56
Summary

• OriginalityWe are the first to have an aptitude
  test that works.
• ObjectivityWe have objective test materials that
  can be used by others.
• SpeculationStudents who can handle meaningless
  rules learn programming more easily.

57

• To be continue ..