Assessment, progress and evaluation of students

1
Assessment, progress and evaluation of students

• Bart Rienties (UM)
• Dirk Tempelaar (UM)
• b.rienties_at_algec.unimaas.nl
• d.tempelaar_at_ke.unimaas.nl
• http//www.web-spijkeren.nl

2
Agenda

• 1. Introduction
• 2. Usage of Assessment within 10 courses
• 3. Tools for evaluation of progress of students
• 4. Set-up for Remedial teaching
• 5. Evaluation results
• 6. Near Future On-line Summercourse
• 7. Discussion

3
1. Introduction

• SURF Web-spijkeren (http//www.web-spijkeren.nl/)
• Hoofddoel van Web-spijkeren project onderdeel
  Assessment is
• het onderzoeken welke rol ICT-gereedschappen
  kunnen spelen in het enerzijds diagnosticeren van
  verschillen in voorkennis, en anderzijds het
  remediërend aanbieden van leerstof wanneer uit de
  diagnose blijkt dat de voorkennis van studenten
  niet aan bepaalde minimumeisen voldoet.
• Emerge-project
• Tempelaar de Gruijter (2005), Eindrapport
  Computertoetsing, TU Delft, te vinden op
  http//www.e-mergeconsortium.nl
• ForUM
• Rienties, Tempelaar, Arts (2004), Eindrapport
  ForUM, Universiteit Maastricht, te vinden op
  http//www.fdewb.unimaas.nl/algec/
• Ontwikkelingen elders Draaijer (2004), DU,
  Appollo-project

4
2. Usage of Assessment within 10 courses
5
3. Tools for evaluation of progress of students

• On-line Assessment
• Intersobit/Self Assessment
• ALEKS
• Blackboard/ELEUM
• Participation in groups
• Assignments/Presentations
• Mindmaps

6
4. Set-up for Remedial teaching
7
Entry-Assessment

• 1. Self-Assessment
• Perceived lacks of knowledge
• - Overestimation/Underestimation
• - Myopic behaviour
• 2. Entry-Assessment
• Real assessment of skills/knowledge
• - Takes a lot of time/effort
• Via ALEKS (adaptive testing)
• Via ELEUM (simple testing)

8
Self-Assessment
9
Entry-Assessment
10
QM ALEKS, entry assessment
11
ALEKS learning pie
12
Prior Knowledge (UCM)
Proportions and percents
54
32
23
Signed numbers and variables difficult items
27
43
27
13
(No Transcript)
14
Didactical forms/tools during course

• On-line (progress) assessment
• (On-line) Collaboration
• Problem-based Learning
• Mindmaps/Portfolio/etc.

15
Example Economics and Business
Lecture
Post-discussion
Pre-discussion
Official ELEUM test Trail I
Mindmaps
Fail
Reading Literature
Pass
Official ELEUM test Trail II
Feedback
16
(No Transcript)
17
(No Transcript)
18
Ideal individual learning-path
Based on outcomes of entry-assessment, a student
could be evaluated at any point on the knowledge
space of topic X. Student A can have a different
learning path than Student D to reach point
f Ideally, the learning materials and teachings
methods should adapt to the knowledge/skills of
each individual student.
19
ALEKS learning path

• Knowledge State can be described by
• All mastered items
• Outer Fringe (Ready to learn ) Inner
  Fringe (Most recently learned)

20
ALEKS Ready to learn Log
21
(No Transcript)
22
End-knowledge test

• On-line End-knowledge test
• Thus far only voluntary
• Reflection of knowledge accumulation
• Self-assessment
• Written exam

23
5. Evaluation Economics and Business

• Number of students 417
• Number of hits in ELEUM 140.070 in 3 months
• 6 trial I, 6 trial II and resit ELEUM-test
  possibilities
• 34 students failed
• 190 error-messages (4 per test)

24
Mastery of domains on scale 1-10
25
Evaluation of progress per educational programme

• Foreign students lower pre-knowledge
• German Advanced students catch up
• Other Foreign and Dutch non-economics students
  stay behind
• Dutch Advanced Economics students no clear
  advantage

Average_PK Average Self Assessed Prior
Knowledge (N 367)Average ELEUM Average of
highest grade of ELEUM trial 1, 2 and resit (N
412)Average EKP1 Average Self Assessed End
Knowledge (N 218)Average EKP2 Average
ELEUM-tested End-knowledge(N 249)Exam Grade
for written exam Q1-Q160 (N 412) VWOE1 Basis
Dutch Economics (N21) VWOE12 Advanced Dutch
Economics (N100) VWO other dutch (N11) GERLM
German Advanced math (n57) GERGM German Basic
Math (N134) OID Foreign non-German (N38)
26
Evaluation of on-line assessment

• ELEUM-tests very useful (8.0 out of 10)
• Germans students sense lack of pre-knowledge
• Students with low pre-knowledge at end more
  positive about ELEUM-tests

Average_PK Average Self Assessed Prior
Knowledge (N 367)ELEUM In this course ELEUM
has helped in my learning (N 234)Pre/Post test
I think that the use of pre and end-knowledge
tests in this course is valuable (N
238)ELEUM-tests I think that the use of the
ELEUM-tests in this course is valuable (N
238)Lack of knowledge At the beginning of the
course, I had the feeling that I lacked certain
knowledge in comparison to my fellow students (N
218) VWOE1 Basis Dutch Economics (N21) VWOE12
Advanced Dutch Economics (N100) VWO other dutch
(N11) GERLM German Advanced math (n57) GERGM
German Basic Math (N134) OID Foreign
non-German (N38)
27
6. Near Future On-line Summercourse

• On-line Summer-course for Mathematics
• 50 students selected in april-may
• Start with ALEKS/ELEUM in june-august
• On-line Summer course for Economics
• ?? Students selected in april-may
• Start with ELEUM in june-august

28
Literature References

• Brouwer, N., Rienties, B., Engelen, A. van.
  (2004). Controlling Document Project
  Web-spijkeren. Universiteit van Amsterdam,
  Universiteit Maastricht, Erasmus Universiteit
  Rotterdam. On http//www.web-spijkeren.nl/.
• Draaijer, S. (2004), Flexibilisering van
  toetsing. Creëren van voorwaarden voor een breed
  toetsrepertoire in het hoger onderwijs. In
  druk/uitgave. Utrecht. Digital Universiteit.
• Falmange, J., Cosyn, E., Doignon, J., Thiéry,
  N. (2004). The Assessment of Knowledge, in Theory
  and in Practice. Te downloaden van
  http//www.business.aleks.com/
• Rienties, B. (eds), Tempelaar, D., Wieland, A. et
  al. (2005). Deliverable 3 Formats beschrijving
  Kennisformats, On-line Assessment bij Erasmus
  Universiteit, Universiteit Maastricht en
  Universiteit van Amsterdam. (Formats description
  of knowledge formats and on-line Asssessment).
  SURF, on http//www.web-spijkeren.nl/
• Rienties, B. and Tempelaar, D. (2004). "Flexibele
  Instroom, Flexibel Aanbod, Er leiden veel wegen
  naar Maastricht". Eindrapportage ForUM,
  Universiteit Maastricht.
• Rienties, B. and Woltjer, G. (2004). "Regular
  On-line Assessment, Motivation and Learning".
  Meteor Research Memorandum, RM04030.
• Tempelaar, D. Gruijter, D. de (2004), 
  "Ervaringen met online toetsen m.b.v. Blackboard
  Course Cartridges". (Experiences with on-line
  testing and Blackboard Course Cartridges), on
  http//www.e-mergeconsortium.nl
• Woltjer, Geert (2004a), "An efficient format for
  a problem based introductory economics course",
  Meteor Research Memorandum RM/04/015,
  Universiteit Maastricht.

29
7. Discussion

• Is ELEUM Assessment a good tool to monitor
  progress of students?
• Are summer courses necessary if UM wants to
  become a truly international university?
• What is the role of ELEUM in providing summer
  courses?

30
Assessment, progress and evaluation of students

• Bart Rienties (UM)
• Dirk Tempelaar (UM)
• b.rienties_at_algec.unimaas.nl
• d.tempelaar_at_ke.unimaas.nl
• http//www.web-spijkeren.nl