Use of an objective assessment tool to evaluate students

1
Use of an objective assessment tool to evaluate
students basic electrical engineering skills

• Nandini Alinier
• University of Hertfordshire, U.K.
• Engineering Education Conference 2006

2
Overview

• Background origins
• Application in an Engineering context
• Stages of OSTE
• Development of stations
• Preparation for sessions
• Pilot
• Implementation
• My perspective
• Students perspective
• Reflections

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
3
Background

• Objective Structured Clinical Examination (OSCE)
  (Harden 1975)
• Originally developed in the mid-seventies in
  Dundee, Scotland
• Used in medical education
• Method to objectively assess the skills of
  medical trainees
• Used in a project funded by the British Heart
  Foundation at UH during 2001-03 (Alinier et al
  2004)
• Involved as an assessor
• Lead investigators involvement
• Perceived positively by students lecturers
  (Alinier 2003)

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
4
OSCE

• Comprises 15 to 20 stations
• Station short task performed in 5 minutes
• Practical exercise
• Requires an assessor
• Theoretical exercise
• Pen paper
• Gap period between stations (1 minute)
• Students rotate individually through the stations
• Allows time for assessors to complete marking
  reset any equipment
• Timing of session by an electronic clock
• Students given explicit written instructions
• Objective marking sheet for each station

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
5
Applied to Engineering

• Project undertaken in the School of Electronic,
  Communication Electrical Engineering at the
  University of Hertfordshire, UK
• Funded by the Higher Education Academy
  Engineering Subject Centre (Learning Teaching
  Support Network (LTSN) Engineering)
• Academic year 2004-05
• Objective Structured Technical Examination (OSTE)

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
6
Study about the OSTE

• Used for formative assessment
• Piloted with 2nd year BEng students
• Conducted with 1st year BEng students
• Circulated feedback questionnaires for students
  assessors

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
7
Stages of the OSTE
Stage No. Stage
1 Development of OSTE stations
2 Preparation for OSTE sessions
3 Pilot of OSTE
4 Running of OSTE
Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
8
Stage 1 Development of OSTE stations

• Gathering a variety of ideas about essential
  electrical engineering skills
• Seminar within School
• Call for ideas via Email
• Identified appropriate range of exercises doable
  in 5 minutes
• Based on learning objectives for relevant group
  of students
• 16 stations were short-listed
• Developed instruction sheets marking criteria
  for each station

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
9
Stage 1 Development of OSTE stations (continued)

• Types of stations
• Theoretical pen paper
• Practical assessor required
• Computer-based assessor required

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
10
(No Transcript)
11
Stage 2 Preparation for OSTE sessions

• Development of PC-based clock with an integrated
  voice prompt
• Briefing of students
• Training of staff to act as assessors
• Designing of student and assessor feedback
  questionnaires

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
12
Stage 3 Pilot of OSTE

• Group of 15 volunteer 2nd year BEng students
• Resulted in changes in
• Tasks to be carried out
• Marking sheet
• Instruction sheet

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Example of a computer-based station
Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
17
Stage 4 Running of OSTE

• 32 BEng students in the second semester of their
  1st year
• 16-station OSTE
• 2 sessions on the same afternoon
• Formative OSTE
• Feedback given to students by assessor during the
  gap period

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
18
My perspective benefits

• Identifying students strengths and weaknesses
• Students forced to do practical work individually
• Being assessment-driven, hence OSTE encourages
  students to participate
• Feedback given individually to students

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
19
My perspective concerns

• Development of the OSTE stations is time
  consuming
• Once the stations have been developed, running
  the OSTE session is fairly straightforward
• Running the OSTE is human resource intensive
• However no need for marking after the session as
  most of it is done during the session
• A few students may find the OSTE stressful due to
  the intense assessment

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
20
Students perspective

• Lets students know what is expected of them and
  what their strengths and weaknesses are
• The OSTE session helped clarify what I was
  supposed to be able to achieve
• I know my weaknesses and what I need to know

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
21
Students perspective (continued)

• Students feel they have learnt something during
  the OSTE
• as a result of immediate feedback from the
  assessors
• OSTEs had addressed areas that would not have
  been picked up by other forms of lab work
• when marking coursework and lab books, the
  lecturer might assume competence (e.g. in setting
  up an oscilloscope), when in fact the student had
  only just coped or had relied on their partner to
  do that part of the experiment
• OSTE would show whether students could do the
  task
• Effectively getting the best settings, not just
  something usable
• Efficiently because of the tight time limit

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
22
Students perspective (continued)

• Unlike my concern, students did not find the
  exercise stressful
• Some people get a bit stressed before exams,
  whereas this is informal, you dont get so
  worried
• For similar reasons, students were against the
  idea of summative OSTEs, when they would probably
  attempt to cram specifically for the exam
• if were assessed for it were all going to
  prepare for it rather than be tested on what we
  know

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
23
Benefits

• Assesses specific key competencies that may be
  missed by other broader assessments
• Your lab skills are not always assessed, they
  just check your logbook and see your final
  result. This OSTE, it checks your method how
  youre connecting equipment not everyone uses
  the equipment properly in the right way

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
24
Benefits (continued)

• Provides an opportunity for students to learn
  core skills, allowing students to focus on deeper
  aspects of experimental work during practical
  sessions
• Lets students know what is expected of them and
  what their strengths and weaknesses are

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
25
Reflections

• Time consuming
• Human resource intensive
• Valuable tool
• Assess core skills
• Highlight deficiencies
• Focussed one-to-one teaching
• Can be tailored to assess learning outcomes
• Students think that OSTEs should be used
  formatively in every module they study

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.
26
Thanks for your attention

• Any questions?
• N.D.Alinier_at_herts.ac.uk

Nandini Alinier, EE2006, 25 July, Liverpool, U.K.