A University Strategy for Delivering Online Assessment

1
a
A University Strategy for Delivering Online
Assessment
Dr Richard Parsons Director of the Learning
Centre University of Dundee r.parsons_at_dundee.ac.u
k
2
University of Dundee

• More than 10 years substantial experience of
  online assessments
• Blackboard clients Academic Suite (virtually
  100 modules)
• Delivering up to 3500 formative assessment
  sessions a day with QM Perception
• Delivering 50 summative exams a year with QM
  Perception
• Range of online assessment activities recognised

3
Blackboard Server Configuration
Summer 2006 Bb upgrade 6.3 to 7.1 on Unix, QM
application from 3.4 to 4.1, Bb - QM bridge
software (twice), QM repositories for
assessments, Blackboard building block
deployments and upgrades, Solaris and Oracle
database upgrades, Annual module rollovers,
Enhanced PDP programmes, Medical School
curriculum map integration, University
restructuring configuration changes
4
Online Assessment System
Assessment hardware design deployment
Assessment software design deployment
Clear policy and procedures
Central assessment support
Learning systems integration
Staff and student education
5
The Hardest Task

• Remains writing quality questions
• Staff time is precious
• Students are more critical than ever before
• Assessments to be aligned
• Questions need to be prepared ahead of time
• Quality runs to feedback, distracters, variety,
  depth, currency
• Authoring Q within the software is secondary

6
Staff and Student Education

• For 4 years at Dundee we have run an Online
  Assessment course, online
• Covers technical and pedagogical aspects
• 5 weeks long, 3 to 6 hours a week.
• Formally accredited by our Education Faculty (10
  SC)
• Building a bank of staff skills
• Most students need little guidance

7
Examples of Assessment Strategies

• 15 Q from pool of 100, max mark taken (Genetics)
• Take assessment first, then repeat in an open
  fashion. Mark taken from first run. (Medicine)
• 10 numerical questions, each from a pool of 50
  randomly generated Q (Life Sciences)
• Student revision students invited to compose 2
  Q each with FB, all compiled automatically,
  checked made available for open assessment (Env
  Sci)
• Open assessment, one submission only (Law)
• Summative (range of departments)

8
Quality Hardware and Systems

• Central servers built maintained by ITS
• LC involved in system design
• Three servers, one formative, one summative, one
  shared repository
• HP BL20p Blades Dual Pentium 3400 MHz 6GB RAM,
  dual NIC backplane.
• Databases
• Formative on Oracle Unix server
• Summative on Oracle Unix server
• Shared on Oracle Unix server

9
University of Dundee Questionmark Perception V4
Systems Model
Participant
Participant
Oracle 10G
Oracle 10G
QMP V4.2 CAA-02 Production Formative
Repository
Participant
QMP V4.2 CAA-01 Production Exam Repository
Participant
LC Administrator
QM / Bb integration
QMP V4 Authoring Manager
Oracle 10G
Oracle 10G
Participant
QMP V4.2 CAA-03 DevelopmentRepository
Blackboard 7.1 My Dundee
Local Author (Staff)
Participant
QMP V4 Authoring Manager
Participant
Local Author (Staff)
QMP V4 Authoring Manager
Local Author (Staff)
David Walker, University of Dundee
10
Clear Policy and Procedures

• Central online assessment system required clear
  guidance
• Prepared draft Policy and Procedures
• Generally follows BS79882002
• Developed for about a year, inc. consultation
• Passed at University Senate
• Excellent for ensuring quality
• Runs to 17 pages
• Additional documentation for Online Assessment
  system administrators

11
Summative Procedures

• Assessment must be tested
• Assessment loaded to both servers, scheduled on
  one
• Central staff attend examinations
• Use special exam accounts
• Use Perception Secure Browser
• Candidates complete Examination Attendance form
• Immediately after the exam a backup of results is
  taken
• Use two way radios or mobiles
• No system is fullproof, 99 success target ?

12
Self Peer Assessment Bb Building Block

• Assessments are added and edited by the
  instructors in any course
• One or more questions, each with one or more
  marking criteria
• Marking criteria as discrete values or range
• Feedback provided as score and text if required
• Students automatically allocated to mark peers (1
  to many)
• Peer assessments are assigned automatically
• Self assessment is an option
• Marks and reports are visible to the Instructor
• All marks can be downloaded for analysis in excel
• Moderated results automatically passed to Bb
  Gradebook
• Exercise text is available for download and
  online plagiarism checks
• Questions and criteria are stored for reuse and
  exchange

13
(No Transcript)
14
(No Transcript)
15
The Dundee My PDP resources
Online resource to support PDP timetabled
induction sessions scope for faculty tailoring.
Available for free under license.
Each student can prepare an online portfolio
based on templates, culminating in prioritised
action points and an enhanced CV. Portfolios can
be shared online.
16
Mobile Campus Map

• Java game-based map
• Rich links with key campus information
• Single download
• Popular with students

www.dundee.ac.uk/mobilemap
17
Additional Online Assessment

• Self Peer
• Framework for plagiarism advice and management
• Procedures for online submission
• Discussion board assessments
• ePortfolio assessment rubrics and tools
• Classroom/personal response systems
• Business intelligence activities
• Overall framework of OA support at Dundee

18
Individual concerns within an online assessment
world
College Officer Can we book an IT suite for 240
students on Wednesday the 25th?Two students have
additional requirements.Will the results
transfer directly to the SRS?
Traditional Lecturer Who can electronically
author my questions?Can we use textbook
questions?Is this assessment driving learning?
eLearning Manager Are the staffing levels
correct?Is our software choice appropriate?Are
the server specifications correct to cope with
the expected load?Are staff education procedures
in place?
Lecturer Are my questions too simple?Is this
technology reliable?Do the students find it
tedious?Where can we obtain challenging
questions?Are these assessments aligned with our
learning outcomes?
IT Desktop Administrator Does the desktop
software operate correctly?Are the exam
accounts secure and available?Are the PCs well
maintained and reliable?
Timetabling Officer Can the proposed exams be
fitted into the best rooms?Are assessments
receiving appropriate prioritisation?
Student This is nice and privateI am getting
real feedback hereDo I know the answers?Can I
learn from this?I am at home in this online
environment
Distance Student Is my technology compatible?Is
my connection reliable?Is this a replacement for
F2F?
IT Server Administrator Is the server fully
patched, reliable and secure?Is the OA software
functioning correctly and reliablyDo we have
monitoring and is it working?Are the logging and
backup procedures working correctly?
Online Assessment Officer Are all assessments
up-to-date and correctly available?Is the
server-client functionality all compatible?Will
the next examination run smoothly?Are the staff
education programmes ready?
Examinations Officer Are these results 100
reliable?secure?Do our procedures represent the
best possible practice?
Richard Parsons, University of Dundee
19
Elements
Actors
Procedural
Each addressed by a tasked individual(s) within a
supported team, and each element able to cope
with growth and changes Revisit annually? The
entire operation has limited inertia, limited
history, sustainable staffing, space, budget.
Represents a framework
OA policy and procedures
ZZ AA
Student and staff training
XX WW
Examination practices
Academic staff XX WW
Room booking procedures
College staff
Student personal PCs
Web guidance VV
University student desktop
BB CC
Question creation
Academic staff XX WW
Question/assessment repositories
XX WW
Network
DD EE
OA application software support
XX WW
13 specialist staff academics college staff
Technical database considerations
FF GG
Operating system web server
XX HH
Hardware infrastructure
HH II
Technical
Richard Parsons, University of Dundee
20
The Annual Cycle of Online Assessment
Sept.
Testing tuning
Modules begin
Online resit exams
Annual frameworkmeeting
Summer upgrade window
Online exams
Dec.
June
Mid semester window
Online exams

• Growth may occur each year
• Changes are made to the systems
• A framework allows the process to be sustained

OA staff education module
Richard Parsons, University of Dundee
March
21
Summary

• Dundee has an institutional online assessment
  some elements of a strategic framework
• Software, hardware, PP, staffing, integration
  and education. 13 specialist staff.
• Secure and reliable, but straightforward to use
  for students and staff
• Let the academics create the Qs to stimulate
  learning
• Costs are neither insignificant or astronomical

22
(No Transcript)
23
(No Transcript)
24
(No Transcript)