DESIGN PRINCIPLES FOR OPEN BOOK OPEN WEB EXAMINATIONS

1
DESIGN PRINCIPLES FOR OPEN BOOK OPEN WEB
EXAMINATIONS
0
Professor Jeremy B Williams Dean, Corporate
Programmes Director of Research Universitas 21
Global www.jeremybwilliams.net
2007 International Conference on Online Foreign
Language Education Weihai City, Shandong, 17-18
August 2007
2
Overview

• Why OBOW exams?
• What is an OBOW exam?
• How to construct an OBOW exam
• THE CONTEXT
• THE TASK
• THE GUIDE TO THE TASK
• OBOW and Web 2.0
• Summary and conclusions

3
Disclaimer

• The aim of this presentation is to provoke
  thought - not to offend sensibilities!
• It is the view of an outsider looking in
• an economist not a linguist ?

4
1. WHY OBOW EXAMS?
5
Defining characteristic
1. Why OBOW exams?
0

• A commitment to authentic assessment
• "... Engaging and worthy problems or questions of
  importance, in which students must use knowledge
  to fashion performances effectively and
  creatively."

Grant Wiggins
6
Authentic assessment is not
1. Why OBOW exams?

• multiple-choice tests
• fill-in-the-blanks
• true-false
• matching words
• Students are passive learners ? surface learning

7
1. Why OBOW exams?
How do I engage my students?
0
Busy and committed language teacher
8
The 'unschooled' mind
1. Why OBOW exams?
Curiosity is a biological drive drummed out
of us by age 4!
9
1. Why OBOW exams?

• 'I hear, I forget.
• I see, I remember. 
•   I do, I understand.'

Confucius (551-479 BC)
10
1. Why OBOW exams?

• 'That what we have to learn to do, we learn by
  doing.'

Aristotle (384-322 BC)
/
11
"Life is an open book exam."
1. Why OBOW exams?

• Students need to be convinced of the authenticity
  of the task if they are to fully engage

Professor Alan Blinder, Princeton University
12
Do people conduct their work like this?
1. Why OBOW exams?
0
13
Or like this?
1. Why OBOW exams?
14
1. Why OBOW exams?

• Does this resemble any real world setting?

15
Constructive alignment (Biggs 1999)
1. Why OBOW exams?

• Is there is constructive alignment between the
  traditional examination instrument, and term time
  pedagogy and defined learning outcomes?

16
How many articles have been published in leading
educational journals over the last 25 years
extolling the virtues of closed book, invigilated
examinations?
1. Why OBOW exams?
17
1. Why OBOW exams?
How often do people solve problems in real life
by locking themselves in a room for 3 hours with
no books, no web access, not talking to anyone,
answering MCQs?
18
1. Why OBOW exams?
Is a closed book, invigilated exam an assessment
instrument more likely to foster cramming/data
dumping, or deep learning?
19
The Structure of Scientific Revolutions (1962)
1. Why OBOW exams?
0

• What's changed in the last 800 years?

Thomas Kuhn
20
1. Why OBOW exams?
0
The extent of the paradigm shift?
21
1. Why OBOW exams?
0

• You have to invigilated assessment because it's
  the only way we know for sure that they aren't
  cheating

22
1. Why OBOW exams?
Seldom observed points
0

• Students cheat during invigilated exams
• In the adult learner context, only a small
  percentage will attempt to cheat
• why pander to the lowest common denominator?

23
So what do you think so far?
1. Why OBOW exams?
So what's the alternative?
So what's the alternative?
So what's the alternative?
So what's the alternative?
So what's the alternative?
So what's the alternative?
So what's the alternative?
24
2. WHAT IS AN OBOW EXAM?
25
In brief
2. What is an OBOW exam?
Information and communication technologies

• A semi-structured mini-case (or caselette)
• Harnesses the power of ICTs to emphasise currency
  and real world authenticity
• A summative assessment item
• invites the student to draw on all that they
  have learnt (determining what is relevant).

26
'Willing suspension of disbelief'
2. What is an OBOW exam?

• Required to enjoy poetry, plays, novels
• and OBOW examinations!
• Enjoyment ? engagement ? deep learning

Samuel T. Coleridge (19th century poet)
27
Dull? Boring? Something to fear?
2. What is an OBOW exam?

• Final assessment ?
• a celebration of learning!
• Boredom and stress not conducive to deep learning
• Important to catch the imagination and appeal to
  the creativity of the learner
• Multimedia enhancements increase student
  satisfaction and learning
• (O'Brien and Seawell 2004 Vaughan 2001)

28
Key features
2. What is an OBOW exam?

• Students play the role of decision-maker,
  auditor, consultant or advisor
• They are presented with a unstructured
  (open-ended) problem that requires resolution
  (usually in the form of a set of recommendations)
• No pre-exam night 'cramming'

29
The template
2. What is an OBOW exam?

• THE CONTEXT the setting in which the
  problem/situation is identified and framed
• THE TASK the project and issues to resolve
• THE GUIDE TO THE TASK the setting of parameters
  and suggestions about methods/concepts/models/tool
  s to employ.

30
The ground rules
2. What is an OBOW exam?
0

• To minimise the scope for unethical behaviour
• Time period for the exam must be sufficiently
  tight
• Make clear (as a stated objective of the subject)
  that demonstrable application of learning is the
  key to success
• 'Text-book' impersonal responses will not attract
  high grades.

31
Confused yet?
2. What is an OBOW exam?
32
3. HOW TO CONSTRUCT AN OBOW EXAM(i) THE CONTEXT
33
Getting started
3. How to construct an OBOW exam the context

• Keep a look out for material all the time (not
  exam time!)
• e.g. Local newspaper, periodical websites,
  magazines, television news or current affairs
  programmes

34
What to look for
3. How to construct an OBOW exam the context

• A 'story' that learners can easily relate to in
  lay terms
• Objective to get them to think deeply about an
  issue
• Student to act as 'expert witness' - an effective
  mechanism for the validation of their learning in
  their own minds

35
Creating a scenario
3. How to construct an OBOW exam the context

• Having settled on a theme, gather together
  various media that can bring the case to life
• The inclusion of hyperlinks, photographs and/or
  streaming media adds a human dimension ?
  authenticity

36
Lead characters
3. How to construct an OBOW exam the context

• No 'story' is complete without lead characters
• Using real people with names, and pictures and
  voices acts as a catalyst to student engagement
• Fictional characters must give the appearance of
  being real!

37
Appropriate media where to look
3. How to construct an OBOW exam the context
38
(No Transcript)
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
(No Transcript)
45
Common problems 1
3. How to construct an OBOW exam the context

• Scenarios taken out of text-books
• Must be unique
• Must be no model solutions on the Internet
  somewhere
• Scenarios having the appearance of being taken
  out of text-books
• Lifeless
• Limited or no interactivity

46
Common problems 2
3. How to construct an OBOW exam the context

• Merging of THE CONTEXT with THE TASK
• Can lead to a problem being inadequately defined,
  and lacking in authenticity
• e.g Context You are the Managing Director of
  Bloggs and Co Ltd

47
Common problems 3
3. How to construct an OBOW exam the context

• Links that are overly academic
• The goal is to create a scenario
• Links to several long and turgid articles defeats
  the object
• Links that are too trivial
• Business periodicals are preferable to Randy's
  Daily Rant
• Links that do not add value
• Links for the sake of having links serve as a
  distraction

48
Common problems 4
3. How to construct an OBOW exam the context

• Boring, corporate-style images, instead of
  'action shots'

49
Common problems 5
3. How to construct an OBOW exam the context

• Audio-video links that are too long (gt 7 minutes)
• Audio-video links that add little value, or where
  the 'story' is mixed in with other stories.

50
3. HOW TO CONSTRUCT AN OBOW EXAM(ii) THE TASK
51
Setting THE TASK
3. How to construct an OBOW exam the task

• Role play ? the bridge between a learner's
  education and their professional practice
• Placing the learner in the role of the key
  decision maker, the expert advisor, or the
  auditor serves to validate the student's learning
• Revisit the stated learning outcomes what
  skills should they have?

52
Defining the parameters
3. How to construct an OBOW exam the task

• The definition of the assessment task might
  amount to no more than a paragraph
• Ideally it should invite a wide of variety of
  'equally correct' answers

53
Common problems 1
3. How to construct an OBOW exam the task

• Students get asked a traditional exam question
• Why did the XYZ company fail in this market?
  Critically discuss.
• Instead of
• Goh Chok Tong is concerned about the future
  viability of the company and he has employed you
  as consultant to advise

54
Common problems 2
3. How to construct an OBOW exam the task

• THE TASK is too structured, or includes too much
  instructional material
• Real life is complex and unstructured let the
  students figure it out for themselves

55
3. HOW TO CONSTRUCT AN OBOW EXAM(iii) THE GUIDE
TO THE TASK
56
Striking a balance
3. How to construct an OBOW exam the guide to
the task

• Avoid 'spoon-feeding' but
• not so unstructured a student is either struck
  by 'writers block' or goes off in the wrong
  direction.

57
Expectations
3. How to construct an OBOW exam the guide to
the task

• Before writing THE GUIDE, it is helpful to
  develop an outline of the kind of response one
  expects from the learner and, importantly,
• how this aligns with the prescribed learning
  outcomes
• Preparing THE GUIDE then becomes easier
• This process may also lead to THE TASK being
  refined

58
Common problems
3. How to construct an OBOW exam the guide to
the task

• THE GUIDE reads like a set of traditional exam
  questions, or includes too much instructional
  material
• Prose lacks authenticity e.g. give reasons for
  your answers, or more marks will awarded for
  reasoned argument emphasis added

59
4. OBOW AND WEB 2.0
60
The 2004 home computer
4. OBOW and Web 2.0
As predicted by scientists in 1954
61
Exams and Web 2.0?
4. OBOW and Web 2.0
62
Transformative pedagogy?
4. OBOW and Web 2.0
63
Online learning is not about transplanting the
F2F model
4. OBOW and Web 2.0
64
I'm here Professor, and I'm ready to learn
4. OBOW and Web 2.0
65
Eat it it's good for you!
4. OBOW and Web 2.0
0

• The instructivist view of education presumes
  knowledge exists independently of the knower, and
  that understanding is coming to know what already
  exists

66
The end of teaching and the start of learning
4. OBOW and Web 2.0
Old World Teacher Centric
New World Learner Centric
67
Constructivism in a digital age
4. OBOW and Web 2.0

• Building ? tinkering ? learning ? sharing ?
  building
• Deep learning can occur as the learner is
  actively engaged in, operating upon, or mentally
  processing, incoming stimuli
• Significantly, the student is both consumer and
  producer of information (John Seely Brown 2007)

68
4. OBOW and Web 2.0
69
Tools for creating digital stories
Windows Movie Maker
70
The role of faculty
4. OBOW and Web 2.0

• I never teach my pupils I onlyattempt to
  provide the conditions in which they can learn
• Albert Einstein

71
New ideas aren't always readily accepted
4. OBOW and Web 2.0
72
5. SUMMARY AND CONCLUSIONS
73
OBOW exams
5. Summary and conclusions

• A form of assessment that fosters understanding
  of learning processes in terms of real-life
  performance as opposed to a display of inert
  knowledge
• Test problem-solving skills not memory

74
What OBOW exams deliver
5. Summary and conclusions

• An assessment instrument that is more relevant to
  goals of the curriculum
• Greater authenticity, where real-world problems
  take centre-stage
• Allow ICTs to be harnessed to encourage
  interaction
• Student engagement with the assessment task ?
  induces deeper learning.

75
5. Summary and conclusions

• Studies show stimulation with audio will increase
  retention rate by 20. If stimulated with
  audiovisual, memory retention climbs to 30. If
  presented with interactive multimedia
  involvement, the retention rate can be as high as
  60.
• -- Tay Vaughan from Multimedia Making it Work,
  5th Edition, 2001 (ISBN 0-07-219095-7)

76
5. Summary and conclusions
Take the plunge
77
References

• (2007) E-xams harnessing the power of ICTs to
  enhance authenticity, (with Wing Lam and Alton
  Chua), Educational Technology and Society, 10
  (3), 209-221.
• (2007) Using digital storytelling as an
  assessment instrument Preliminary findings at an
  onlineuniversity, (with Kanishka Bedi),
  Proceedings of the 11th CAA Conference,
  pp.433-447, Loughborough, England, 10-11 July.
• (2006) The place of the closed book, invigilated
  final examination in a knowledge economy,
  Educational Media International, 43(2), 107119.