Meta-cognitive Education or University Challenge?

1
Meta-cognitive Education or University Challenge?
Do we really need examinations?
or Beware External Examiners Bearing Comments

• Brian Whalley
• Queens University
• Belfast
• b.whalley_at_qub.ac.uk

NB, as well as the slide show, there are notes
and references in the Notes section.
2
Some things Im going to cover (but not
necessarily the order)

• A Victorian system?
• A quick analysis of examinations at Level1
• What do examinations do?
• Deep Learning
• Feedback in (assessment) systems
• Metacognition (Ill explain about this)
• Where should assessment fit?
• Criteria-based assessment
• Just-in-Time Teaching, preflights
• What type of assessment?
• Deep Assessment (a meaningful concept?)

Education is a highly parallel and occasional,
not serial, system
3
Trial and error - how can we provide good
learning experiences?

• 'You know what a learning experience is? A
  learning experience is one of those things that
  says, 'You know that thing you just did? Don't do
  that.
• (Douglas N Adams, 1992)

Do examinations provide good learning experiences?
Photo Chris Ogle
How to avoid the panic?
4
A Victorian system?

• On the basis of University Challenge (my rules)
• Please answer the Qs on the folded sheet,
• You have 3 minutes to do this
• This is not anonymous as your answers will
  determine whether you get lunch
• (1 course, 2 courses, 3 courses)
• Ill collect them in, mark them and then tell you
  what you get for lunch
• (or not, if you have not done your revision)
• But Im not going to tell you how near you got to
  a better meal
• Oh, and theres no veggie option!

5
Questions regarding Level 1 (semester 1)

• Do you have end of semester exams?
• Are these essay type?
• Two questions in 2 hours?
• Do these exams count towards final degree
• (and if yes, what )
• Do you give students guidance (or practice) on
  what to expect in exams?
• Are any essays set which are designed?
• Do you give students remarks or discussion on how
  well they have done post exams?
• (and how is this done)?
• Do you have MCQs (or variant) rather than essays.

6
Assessment(some more issues)

• Do you want me to be the sole maker?
• Do you want my values despite 10?
• Do you want a second markers values despite
  10?
• Do these values suggest you really understand
  the questions?
• Should I give you marks for just trying?
• What standards should I apply?
• And what about criteria for pieces of work?

7
7 Principlesof Good Practice in Education

• 1 Encourages contacts between students and tutors
• 2 Develops reciprocity and cooperation among
  students
• 3 Uses active learning technologies
• 4 Gives prompt and appropriate feedback
• 5 Emphasises time on task
• 6 Communicates high expectations
• 7 Respects diverse talents and ways of learning

(Chickering and Ehrmann 1996)
8
6 Competenciesstudents need to gain
Marcia Mentkowski
Mihály Csíkszentmihályi

• Competence encouragement by challenge and
  remarks to achieve skills levels
• Confidence promoting remarks to show
  themselves, and others, their achievements
• Critical thinking which is what we have been
  wanting all along in 'Thinking skills, used in
  problem solving
• Creativity in what students do and how they do
  it
• Collaboration bringing in team-working and
  ethics
• Commonality of purpose, to achieve specified
  (and unspecified) objectives
• Curtiosity which is more than curiosity.

9
Itiel Dror(University of Southampton, now
Cognitive Consultants International)
3 Controls on Educational Implementation

• Control handing appropriate responsibilities to
  students
• Challenge student abilities
• Commitment student commitment to learning
  opportunities

What might be the best ways (note plural)
to develop these? But our usual question is not
this.
Are these competencies, abilities and
responsibilities measurable?
Using cognitive psychology might be a help
10
The Ten Principles of Assessment and Feedback

• Help clarify what good performance is
• via aims, criteria, standards
• 2. Encourage time and effort on challenging
  learning tasks.
• 3. Deliver high quality feedback information that
  helps learners self-correct.
• 4. Encourage positive motivational beliefs and
  self-esteem.
• 5. Encourage interaction and dialogue around
  learning (peer and teacher student)

(Nicol 2007)
11
10 principles continued

• 6. Facilitate the development of self-assessment
  and reflection in learning.
• 7. Give learners choice in assessment content
  and processes.
• 8. Involve students in decision-making about
  assessment policy and practice.
• 9. Support the development of learning
  communities.
• 10. Help teachers adapt teaching to student needs.

How do we use exams (and coursework) to deliver
these?
12
What actually is deep learning

• Something that is promoted by the 7P, 6C, 3C, 10P
• Experiential?
• Thought-exchanging (dialogue)
• Game-playing
• Exemplar-exploring
• Art-expressing
• Problem-solving
• (Using the 200 hours effectively)
• Something that is assessed in ways other than
  unseen examinations?

13
Assessment - whats it for?

• Quick post-it storm (nerd-storm)
• What do you think assessment is for?
• As opposed to what it is
• How do we measure (if we are to give marks)?
• Is it measurable (adequately)?

In particular If we promote Deep Learning how
do we produce Deep Assessment?
14
Criterion Referenced Assessment

• Provide criteria for each piece of assessed work
• Avoid the connoisseur approach (I know this is
  57)
• Show students what is needed to accomplish tasks
• Provide remarks with the marks
• Marks and remarks go hand in hand
• Give a sense of achievement
• Thus, criteria need to be assignment specific and
  so
• Link all this back to Just in Time Teaching
  (JITT)
• and After Action Reviews (AAR)
• Providing proper feedback and feed-forward

15
External Examiners may say(after they have
agreed about the marking)

• You over-assess
• what does this actually mean?
• You have too many pieces of assessment
• Why are there so many poor 2nd (3rd) answers in
  the scripts?
• You do not have enough gt 60
• Your fail rate is too high

What they dont (usually) ask How do you provide
feedback in this module How do students benefit
from feedback? How do you assess your learning
outcomes? How do you assess the skills stated for
the module?
16
Metacognitionreflecting on ones own knowledge
base as well as that of students

• What is the meaning of the musical term
  allegro?
• A. loud, B. soft, C. quick, D. slow.
• 2. In Greek Myth, who is the God of wines?
• A. Demeter, B. Hestia, C. Artemis, D. Dionysus.
• 3. How many years was Margaret Thatcher Prime
  Minister?
• A. 9, B. 10, C. 11, D. 12.
• 4. What was Lady Chatterleys first name?
• A. Eleanor, B. Constance, C. Margaret, D.
  Elizabeth.
• 5. What year was the Lockerbie disaster?
• A. 1986, B. 1987, C. 1988, D. 1989.
• 6. Londons population (2001) is approximately,
  in millions
• A. Seven, B. Eight, C. Nine, D. Ten

In the questions about to be revealed you have to
answer as you think appropriate A, B, C, or
D BUT you will be awarded marks as
follows Correct answer 1 Incorrect -2 Unanswer
ed 0 So you may want to pass However,
later, you will have a further chance to gain
marks
17
And a few more

• 7. What is the most common element on Earth ?
• A. Oxygen, B. Nitrogen, C. Iron, D. Hydrogen.
• 8. There is no Nobel Prize in?
• A. Physics, B. Chemistry, C. Biology, D.
  Medicine.
• 9. What year was John F Kennedy shot?
• A.1962, B. 1963, C. 1964, D. 1966.
• 10. What countries does Luxembourg have borders
  with ?
• A. Belgium, Holland, Germany
• B. Germany, Switzerland, Belgium
• C. Holland, Belgium, France
• D. Belgium, France, Germany.
• 11. What was the last album the Beatles recorded
  together?
• A. Let It Be, B. Rubber Soul, C. Magical Mystery
  Tour, D. Abbey Road

18
To mark

• When you have finished round one
• Have another go for those you did not answer,
  this time, guess, youll get 1 for correct and 0
  for incorrect. (use a second column, different
  colour pen)
• Here are the answers mark accordingly to the two
  rule sets (1,0,-2) (1,0)
• Round 1 Negative suggests 1/3 answers were wrong
  - you overestimated your knowledge
• Round 2 gt25 correct suggests you should have
  answered more in R1 - you didnt know you knew
  the answers!

19
MCQ or Certainty?

• Are MCQs just glorified University Challenges?
• Useful for focusing expectations rather than (or
  just than) knowledge per se?
• Use them as preflights to indicate knowledge/or
  understanding - ie formative rather than
  summative.

20
Using metacognition

• Now we have learned something about your
  cognitive abilities - and received some remarks
  about the processes
• How do we apply this to students?
• Better to give MCQs before you embark on some
  task to show students what they know and what
  they need to know
• Tony Gardner-Medwin and confidence-based
  assessment (rather than just MCQs)
• Students need to know the rules applied
• (in general not just MCQs)
• The delayed knowledge effect (sub-conscious)
• EVEN!
• Should exams be set well before the end of the
  semester so students can get remarks with their
  marks.

21
Rewarding the experience

• If students have problems with a troublesome
  concept or fail to grasp tacit knowledge do we
  penalise them?
• How do we provide scaffolding in the learning
  experience? (The right amount of scaffolding)
• If deep learning is experiential, then how do we
  reward it?
• How many marks do we give for just having done
  something? (Attainment)
• I dont think an essay or MCQ scores well
• Report writing is better
• Use criteria referencing
• Treat exams as problem solving exercises

22
Towards a proper educational feedback system

• What do I mean by this?
• Forget about Kolb cycles and Cowan loops or,
  perhaps better, use them to give a system that
  works for all students (and staff)
• Construct (with Alignment) a system (for a
  module) that maximises learning and understanding
  
• (rather than memory recall)
• To produce proper alignment we really need more
  specific instructions per task/activity
• And to provide remarks with the marks is use a
  feedback in an educational control system

23
Feedback and Feedforward in a control system
sense

Do exams provide a reference input (or even
output?) A MCQ might do this. Criterion
Referencing might too.
24
Educational ramp up

• To raise the level of attainment
• We need controlled input, ramping up.
• Output, achievement, results

Semester start
Semester end
Ee
Marks, no remarks for Exam events
25
Can I be more constructive?
How do we increase the gradient?
Task Activity Marks Remarks
Two Activity c/work
Two Activity c/work exam
Gradient learning rate
26
Delays and Troublesome Knowledge Sticking Points

• Troublesome knowledge (Land)
• Tacit knowledge (Polanyi)
• Delays doing activities Sticking Points
• Preflights (Just-in-Time Teaching G Novak)
• Feedforward (controlled use of advance info)
• After Action Reviews
• ve Feedback or remarks

27
Task
Task
Problem specified
Task
Manipulation of system
Problem solution
S
Assessment
T
R
R
Rules System Examples Procedures
Clues Guidance Help
Provision of Feedback on solutions
Support
Support
Resources
T Tacit knowledge required
S Sticking point(s) likely
Fig 5.1 A temporal sequence describing a
rule-based learning design, In Oliver et al.
2007. Describing ICT-based learning designs that
promote quality learning outcomes
28
What drives learning

• Not the assessment (itself)
• This (best) drives checking (controller in the
  system). Its an indicator
• Driving learning (ve feedback)
• Internal (student doing the activity)
• External (comments from others, peers, tutor,
  manuals)
• Inhibitors to learning (-ve feedback)
• No remarks (feedback)
• Exams with no remarks (feedback) before the end
  of the module

29
If we believe in these principles of deep
learningthen (in general)

• Why do we set essay exams?
• and seem to rely on closed exams?
• and not used open book or seen?
• Why do we set exams at the end of modules?
• Why do we not give feedback on exams
• Why do we believe MCQs are ok better?
• Why there are several cetls involved with active
  learning etc but few with Assessment?
• and you can probably think of more

30
Some possible conclusions

• Students learn best by experience
• Build this into assessment
• Ask when, and how much, assessment is needed
• Start by designing the feedback which reinforces
  the education
• Use the best-fitted assessment method
• Do students know the criteria?
• Get students (esp peer viewing/discussion) to
  look at criteria and then the answer(s)
• Give exams (and remarks) before the end of the
  module
• Fit these into the 10 Principles (etc)

31
Gibbs and Simpsons (4) Conditions

• Conditions under which assessment supports
  students learning
• 11 conditions of which 4
• 1. Capture sufficient study time and effort
• (in and out of class)
• 2. Are spread evenly across topics and weeks
• 3. Lead to productive learning activity
• (deep learning)
• 4. Communicate clear and high expectations

32
How might we use exams?

• Set questions which relate to problem solving
  (rather than being able to answer them by
  memorising)
• This might be solving an actual problem
• By providing checks or reinforcements of what has
  been experienced
• if not experientially learned
• By using criterion referencing more
• Ie by showing students what we expect
• which is what (tends to) happen in e.g.
  physics, maths, chemistry and engineering.

33
Finally!

• In the light of what I have been talking about,
• Do you consider that you need to reconsider your
  (first year) exam process?
• Was the discussion helpful?

34
Exams Jeremy? but probably not as you knew them
b.whalley_at_qub.ac.uk