A Fresh Look at Assessment

1
Smarter Feedback and Feed-Forward Giving more
feedback, of better value, to more students, in
less time!
Queens University, Belfast
Phil Race BSc PhD PGCE FCIPD SFHEA NTF Visiting
Professor, assessment, learning and
teaching Leeds Metropolitan University Tuesday,
10 November 2009
2
Rationale

• This workshop is about making feedback work for
  us and for our students. The National Student
  Survey of 2005-7 shows that the areas where
  students are least satisfied with their
  experience of higher education are feedback and
  assessment. In particular, they dont seem to get
  enough formative feedback, and it is not helping
  them as much as it should. Whats wrong with
  formative feedback? It can be too late. It can
  demotivate students instead of motivating them.
  It can take too much of our time, and yet
  students may take little notice of it. Too often,
  it can be feedback only, rather than also being
  feed-forward. This workshop aims to work out how
  we can give more and better feedback to more
  students in less time! Not least, we need to
  step back and change some of our systems to
  address the problems we have with managing
  feedback to students.

3
Exploring feedback to students

• In this workshop, well review a wide range of
  feedback methods, so that you can make your
  feedback more effective, less time-consuming, and
  more suitable to evidence the high quality of
  your teaching to those looking for such evidence.
  
• We will explore the pros and cons of paper-based,
  word-processed, and electronically delivered
  feedback, in terms of the learning payoff which
  students derive from feedback, and the efficiency
  for us in providing it for them. We will also
  explore how best we can find out from our
  students how our feedback is working for them
  they know.

4
Intended learning outcomes

• After participating in this workshop, you should
  be able to
• Regard feedback to students as the lifeblood of
  successful learning
• Increase the learning payoff your feedback
  delivers, while reducing the burden of the time
  it takes you to give students this feedback
• See how we can give students useful feedback
  within 24 hours of them doing some work, and mark
  their work in a third of the time it used to take
  us!
• Find out from your students more about how your
  feedback is working for them.

5
Reference materials

• Race, P and Pickford, R (2007) Making teaching
  Work London Sage Publications.
• Race, P (2006) The Lecturers Toolkit 3rd
  edition London Routledge.
• Race P (2005) Making learning happen London Sage
  Publications.
• (theres a compendium of writings on feedback
  as a download near the top of the downloads
  page on my website).
• Knight P and Yorke M (2003) Assessment, learning
  and employability Maidenhead, UK SRHE/Open
  University Press.
• Bowl M (2003) Non-traditional entrants to higher
  education they talk about people like me Stoke
  on Trent, UK, Trentham Books.
• See also http//www.bioscience.heacademy.ac.uk/ftp
  /newsletters/bulletin22.pdf
• (Ill put the main slides Ive used up on my
  website www.phil-race.com as soon as I can, and
  leave them there for a few days I dont believe
  in 3-per-page PowerPoint slides theres a short
  discussion about this on my website too).

6
Workshop outline programme

• 1400 What goes wrong with student feedback?
  Exercise giving feedback to my students would
  be much better for me if only I
• 1415 How to give better feedback to more students
  in less time! Short presentation followed by
  workshop exercises.
• 1500 Coffee break
• 1515 Getting feedback from students on how well
  our feedback is helping them.
• 1545 Using student self-assessment to give
  students feedback they want, as well as feedback
  they need.
• 1630 Workshop close.

7
Name labels

• Please write your first name, big and bold, on a
  coded sticky label, and stick it to your clothing
  (not onto a fabric which would be damaged).

?A3
Phil
8
Getting to know students

• Asking students to write their names on labels
  helps them to get to know each other, and helps
  us to know what they want to be called.
• When they feel theyre known to us as individuals
  (in a good way), theyre less likely to drop out
  especially in big groups in HE.
• Labels are cheap and environmental-friendly.
• The codes can be used to get students into
  different group configurations, or to quiz the
  whole class.

9
Introductory task

• On a post-it, please write your own short
  completion of
• giving feedback to my students would be much
  better for me if only I
• Please swap post-its until you dont know whose
  you have.
• If asked, read out with passion and drama whats
  on the post-it you now have.
• Finally, please stick all the post-its up as
  directed.

10
Using post-its to find out students real
intended learning outcomes
11
Post-its

• A small, equal opportunities,
• non-threatening space.
• Just about everyone is willing to jot something
  down on a post-it in answer to a question,
  whereas they may not offer a spoken answer to a
  question, or write responses on a blank sheet of
  paper.
• Post-its allow everyone the same opportunity to
  respond, including the quiet or shy students.
• Post-its can be swapped, and students can read
  out someone elses ideas, in the relative comfort
  of anonymity.

12
Finding out where a group is starting from

• Post-its are particularly useful for open-ended
  questions, such as economics would be much
  better for me if only I
• Responses can be posted on a flipchart or wall,
  and used as an exhibit.
• They can be photocopied and returned to students.
• Post-its can be a fast way of finding out what
  the real intended learning outcomes are for a
  group.
• They can also provide a measure of the learning
  incomes of the group.

13
Face-to-face communication

• Making all the channels work

14
Face-to-face one-to-one feedback activity

• Please work in pairs, moving around the room,
  talking to different people using the script
  which follows
• The script
• A Hello.
• B Hello.
• A You are late.
• B I know.
• Try to do it completely differently each time.

G
15
The power of face-to-face communication

• When explaining assessment criteria to students,
  and when linking these to evidence of achievement
  of the intended learning outcomes, we need to
  make the most of face-to-face whole group
  contexts and,,,
• Tone of voice
• Body language
• Facial expression
• Eye contact
• The chance to repeat things
• The chance to respond to puzzled looks
• Some things cant work nearly so well just on
  paper or on screens.

16
How not to give feedback

• Some actual examples!!

17
Mark 30

• This essay is abysmal. There is no evidence
  whatsoever that you (1) read and understood the
  assignment topic with due care, (2)read the
  accompanying guidelines, or (3) read anything at
  all on the debate between legal moralism and
  legal liberalism or any of the supplied notes
  either. Hence the mark for this long, rambling
  piece of work. Considerable work is required
  for a pass standard or higher to be obtained.

18
More

• I really can not pass this at all
• It completely fails to address the question
  topic
• It presents totally inadequate information
• It is basically an incoherent ramble.
• A waste of your time writing it, and mine reading
  it. You could do so much better with some real
  study of the material.

19
Feedback, and

• Feed-forward

20
Feedback (including feed forward) works well when
it

• Motivates students helps them to want to learn
• Helps students to identify what they need to do
  next
• Helps students to take action to improve their
  learning
• Helps students to make sense of what they are
  learning
• Helps them to make realistic evaluations of their
  own abilities and achievements
• Helps them to reflect on their past work in ways
  they can use to improve their future work.

Wanting/ Needing
Doing
Making sense
Feedback
21
Feedback works badly when it

• Saps students confidence
• Directs students activities in inappropriate
  directions
• Fails to articulate with learning outcomes
• Fails to relate clearly to evidence of
  achievement of the assessment criteria
• Relates only to what is easy to assess rather
  than what is at the heart of learning
• Focuses on failings rather than achievements.

22
Why does formative feedback cause us problems?

• If the issues surrounding giving feedback
  effectively and efficiently were straightforward,
  they would have been solved long ago!
• Nevertheless, a growing body of research
  indicates that giving effective feedback is the
  most powerfully positive thing we can do to
  foster and maintain student learning.
• When we talk to unhappy students, minimal or
  over-negative feedback is often the biggest
  bugbear and the NSS bears out that this is
  widespread at present.

23
Auditing assessment and feedback

• We know from the NSS that weve got a problem
  with assessment and feedback, and theres no
  better time to start tackling it than now.
• Theres too much summative assessment in our
  systems, and not enough opportunity for
  feed-forward.
• We need to streamline assessment, and make the
  feedback associated with assessment work better
  for ourselves and for our students.

24
Evidencing our good practice

• Feedback to students, when written (or
  word-processed) is one of the most accessible
  indicators of the quality of our teaching.
• Such evidence is looked at very thoroughly by
  external agencies, professional bodies, funding
  council, external examiners.
• And they usually talk to students, who know best
  how well (or badly) our feedback actually works.

25
Seven key questions about feedback

• What do we say to students about their work?
• How do we say it?
• Do they take any notice?
• How much does it help their learning payoff?
• How well does it relate to students evidence of
  achievement of the intended learning outcomes?
• How well does it help them to achieve their next
  learning outcomes?
• How efficient is it for us?

26
Life is too short to

• Spend time and energy writing feedback which
  wont actually be used by students (sometimes not
  even collected by them)
• Write feedback just for external examiners to
  see.
• Approach giving feedback only in the read-write
  dimension, when many students gain more from it
  through auditory, or visual, or kinaesthetic
  channels.
• (see Neil Flemings excellent (and free!) VARK
  work on www.vark-learn.com).

27

How to give more and better formative feedback to
more students in less time!
28
Feedback on paper

• Handwritten comments on returned assessed work
• Word-processed comments about assessed work, e.g.
  front sheets and so on
• Model answers
• Assignment return sheets
• Code letters or numbers written onto students
  work
• Word-processed overall reports on the work of a
  class

29
Face-to-face feedback

• Face-to-face to whole lecture groups
• Face-to-face to individual students
• Face-to-face to groups of students

30
E-feedback and more

• Emailed comments on students individual work
• Overall comments delivered electronically through
  a computer conference
• Computer-delivered feedback
• Peer-group feedback associated with
  peer-assessment
• Feedback you give students who have self-assessed
  their work.

31
Individual task on ways of giving students
feedback

• Think of the main ways you give feedback to your
  students. Think also of other ways they get
  feedback without you being involved.
• Privately, make a list of all these ways your
  students get feedback on their learning.
• Now in groups, write each feedback method on a
  separate post-it, and place the post-its on a
  flipchart as follows

32
High learning payoff for students
Highly efficient for us
Not highly efficient for us
Low learning payoff for students
33
5
25
High learning payoff
We need to be making better use of the
feedback processes in this quadrant
16
4
Highly efficient
Not highly efficient
3
2
4
5
1
2
4
Low learning payoff
We need to be making much less use of the
feedback processes in this quadrant
1
1
34
Queens University Belfast high and low scoring
feedback methods

• High scoring
• 25 prs
• 25 questioning getting discussion going
• 20 frequently asked questions on paper
• 20 immediate verbal feedback to small group
• 20 multiple choice using prs
• 16 peer feedback
• 16 electronic feedback

• Low scoring
• 3 written feedback comments on written work
• 2 progress committee discussion
• 2 late written feedback
• 1 written feedback assessment sheets on top of
  assignment when past caring
• 1 commented feedback sheets
• 1 exams

35
Birmingham City University high and low scoring
feedback methods
High scoring 25 one to one feedback from
practitioners in field, not lecturers 25
one-to-one oral feedback on practical skills 20
moodle formative feedback on online activities 20
written feedback on Moodle interaction 20
structured peer review exercise 20 self feedback
based on model answer 20 peer review 18 verbal
feedback on assessment 16 oral podcasting online
Low scoring 4 written feedback using electronic
form 4 formal electronic feedback 3 written
feedback sheets 3 paper copies of electronically
written feedback on assessment 1 written feedback
at the end of the module (ditto)
36
Kingston highest and lowest scoring feedback
methods

• High scoring
• 25 consumer feedback
• 25 wikis
• (e.g. patient)
• 25 in class peer feedback (ditto)
• 20 students giving verbal feedback on their own
  presentations
• 20 tick-box criteria
• 18 common mistakes
• 16 short online quiz
• 16 peer group
• 15 watching and commenting
• 14 Excel spreadsheet

• Low scoring
• 4 written feedback on assignments
• 4 telephone
• 3 oral feedback to individuals
• 1 written comments on work (ditto)

37
Essex feedback methods
High scoring 24 class discussion on essay
titles 23 peer feedback dependent on age, level,
subject 20 group tutorials and discussions 20
class discussions 20 small groups 20 self
assessment 16 critique 16 questions and answers
in class 16 online interactive something or other
Low scoring 9 emails 8 detailed comments on
essays 6 end of year exam mark 5 no feedback 5
comments on essays 5 office hours 3 handwritten
comments on essays 1 one to one tutorials
38
Brunel feedback methods
High scoring 25 set a task, then do it yourself
in front of them 20 Group tutorials 20
Computer-aided assessment 18 U-link discussion
groups 15 peer group assessment 15 outside
feedback e.g. friends, relatives 15 group
feedback oral
Low scoring 1 exams 1 written feedback 3 written
feedback on coursework 3 ad-hoc answering of
questions by email etc.
39
York St John feedback methods
High scoring 25 web ct tests 25
verbal/non-verbal feedback within teaching 24
webct discussion boards 23 looking at past work
from self 23 looking at past work from
students 20 post-its on feedback (not going
through all channels) 20 peer-feedback during
action learning 20 show-and-tell with families
and friends 15 group projects with peers 15 email
response to drafts
Low scoring 3 marginal notes feedback on essays 5
feedback on front-sheets 6 just a grade 10
one-to-one tutorials
40
NESCOT feedback methods
High scoring 25 video 22 one-to-one feedback
during practical sessions 18 putting model
answers on Blackboard afterwards 16 class group
feedback 16 playback of video presentations 12
class discussion
Low scoring 3 written feedback (ditto) 3.5 email
4 email to individual students 4 class list of
marks on notice board 6 email
41
Tyne Met feedback methods
High scoring 25 self-assessment with target
setting 25 Self assessment against criteria 25
peer feedback 22 self-reflection and
assessment 20 client comments on work 20 peer
feedback 16 peer assessment 16 group feedback
immediately after an assessment 16 Informal
feedback in monitoring in class
Low scoring 5 written feedback 5 empty praise,
e.g. youve all done very well 4 front sheets 2
summative feedback at the end of the course 1 end
of term reports
42
Curtin University feedback methods
High scoring 25 verbal instant feedback on
activities e.g. in prac class 25 summary feedback
to whole class 20 students mark each others work
providing feedback on marking scheme 20 next
class put up examples on screen 20 poster
presentation in tutorials 16 small group
feedback by tutor in seminar
Low scoring 9 quiz e.g. paper-based mcq 4 written
feedback on individuals work 3 as above 4
providing good and bad examples of work before it
is due 1 exams without feedback
43
South East Essex College feedback methods
High scoring 25 body language 25 peer
assessment, critique 25 timetabled tutorial
11 20 managed peer discussion 20 class
discussion 20 electronic feedback sheets 20
on-line guided learning and assessment 18
tutor-led group critique
Low scoring 5 written feedback on front sheets 1
written feedback on front sheets 1 written
feedback with grade 4 just a mark 1 written
assignment front sheets 6 comments on feedback
sheet 4 written feedback 4 email comments on
drafts
44
Kingston University feedback methods
High scoring 25 repeat exercises 25 service
users in simulations 25 group discussion 23 group
feedback from us to them 22 generic comments 20
one-to-one feedback in labs 20 products give
instant feedback 20 poster presentations 16
face-to-face between sessions 16 diagnostic
tests
Low scoring 5 just a grade 2.5 grade 2 written
comments 1 written comments 6 exam marks
45
Syddansk Universitet feedback methods
High scoring 25 feedback during active learning
during teaching 20 oral presentation with instant
feedback from teacher and class 20 peer review
students critiquing paper 15 students in
tutor-less groups reviewing their learning and
making summaries 12 discussion session and
solutions on Blackboard
Low scoring 4/3/2 normal written exam with mark
only
46
University of Central Lancashire feedback methods
High scoring 25 student evaluation where they
analyse and review their own work 20 gold
star One-to one meetings before handing in
assignment 20 one-to-one meetings at students
request 20 individual presentations with
immediate feedback 18 peers discussing their own
work 12 web-based feedback
Low scoring 2 written comments using codes
front sheet 1 cover sheets 4 email comments on
individual drafts 3 individual written comments
on text
47
De Montfort feedback methods
High scoring 25 visual and verbal feedback on
best practice from tutors one-to-many 25 students
talking to each other 25 work placement 20
several-to-one viva 20 common pitfalls and what a
good one looks like one to many 20 external
competitions 16 peer-assessment feedback
Low scoring 1 PDP 1-1tutorial 1 module evaluation
forms 4 individual critiques 4 general feedback
on blackboard 2 written feedback sheets in a box
5 generic written feedback sent out
48
University of Aalborg methods
High scoring 20 students discussing group
work 20 ditto 20 peer assessment 20 students
reviewing each other 16 peer assessment
Low scoring 2 individual oral examination 3
comments on assignment sheets 3 written feedback
on exercises 3 giving marks to exercises 5
face-to-face feedback, and written comments
49
Leeds Met feedback methods
High scoring 25 verbal feedback to whole
group 25 general feedback to the whole group with
individual comments 25 instant feedback by self
assessment 16 reflective journal self
evaluation 16 peer feedback 14 whole group verbal
feedback
Low scoring 1 written individual (ditto) 5 no
feedback
50
Teesside highest and lowest scores

• Highest
• 25 feedback from patients/clients
• 20 self-assessment
• 20 email
• 16 collaborative feedback with mentors in
  practice
• 16 face-to-face in groups
• 16 computer-based self-assessment
• 16 handwritten comments on draft work
• 16 handwritten sheets during presentations

• Lowest
• 2 model answers on electronic discussion board
• 4 written feedback on past exams
• 8 written comments in portfolios

51
Maynooth top and bottom scoring feedback methods

• 25 discussion
• 20 e-learning with instant feedback
• 20 individual peer assessment
• 16 supervised peer-assessment
• 16 Coded grades with email discussion

• 5 giving marks only
• 4 grades with oral correction
• 4 written exams

52
University of South Australia top and bottom
scoring feedback methods

• Top scoring methods
• 25 emails to group
• 25 peer assessment
• 20 moderated self assessment
• 20 group peer review
• 16 model answers
• 14 criterion based written feedback

• Bottom scoring methods
• 6 handwritten on assignments
• 5 single word comment
• 4 exams
• 3 just a mark
• 3 grabbing comments from a bank
• 3 Web conferencing

53
Getting feedback to students quickly

• The next two slides are about a way of giving
  students feedback on their work within 24 hours
  of them doing it.
• There are three or more yes, but what ifs with
  this idea, but please hold these for a couple of
  minutes.

54
Speeding up marking by giving feedback at the
time of submission

• This is a suggestion regarding how we can give
  very quick feedback to a large group of students
  at the point of them submitting their work for
  assessment.
• For example, collect their work for marking at
  the start of a whole-group session, then
  immediately issuing a pre-prepared 1-sheet
  handout to the class covering
• Illustration of what is expected as evidence of
  achievement of each of the intended learning
  outcomes
• Likely mistakes
• Features of a good answer
• Frequently needed explanations
• All the better if you can spend a few minutes
  de-briefing the whole group and talking them
  through the handout
• adding tone-of-voice, facial expression, body
  language, emphasis, and so on to the feedback.

55
How this saves time

• Since many students will have done the work in
  the last 24 hours before handing it in, youre
  giving them feedback while they still remember
  what they were doing.
• You then waste far less writing the same old
  things on one piece of work after another,
  regarding frequently occurring mistakes
• You can make your comments relate more to each
  individual piece of work
• This means when students get their marked work
  back with feedback, they are more likely to use
  it, as its personal to them.
• Also, theyve by then already had the chance to
  make sense of their own piece of work in the
  light of the generic feedback you gave them when
  they handed it in.

56
Feedback without marks

• Marks often destroy the value of our feedback to
  students!

57
Feedback versus marks or grades

• Feedback may be eclipsed by marks or grades.
• Students may be blinded by the mark or grade, and
  not even try to make sense of the feedback they
  receive.

58
Just a mark (or grade) is the least effective
form of feedback!

• Its what students look at first.
• If the mark is good, they smile and file quite
  often they dont even read the feedback.
• If its low, they frown and bin it very often
  without reading the feedback.

59
But there are ways round this...

• Give students back their work with feedback but
  with no marks or grades (keeping your record of
  their marks).
• Ask them to work out their marks from the
  feedback you have given them (and from the
  feedback you gave to others too).

60
Suggest that their marks will count!

• Tell them that if their self-assessment scores
  are within (say) 5 of your own scores, the
  higher number will go forward into their
  assessment record.
• But explain that you will talk individually to
  those students whose score is different by more
  than 5 from yours.

61
Get them to self-assess...

• Collect their scores or grades e.g. pass a
  sheet round in a lecture.
• Most students (e.g. 9 in 10) will be within the
  5.
• Arrange to talk to those where the difference is
  more than 5 or one grade.

62
Students who under-estimate their grade

• These students often need their esteem boosted.
• Remind them about the assessment criteria, and
  how these illustrate the intended standards
  associated with the learning outcomes.
• Check that they werent just trying not to be
  seen as over-confident.

63
Students who over-estimate their grade

• This usually indicates theyve got a blind spot.
• Talk them through their work, and find out
  exactly where theyve lost marks which they
  thought they had gained.
• Check with them that they can now see what was
  being looked for.

64
Developing this idea further

• Think about getting students to indicate their
  expected score or grade at the point of handing
  their work in for example on a self-assessment
  proforma.
• Think about getting students to work out how well
  they believe they have achieved each of the
  intended learning outcomes for the work
  concerned
• Or how well they believe they have met each of
  the assessment criteria for the work.

65
Building on Feedback

• Phil Race
• Visiting Professor Assessment, Learning and
  Teaching
• Leeds Metropolitan University

66
Helping students to build on feedback

• Adapted from How to Get a Good Degree
• (2nd edition) Phil Race, 2007, Maidenhead, Open
  University Press.

67
Advice to students (1)

• Look carefully at the comments and the aspects of
  your work they relate to, and try to work out for
  yourself exactly what is being commented upon,
  and how you may be able to respond to each
  feedback comment in your future work (especially
  future work for that particular tutor, or exam
  answers which may be marked by this person).

68
Advice to students (2)

• Compare the feedback comments you have received
  with those received by fellow-students on the
  course or module. Ask them what they think tutors
  mean by particular comments. This will at least
  give you some more information about the standard
  of your own work, and probably will make you feel
  better about your own work when you notice
  critical feedback comments on other students
  work which were not written on your work.
  However, there may still be some guesswork
  regarding what the tutors may actually mean by
  their comments.

69
Interpreting the feedback
70
Agreeing or not
71
Analysing the feedback
72
Planning future actions
73
Getting yet more feedback
74
Prioritising actions
75
Ten tips for formative feedback

• Help students to want feedback. Stop them
  concentrating on the mark.
• Get the timing right, while students still care
  about what theyve done.
• Make feedback interesting - not routine or
  mundane.
• Give at least some feedback very quickly.
• Put technology to work to save repetition and
  increase speed.

76
Ten tips for formative feedback

• Link feedback to the achievement of learning
  outcomes.
• Provide most feedback at the beginning
  scaffolding.
• Use feedback to clarify targets, standards and
  expectations.
• Ensure that students know that your feedback is
  about their work, not about them as people.
• Make full use of auditory, visual and
  kinaesthetic dont give feedback just in the
  read-write domain.

77
Back to our intended outcomes

• Do you now feel better able to
• 2 hands very much better, one hand somewhat
  better, no hands no better)
• After participating in this workshop, you should
  be able to
• Regard feedback to students as the lifeblood of
  successful learning?
• Increase the learning payoff your feedback
  delivers, while reducing the burden of the time
  it takes you to give students this feedback?
• See how we can give students useful feedback
  within 24 hours of them doing some work, and mark
  their work in a third of the time it used to take
  us!
• Find out from your students more about how your
  feedback is working for them?

78
Action planning statements

• One thing Im going to do is
• One idea Im taking away is
• Im going to think more about
• I have found out that
• Id like to know
• In future, Im not going to

Tuesday, 10 November 2009
79
Thank you www.Phil-Race.com e-mail
p.race_at_leedsmet.ac.uk