University of York

1

• University of York
• May 2008
• Using assessment
• to support student learning
• Graham Gibbs

2
Personal Background

• Open University 1975-80 and 1997-2003
• Top in National Student Survey, especially
  assessment and feedback ratings
• Oxford Brookes University 1980-1997
• Most coursework assessment
• Systematic course design
• University of Oxford 2004-
• Least coursework assessment
• Top in National Student Survey, especially
  assessment and feedback ratings
• No course design

3
Personal Background

• Practical books and articles about assessment
• 53 Interesting ways to assess your students
• Assessing Student Centred Courses
• Assessing Large Classes
• Consultancy to universities on strategic
  decisions about assessment policy
• Research into the impact of assessment on student
  learning
•  

4
Student experience of assessment
5

• I just dont bother doing the homework now. I
  approach the courses so I can get an A in the
  easiest manner, and its amazing how little work
  you have to do if you really dont like the
  course.

6

• I am positive there is an examination game. You
  dont learn certain facts, for instance, you
  dont take the whole course, you go and look at
  the examination papers and you say looks as
  though there have been four questions on a
  certain theme this year, last year the professor
  said that the examination would be much the same
  as before, so you excise a good bit of the
  course immediately

7

• The feedback on my assignments comes back so
  slowly that we are already on the topic after
  next and Ive already submitted the next
  assignment. I just look at the mark and throw it
  in the bin

8

• The tutor likes to see the right answer circled
  in red at the bottom of the problem sheet. He
  likes to think youve got it right first time.
  You dont include any workings or corrections
  you make it look perfect. The trouble is when you
  go back to it later you cant work out how you
  did it and you make the same mistakes all over
  again

9

• One course I tried to understand the material
  and failed the exam. When I took the resit I just
  concentrated on passing and got 98. My tutor
  couldnt understand how I failed the first time.
  I still dont understand the subject so it
  defeated the object, in a way

10

• I do not like the on-line assessment methodit
  was too easy to only study to answer the
  questions and still get a good mark the wrong
  reasoning can still result in the right answer so
  the student can be misled into thinking she
  understands something I think there should have
  been a tutor-marked assessment part way through
  the course so someone could comment on methods of
  working, layout etc.

11

• We were told this course was going to be an
  opportunity to be creative, to take risks. Then
  in week five we were hit with a multiple choice
  question test and we realised what it was really
  all about.

12
Summative assessment that is redundant

• Most students can, in their first year, predict
  their final results with some accuracy
• As few as 5 of assessments are necessary to
  produce the same overall grades

13
Assessment that improves learning

• The case of the Engineer
• The case of the Manager
• The case of the Pharmacist
• The case of the Psychologist
• The case of the Accountant

14
The case of the engineer

• Weekly lectures, problem sheets and classes
• Marking impossible
• Problem classes large enough to hide in
• Students didnt tackle the problems
• Exam marks 45

15
The case of the engineer

• Course requirement to complete 50 problems
• Peer assessed in six lecture slots
• Marks do not count
• Lectures, problems, classes, exams unchanged

16
The case of the engineer

• Course requirement to complete 50 problems
• Peer assessed in six lecture slots
• Marks do not count
• Lectures, problems, classes, exams unchanged
• Exam marks increased from 45 to 85
• Why did it work?

17
The case of the engineer

• time on task
• social learning and peer pressure
• timely and influential feedback
• learning by assessing
• error spotting
• developing judgement (internalisation of
  standards)
• self-supervision (meta-cognitive awareness)

18
Assessment that improves learning

• The case of the Engineer
• The case of the Manager
• The case of the Pharmacist
• The case of the Psychologist
• The case of the Accountant

19

• Conditions under which assessment
  supports student learning

20
Quantity and distribution of student effort

• 1 Assessed tasks capture sufficient student time
  and effort
• 2 These tasks distribute student effort evenly
  across topics and weeks

21
Quality and level of student effort

• 3 These tasks engage students in productive
  learning activity
• 4 Assessment communicates clear and high
  expectations to students

22
Quantity and timing of feedback

• 5 Sufficient feedback is provided, both often
  enough and in enough detail
• 6 The feedback is provided quickly enough to be
  useful to students

23
Quality of feedback

• 7 Feedback focuses on learning rather than on
  marks or students themselves
• 8 Feedback is understandable to students, given
  their sophistication

24
Student response to feedback

• 9 Feedback is received by students and attended
  to, and is acted upon by students to improve
  their work or their learning

25
Effective assessment tactics

• Bioscience poster reports
• Engineering sampling lab reports cheap
  feedback
• Law essay requirements sampling models
• Estates management project exams
• French Literature critiquing texts under
  examination conditions

26
Assessment Experience Questionnaire

• Measures extent to which the conditions are
  perceived to be met
• Quantity and distribution of effort
• Quality, quantity and timeliness of feedback
• Use of feedback
• Impact of exams on quality of learning
• Deep approach
• Surface approach
• Clarity of goals and standards
• Appropriateness of assessment

27
(No Transcript)
28
(No Transcript)
29
University A is the Open University

• 8 assignments per course
• Detailed written feedback on every assignment
• Quality assurance of feedback
• Less funding per student than any other
  university, mainly spent on feedback
• Best student ratings nationally
• Much pedagogic research
• Formative-only early assignments improve
  retention
• Computer-based assignments reduce retention and
  performance

30
then I went to Oxford

• Oxford responds in a limited way to most national
  quality assurance guidelines
• learning outcomes
• assessment criteria
• alignment of assessment with aims
• Oxford has not modernised its assessment
• reliance on examinations, little assessed
  coursework, little summative assessment of any
  kind, no modularisation

31
then I went to Oxford

• Oxford responds in a limited way to most national
  quality assurance guidelines
• learning outcomes
• assessment criteria
• alignment of assessment with aims
• Oxford has not modernised its assessment
• reliance on examinations, little assessed
  coursework, little assessment of any kind, no
  modularisation
• Outstanding quality of student experience at
  Oxford
• student retention of 98 (1st in UK)
• Oxford ranked 1st for teaching in UK (Times,
  Guardian)
• better CEQ scores than elsewhere in world
• better NSS ratings than the Open University

32
Research questions

• What are the characteristics of programme level
  assessment environments that are associated with
  positive student learning responses?
• Are the characteristics of programme level
  assessment environments that are most closely
  associated with positive student learning
  responses those that quality assurance
  regulations emphasise?

33
Research design

• Three contrasting universities (Oxford, pre-1992,
  post 1992)
• Three contrasting programmes in each (Humanities,
  Science, Applied Social Science)
• Characterise assessment environments
• Read documentation (all modules)
• Interview programme leader, lecturers and
  students
• Administer AEQ
• Explore relationships between characteristics of
  programme level assessment design and qualities
  of student learning
• with Harriet Dunbar-Goddet, Chris Rust and Sue
  Law
• funded by the Higher Education Academy

34
Coding characteristics of programme level
assessment environments

• marks from examinations
• Volume of summative assessment
• Volume of formative only assessment
• Volume of (formal) oral feedback
• Volume of written feedback
• Timeliness days after submission before feedback
  provided
• Explicitness of criteria and standards
• Alignment of goals and assessment

35
Coding characteristics of programme level
assessment environments

• marks from examinations
• High more than 70
• Med between 40 and 70
• Low less than 40

36
Coding characteristics of programme level
assessment environments

• Explicitness of criteria and standards
• High clear criteria for most assignments
  exams link to grades effort made to enable
  students to internalise criteria standards
• Low explicit criteria and standards rare and/or
  nebulously formulated marks/grades arrived at
  through global judgment in tacit way no effort
  to enable students to internalise criteria and
  standards

37
Range of characteristics of programme level
assessment environments
38
Range of characteristics of programme level
assessment environments

• marks from exams 17 - 100

39
Range of characteristics of programme level
assessment environments

• marks from exams 17 - 100
• number of times work marked 11 - 95

40
Range of characteristics of programme level
assessment environments

• marks from exams 17 - 100
• number of times work marked 11 - 95
• number of times formative-only assessment 2 -
  134

41
Range of characteristics of programme level
assessment environments

• marks from exams 17 - 100
• number of times work marked 11 - 95
• number of times formative-only assessment 2 -
  134
• number of hours of oral feedback 3 - 68

42
Institutional assessment environments
 
43
Patterns of assessment features within programmes

• every programme that is high on the volume of
  formative assessment is low on the volume of
  summative assessment
• no examples of high volume of summative
  assessment and high volume of feedback

44
Patterns of assessment features within programmes

• every programme that is low on the volume of
  summative assessment is high on the volume of
  formative assessment
• no examples of high volume of summative
  assessment and high volume of feedback
• there may be enough resources to mark student
  work many times, or to give feedback many times,
  but not enough resources to do both

45
Relationships between assessment characteristics
and student learning
46
Assessment characteristics and student learning
response 1

• When the level of explicitness of criteria and
  standards is high, students experience is
  characterised by
• Less coverage of the syllabus
• Less and poorer quality feedback
• Less use of feedback
• Less learning from the examination
• Less deep approach

47
Assessment characteristics and student learning
response 2

• When the level of alignment of goals and
  standards is high, students experience is
  characterised by
• Less coverage of the syllabus
• Less and poorer quality feedback
• Less use of feedback
• Less appropriate assessment
• Less clear goals and standards
• Less learning from the examination
• Less deep approach

48
Assessment characteristics and student learning
response 3

• When the level variety of assessment methods is
  high, students experience is characterised by
• Less and poorer quality feedback
• Less use of feedback
• Less appropriate assessment
• Less clear goals and standards
• Less learning from the examination
• Less deep approach
• More surface approach
• Less overall satisfaction

49
Assessment characteristics and student learning
response 4

• When the volume of formative-only assessment is
  high, students experience is characterised by
• More coverage of the syllabus
• More and better quality feedback
• More use of feedback
• More appropriate assessment
• More clear goals and standards
• More learning from the examination
• More deep approach
• More overall satisfaction

50
Assessment characteristics and student learning
response 5

• When the volume of oral feedback is high,
  students experience is characterised by
• More coverage of the syllabus
• More and better quality feedback
• More use of feedback
• More appropriate assessment
• More clear goals and standards
• More learning from the examination
• More deep approach
• More overall satisfaction

51
Assessment characteristics and student learning
response 6

• When the timeliness of feedback is high,
  students experience is characterised by
• More effort
• More coverage of the syllabus
• More and better quality feedback
• More use of feedback
• More appropriate assessment
• More clear goals and standards
• More learning from the examination

52
Summary

• Explicitness of criteria and standards, alignment
  of goals and assessment and variety of assessment
  are all associated with a negative learning
  experience
• they are also associated with more summative
  and less formative-only assessment, less oral
  feedback and less prompt feedback

53
Summary

• Explicitness of criteria and standards, alignment
  of goals and assessment and variety of assessment
  are all associated with a negative learning
  experience
• they are also associated with more summative
  and less formative-only assessment, less oral
  feedback and less prompt feedback
• Formative only assessment, oral feedback and
  prompt feedback are all associated with positive
  learning experience
• even when they are also associated with lack of
  explicitness of criteria and standards, lack of
  alignment of goals and assessment and a narrow
  range of assessment.

54
Why?

• being explicit does not result in students being
  clear about what they are supposed to be doing or
  what counts as high quality
• legitimate peripheral engagement in a community
  of practice (Lave and Wenger Price et al)

55
Why?

• Students experience very varied forms of
  assessment as confusing ambiguity and anxiety
  are associated with a surface approach
• Feedback improves learning most when there are no
  marks
• Possible to turn feedback round quickly when
  there are no QA worries about marks

56
alternative explanation A

• The features of assessment environments
  identified here that appear to have negative
  consequences for student learning are also the
  features that are associated with modular courses
  in which each separate module has to have
  self-contained assessment within a short time
  frame.
• Conclusion
• It may be modularity, rather than QA regimes,
  that have caused some of the problems. Oxbridge
  is not modular (the Open University isbut has
  huge and long modules that are usually studied
  one at a time)

57
alternative explanation B

• High volumes of assessed coursework have been
  introduced in part to increase student engagement
• Student engagement improves learning outcomes
• High of marks from coursework is associated
  with higher marks and better degrees (at post 92
  universities)
• However
• The effect of innovations that enhance engagement
  on learning outcomes only holds for low ability
  students (high ability students engage
  themselves) (Carini et al, in press)
• The present study did not control for student
  ability

58
Conclusions

• Assessment has more impact on how students go
  about studying, on the quantity and quality of
  their effort, and on their performance than
  does teaching
• It is relatively easy (and often cheap) to change
  student learning by changing assessment, provided
  the conditions are met effectively
• Whole universities have implicit conventions
  about what is acceptable in terms of assessment
  practice
• Some of these conventions are ill-informed and
  damaging and are built in to QA systems
• Local contexts are likely to require different
  assessment strategies to engage their students.

59
References

• Carini, R.M., Kuh, G.D. Klein, S.P. (in press)
  Student engagement and student learning testing
  the outcomes. Research in Higher Education
• Dunbar-Goddet, H. Gibbs, G. (under review) A
  methodology for evaluating the effects of
  programme assessment environments on student
  learning. European Association for Research into
  Learning and Instruction, Assessment Conference,
  Northumbria.
• Gibbs, G. (2002) Evaluation of the impact of
  formative assessment on student learning
  behaviour. European Association for Research
  into Learning and Instruction. Newcastle
  Northumbria University. August 2002.
• Gibbs, G. Simpson, C. (2003) Measuring the
  response of students to assessment the
  Assessment Experience Questionnaire. 11th
  International Improving Student Learning
  Symposium, Hinckley.
• Gibbs, G. Simpson, C. (2004) Conditions under
  which assessment supports student learning.
  Learning and Teaching in Higher Education 1,
  pp3-31.
• Gibbs, G., Simpson, C. Macdonald, R. (2003)
  Improving student learning through changing
  assessment a conceptual and practical
  framework. European Association for Research into
  Learning and Instruction Conference, Padova,
  Italy.