Evaluation for Learning

1
Evaluation for Learning

• Paul Ramsden
• University of Sydney

2
The problem
To evaluate university teaching effectively, we
should apply the principles of good practice in
assessing student learning.
Sounds elementary...
3
Chapter 2 has demonstrated that even the best of
current practices are by and large not good
practice... Lewis Elton and Brenda
Johnston Assessment in universities a critical
review (http//www.ltsn.ac.uk/genericcentre/docs/
Critical20review20of20assessment
20research.rtf)
4

Source McInnis et al, 2000
5
There are no practice exercises every piece of
work is assessed, which tends to focus me on
attaining marks rather than exploring
ideas Assessment and marking have not been
good. Its quite subjective, and group work
assignments are sometimes unfair in their
assessment criteria (for example, everyone gets
the same mark even though one person may do less
work than another) The amount of multiple
choice questions in some subjects does not
provide opportunities to show how much you have
learnt and understood. We need more feedback
during semester. It is not reasonable to get a
mark at the end of the year and have no idea what
it is based on. It is also not ideal to front up
to an exam not knowing how well we have been
going in earlier assessments.
6
Bad practice remains common

• Multiple purposes, same techniques
• No formative assessment at all
• No match to learning objectives
• Feedback too little, too late
• Narrow range of methods unreflectively chosen
• No student choice
• Incorrect use of group assessment
• Ignorance of Heisenberg!
• Standards assumed to depend on norm referencing

7
Bad practice remains common

• Interaction reliability x validity not
  understood
• Unreliable assessments weighted less
• Limited application of grade descriptors
• Use of MCQs without professional training
• Plagiarism opportunities not designed out
  focus on technical fixes

8

• Underlying all this is...
• a focus on the producers concerns
• a continuing emphasis on teaching (rather than
  learning)
• a remarkable neglect of existing evidence
  (people seem to prefer dissemination and
  projects despite overwhelming proof that it
  doesnt work)
• and a tendency to blame the student
• We should try adapting these conclusions to the
  evaluation (and management) of university
  teaching.

9
(No Transcript)
10
Sydney 1999-2003

• Assignment
• Design a system to evaluate teaching which leads
  to a better student experience and improved
  learning outcomes.

and make it work.
11
The response

• 1. Break the problem into manageable parts
• 2. Design systems that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

12
The response

• 1. Break the problem into manageable parts
• 2. Design systems that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

13
Break the problem into manageable parts

• Make goals, methods, measurement and outcomes
  cohere ( aka alignment)
• Recognise importance of perceptions/ theories in
  use
• Design the system around the culture (avoid
  one-size-fits-all solutions)
• Learn from others mistakes

14
The response

• 1. Break the problem into manageable parts
• 2. Use strategies that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

15
Principles

• Adopt a single, but flexible, SAL perspective,
  derived from research evidence. Use it to inform
  every policy and process
• Plan for coherence between collegial and
  managerial strategies
• Use an evidence-based approach to change and
  leadership, aligned with academic values

16
Mechanisms to leverage strengths

• Academic Board reviews
• Rigorous, peer review-driven QA process
• Manage teaching proactively
• Funding drivers aligned with research measures
  and national indicators
• Plans that work
• New role for academic development unit
• Strategic projects

17
Management of teaching

• Teaching Dividend currently 4.5M
• Scholarship Index c. 650,000
• Teaching Improvement Fund 1.3M
• Required TL plans
• Annually updated operational plans
• Assess progress against targets
• Condition of access to performance-based funds
• Interrogated in Academic Board reviews

18
(No Transcript)
19
Strategic projects

• Re-engineered academic development unit
• First year experience learning community
• Expansion of training opportunities
• including mandatory 21 hour training
• Research-led teaching, including PIs
• Evaluation and QA working group
• Graduate attributes for a research university
• Research supervision initiatives

20
The response

• 1. Break the problem into manageable parts
• 2. Use strategies that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

21
Align evaluation with desired outcomes

• Rewards and recognition at multiple levels
• SI rewards trained staff and scholarly outputs
• Array of student-focused evaluation instruments,
  consistent with SAL theory
• New teaching awards
• Performance linked to funding at Faculty level
• Material support for changes
• New promotions policy

22
Evidence-based academic promotions criteria

• 1. Fundamentals
• 2. Criteria
• Performance
• Research-led
• Student-focused
• Scholarship
• Leadership
• 3. Evidence

23
Fundamentals
Fundamental things have got to be simple we must
look for simplicity in the system first. Ernest
Rutherford
24
Fundamentals

• Interest and explanation
• Respect for students
• Appropriate assessment
• Clear goals and challenge
• Independence student control
• Learning from students
• (Learning to Teach in Higher Education, Chapter 6)

25
Even more fundamental...

• Positive attitude towards students
• Ability to communicate well
• Lively interest in improving teaching

26
And plainer still...

• The aim of teaching is simple it is to make
• student learning possible.

27
Performance
A lecturer should appear easy and collected,
undaunted and unconcerned, his thoughts about him
and his mind clear for the contemplation and
description of his subject His whole behaviour
should evince a respect for his audience Michael
Faraday
28
Performance

• Planning (e.g. effective subject design, clear
  objectives)
• Process (e.g. presentation technique, WebCT
  design)
• Assessment (e.g. use of variety of appropriate
  methods)
• Outcomes (some evidence of link to learning)
• Evaluation (some evidence of use of evaluation
  to improve)

29
Research-led teaching
This atmosphere of excitement, arising from
imaginative consideration of knowledge,
transforms knowledge. A. N. Whitehead
30
Research-led teaching

• Imagination and enthusiasm a shared journey to
  understanding rather than delivery of
  content
• Effective design of curricula to engage students
  in inquiry
• Materials make use of primary sources, recent
  discoveries, progress in field
• (If you cant explain it to the charlady, you
  dont know anything about it)

31
Student-focused teaching
The two secrets of lecturing from which
everything else follows first, to believe that
you have something worth telling your audience
second, to imagine yourself as one of that
audience. R.V. Jones
32
Student-focused teaching

• Use of evaluation evidence to redesign
  curriculum
• Use of assessment data to modify teaching
  strategy
• Focus on relation between students and subject
  matter
• Choice of technique reflects level of student
  knowledge
• (From Did I make the goals clear? to Are the
  goals clear to the students?)

33
Scholarship in teaching
What is needed is for teachers in higher
education to bring to their teaching activities
the same critical, doubting and creative attitude
which they bring habitually to their research
activities. Lewis Elton
34
Scholarship in teaching

• Systematic use of best available evidence to
  select and deploy teaching and assessment
  strategies
• Publication of refereed journal articles on
  university teaching in discipline
• Invitations to address international conferences
  on university teaching

35
Leadership in teaching
She successfully inspired us to transform the
course and to re-focus on our students. She
melded a diverse group of academics into a team
of great teachers. A lecturer
36
Leadership in teaching

• Policy development and implementation
• Successful re-design and coordination of
  courses team leadership in teaching
• Mentoring of junior academics as teachers
• Application of teaching strategies and
  curriculum designs in other institutions
• Coordination of benchmarking activity with other
  universities

37
Criteria are hierarchically ordered...

• Non-negotiable basis Performance
• Second level Research-led
• Third level Student-focused
• Fourth level Scholarship
• Fifth level Leadership
• leading to a structure that can be mapped on to
  promotion at different levels.

38
And the evidence?

• Are the basics in place?
• Use multiple sources (never rely on student
  evaluations alone)
• Evaluate teaching like research
• Use peer review if possible
• Use hard data when available (e.g. S of T
  publications)
• Do the different sources tell a similar story?
• Do the claims made by the teacher match the
  evidence?

39
Required
Administrative
Academic
Recognised and encouraged
40
Required

• Fundamental values
• research intensive
• academic-led
• self-regulation
• evidence base
• international
• referencing
• focus on student
• experience

Administrative
Academic
Recognised and encouraged
41
Required
Academic Board reviews (self-evaluation, visit,
report) Policies on teaching evaluation,
assessment, ICT QA, promotions
Surveys TPIs and performance funding External QA
benchmarks Required training in teaching Teaching
Learning Plans
Administrative
Academic
University teaching awards Supervision
awards Teaching Improvement Fund Scholarship
Index Research-led teaching (policy and
indicators)
Guidelines for Good Practice (teaching,
learning with ICT) ITL courses and support
groups quality, graduate attributes, first year
experience, research-led teaching
Recognised and encouraged
42
The response

• 1. Identify the problem
• 2. Use strategies that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

43
Benchmarks

• Oxford Student surveys (SCEQ), QA policies
• Lund QA policies
• UCL QA policies
• OU ICT evaluation and QA
• ANU ICT in TL for research universities
• Monash Research-led teaching PIs
• Queensland Student surveys (SCEQ)
• Hong Kong Academic development standards

44
The response

• 1. Identify the problem
• 2. Use strategies that resonate with values and
  leverage strengths
• 3. Align evaluation with outcomes
• 4. Benchmark good practice
• 5. Test the impact against the evidence

45
(No Transcript)
46
Demand indicator for high quality
students (percentage of offers to students with
UAIs 95 or greater 1999-2003)
47
Changes in the Sydney first year experience,
1999-2002 1999 2002 change Teaching
staff give helpful feedback 39 48
9 Teamwork skills developed 48 58
10 Motivated to do best work 38 48
10 More confident to tackle new problems 47
54 7 IT supports my learning 56 64
8 Problem solving skills developed 52 58
6 Feel part of a learning community 39 53
14 Satisfied with dept/faculty admin. 51 68
17 Overall course satisfaction 66 71
5 Overall satisfaction (services admin) 57
65 8 Percentage agreements, annual survey
48

49

50
Oxford vs. Sydney, 3rd year Undergraduates (broad
agreement)
51
Obstacles to an evidence-based approach

• Rationality in an audit society (Smith,
  J.R.Statist.Soc. 1996)
• EBM is an example of a very uncommon phenomenon
• Motivation whats in it for me?
• Focus on teaching and methods
• Antediluvian attitude to staff development
• yet more projects, action research,
  dissemination, skills, centralist policies .
  itself not evidence-based
• Management inadequacies

52
Advice to the National Institute for L T

• Staff development dissemination is the
  front end only management structures are
  required
• Design for your users focus on learning have
  a bold vision capture academic imaginations
  through trust and credible leadership
• Celebrate diversity in the sector be inclusive
• Operate at multiple levels use a systems
  perspective
• Forget about accreditation
• Make it easy to share good practice
• Dont become an arm of the audit society

53
Further reading (1) Management of TL, teaching
quality Paul Ramsden Learning to Teach in
Higher Education Second Edition 2003 Foreword
by Sir David Watson London RoutledgeFalmer  th
e classic text fully revised and
updated (2) Practical advice for heads Paul
Ramsden Learning to Lead in Higher Education
1998 London RoutledgeFalmer www.routledgefalmer.
com