Evaluation The 3 musketeers and Ivan Moore

1
EvaluationThe 3 musketeers and Ivan Moore

• Drawing on the work of Ivan Moore (educational
  consultant, University of Manchester)

2
Why evaluate?

• What do we know about the role of evaluation in
  teaching and learning?
• What kinds of evaluation do we currently use?

3
The purposes of evaluation

• Evaluation for accountability (measuring results
  or efficiency)
• Evaluation for development (providing information
  to help to improve practice)
• Evaluation for knowledge (to obtain a deeper
  understanding of some particular area of practice
  such as student learning or change management)
• Chelimsky (1997)

• Summative
• Formative
• Research

4
Principles of evaluation

• An integral part of our teaching practice
• An ongoing process, so that we learn from
  systematic reflection
• Should be participatory
• Should enable us to make appropriate
  modifications along the way
• Should enable us to make judgments on specific
  sessions, but also to draw out wider implications

5
Useful evaluation resources

• http//www.icbl.hw.ac.uk/ltdi/cookbook/contents.ht
  ml
• http//www.materials.ac.uk/guides/evaluating.asp

6
Evaluation To Date

• There have been a number of different evaluations
  used to date
• Discuss in groups of 3 the relative merits for
  each type of elevation and the value of the
  information provided (remember PIE)
• Prepare to feedback to the group

7
When should we evaluate?

• At the beginning of a course
• Uncover students' motivations, expectations,
  skills and abilities in order to take account of
  them in course design and teaching practice
• During the course
• to focus on your student's learning, identify
  gaps, provide support materials and adjust
  teaching in response to feedback
• At the end of the course
• to inform any redesign for the next running of
  the course
• All 3 can be used or a combinaition

8
Purposes of evaluation

• Mike Prosser
• Quality Assurance
• student satisfaction
• the mean score is important as a measure of
  quality
• Quality Enhancement
• Student conceptions/how they experience the
  course
• The deviation is important more focused view

9

• Mike Prosser
• Is the learning environment/teaching approach
  having any influence on student
  conceptions/approaches?
• A student experience survey is more important
  than a student satisfaction survey

10
Approaches

• Off the peg questionnaires
• Learning Styles (Honey and Mumford)
• Motivated Strategies for Learning
• Learning and Study Strategies Inventory (LASSI)
• Approaches to Study Inventory
• Course Experience Questionnaire (Ramsden)

11
The Course experience questionnaire(P. Ramsden)

• Designed as a performance indicator
• 24 statements relating to 5 aspects
• 1 overall satisfaction statement
• Research-based
• Drawn from statements made by students in
  interviews
• Students with positive responses take a deep
  approach

12
The five sub-scales

• Good teaching
• Clear goals and standards
• Appropriate assessment
• Appropriate workload
• Generic skills

13
Designing an evaluation questionnaire

• In groups 30 minutes
• Outcomes based design
• Agree up to 4 goals/objectives for your group
• For each, design 4 statements that will help to
  determine if the goal is being achieved
• Record the goals and statements on a flip chart

14
Evaluation exercise

• Design an evaluation plan for the course which
  you have designed. Include an account of the
  specific areas on which you would like feedback,
  the resources that you will need and the
  timescale for the evaluation.
• Reflect on how your think this evaluation plan
  would aid you in planning improvements in the
  course.

15
Goal-free evaluation

• In pairs
• Choose another pair on which to focus
• Devise up to 5 questions that you might ask
  students to answer that might provide information
  on what they are experiencing
• Record your questions
• Compare them with the other pair
• Discuss what information you might receive from
  these questions
• Draw up your conclusions for feedback

16
Context-free evaluation

• Since the beginning of the year
• What skills, if any, have you developed?
• What helped you to develop these skills?
• How have you changed the way you study?
• What prompted you to make these changes?

17
Other methods

• Focus groups
• Structured interviews
• Continuous feedback
• What did you find most difficult/confusing today?
• What things did you find helped you learn last
  week?
• What should I
• Start/stop/continue?

18
The minute paper Classroom Assessment Technique
(Angelo and Cross 1992) (my favourite!!)

• What was the most useful of meaningful thing you
  learned during this session?
• What question(s) remain uppermost in your mind as
  we end this session?
• What was the muddiest point in this session?
• OR
• What would you like me to stop doing?
• What would you like me to start doing?
• What would you like me to continue doing?

19
Other instruments

• Kolbs Learning Style Inventory (LSI)
• Honey and Mumfords LSQ
• Sternbergs thinking styles
• Felders Index of Learning Styles
• Weinsteins Learning and Study Strategies
  Inventory (LASSI)
• Entwistles Approaches to Study Inventory