Using Learning Outcomes and Assessment Criteria

1
Using Learning Outcomes and Assessment Criteria

• Peter Noakes
• Department of Electronic Systems Engineering
• University of Essex

2
Purpose of this Talk

• To provide an introduction to topics and related
  background information that will help us to
  provide more concise module specifications
• To improve your efficiency in generating
  appropriate Learning Outcomes, Assessment Methods
  and Assessment Criteria for the modules you teach
• At the same time provide better transparency for
  students by making module specifications clearer
  and unambiguous
• and improve students commitment to self learning
  by clarifying what is expected of a student
• Consequently this should improve student
  progression and encourage better student retention

3
Structure of Talk

• Background
• Learning Outcomes
• Level Descriptors (QAA, SEEC, EPC)
• Assessment Methods
• Assessment Criteria
• An Example for You
• Report Back and Discussion

4
Reference Sources

• This presentation is based on the content of the
  following publications
• How to use Level Descriptors by Jenny Moon,
  SEEC, 2002
• How to Use Learning Outcomes and Assessment
  Criteria by David Gosling and Jenny Moon, SEEC,
  3rd Edition 2002
• Assessment A Guide for Lecturers by George
  Brown, LTSN Generic Centre, 2001
• Guide for Busy Academics, LTSN Generic Centre
• Southern England Consortium for Credit
  Accumulation and Transfer
• (http//www.seec-office.org.uk )

5
Background

• A Programme defines study or learning required to
  achieve an award or qualification
• A Programme Specification is required by the QAA
  for each award or qualification and defines the
  threshold learning outcomes for the programme
• A Programme comprises a number of Modules each of
  which is separately assessed and earns credit
  when successfully completed
• Using the outcomes model each Module Description
  defines the intended (threshold?) learning
  outcomes, the syllabus coverage and the
  assessment methods and criteria for the module.
• Achievement of Module Learning Outcome should
  contribute to a students satisfaction of the
  programme learning outcomes

6
Learning Outcome-based Model

• Traditionally an academic would first define the
  syllabus coverage, then develop how its taught
  and finally determine the method of assessing the
  students absorption of the material.
• The outcome-based model has three interconnected
  components
• An explicit statement of learning intent
  (intended learning outcome) which focuses on what
  the student is expected to know and be able to do
  by the end of the module, expressed in a form
  that permits their achievement to be demonstrated
  and measured
• The processes and resources to enable the
  outcomes to be achieved and demonstrated
  (curriculum, teaching, learning methods and
  materials, assessment and support and guidance
  methods)
• The criteria for assessing whether the intended
  learning outcomes have been achieved and for
  differentiating the performance of students.
• They are dependent on the level at which the
  module is targeted

7
Level / Qualification Descriptors?

• A Level is an indicator of the relative demand,
  complexity, depth of study and learner autonomy
• A Level (Qualification) Descriptor is a generic
  statement describing the characteristics and
  context of learning expected at each specific
  level.
• Module Learning Outcomes and Assessment Criteria
  are reviewed with respect to a level descriptor
  when developing a module and assigning credit at
  the appropriate level.
• QAA defines Qualification Descriptors in the
  Qualification Framework for Higher Education as
  Level C, Level I, Level H and Level M. These
  broadly correspond to Years 1, 2, 3 and first
  year postgraduate level respectively
• SEEC defines Level Descriptors as Level Zero,
  Level 1, Level 2, Level 3 and Masters Level.
  These broadly correspond to preliminary Year,
  Years 1, 2, 3 and first year postgraduate level
  respectively

8
Level Descriptors?

• Think of Level Descriptors as a means of
  communication about expectations of students
  study
• They are not rigid but developmental
• Student Learning is commonly described in terms
  of
• complexity of knowledge and understanding
• standard of cognitive skills
• key or transferable skills achieved
• the expected responsibility of the learner
• the autonomy or independence of the learner
• amount of guidance required by the learner
• Be careful of differences in the implied standard
  of learning!
• Is it defined for a threshold student, an average
  (or typical!) student or the best student?

9
Hierarchy of the Cognitive Domain

• Evaluation Ability to make a judgement of the
  worth of something
• Synthesis Ability to combine separate elements
  into a whole
• Analysis Ability to break a problem into its
  constituent parts and establish the
  relationships between each one
• Application Ability to apply rephrased knowledge
  in a novel situation
• Manipulation Ability to rephrase knowledge
• 1 Knowledge That which can be recalled
• Based on Blooms Taxonomy of Educational
  Objectives

10
Programme Learning Outcomes

• Guidance is provided by
• QAA Level H Descriptors and Benchmark Statements
  for
• Engineering
• Computing
• Biosciences
• SEECs generic HE Level 3 Definition
• Engineering Professors Conferences 26 Ability
  to Statements for Engineering Programmes
• Also see the accreditation requirements used by
  Professional Bodies (e.g. Engineering Councils
  SARTOR 97 or latest UK SPEC)

11
QAA General Honours Descriptors Level H
- A

• Students successfully completing programme
  requirements at this level will have
  demonstrated
• a systematic understanding of key aspects of
  their field of study, including acquisition of
  coherent and detailed knowledge, at least some of
  which is at or informed by, the forefront of
  defined aspects of a discipline
• an ability to deploy accurately established
  techniques of analysis and enquiry within a
  discipline
• conceptual understanding that enables the
  student
• to devise and sustain arguments, and/or to solve
  problems, using ideas and techniques, some of
  which are at the forefront of a discipline and
• to describe and comment upon particular aspects
  of current research, or equivalent advanced
  scholarship, in the discipline
• an appreciation of the uncertainty, ambiguity and
  limits of knowledge
• the ability to manage their own learning, and to
  make use of scholarly reviews and primary sources
  (e.g. refereed research articles and/or original
  materials appropriate to the discipline).

12
QAA General Honours Descriptors Level H
- B

• Typically, successful students at this level will
  be able to
• apply the methods and techniques that they have
  learned to review, consolidate, extend and apply
  their knowledge and understanding, and to
  initiate and carry out projects
• critically evaluate arguments, assumptions,
  abstract concepts and data (that may be
  incomplete), to make judgements, and to frame
  appropriate questions to achieve a solution - or
  identify a range of solutions - to a problem
• communicate information, ideas, problems, and
  solutions to both specialist and non-specialist
  audiences
• and will have
• qualities and transferable skills necessary for
  employment requiring
• the exercise of initiative and personal
  responsibility
• decision-making in complex and unpredictable
  contexts and
• the learning ability needed to undertake
  appropriate further training of a professional or
  equivalent nature.

13
SEEC Generic Level Descriptors Development of
Knowledge and Understanding
Knowledge Base
Ethical Issues
14
SEEC Generic Level Descriptors Cognitive /
Intellectual Skills - A
Analysis
Synthesis
15
SEEC Generic Level Descriptors Cognitive /
Intellectual Skills - B
Evaluation
Application
16
SEEC Generic Level Descriptors Key /
Transferable Skills - A
Group Working
Learning Resources
17
SEEC Generic Level Descriptors Key /
Transferable Skills - B
Self Evaluation
Information Mangement
18
SEEC Generic Level Descriptors Key /
Transferable Skills - C
Autonomy
Communications
19
SEEC Generic Level Descriptors Key /
Transferable Skills - D
Problem Solving
20
SEEC Generic Level Descriptors Practical Skills
Application of Skills
Autonomy in Skill use
21
Engineering Professors Conference EPC Generic
Ability to statements - A

• Ability to exercise Key Skills in the completion
  of engineering-related tasks at a level implied
  by the benchmarks associated with the following
  statements.
• Key Skills for engineering are Communication, IT,
  Application of Number, Working with Others,
  Problem Solving, Improving Own Learning and
  Performance.
• Ability to transform existing systems into
  conceptual models.
• This means the ability to
• Elicit and clarify client's true needs.
• Identify, classify and describe engineering
  systems.
• Define real target systems in terms of objective
  functions, performance specifications and other
  constraints (i.e. define the problem).
• Take account of risk assessment, and social and
  environmental impacts, in the setting of
  constraints (including legal, and health and
  safety issues).
• Select, review and experiment with existing
  engineering systems in order to obtain a database
  of knowledge and understanding that will
  contribute to the creation of specific real
  target systems.
• Resolve difficulties created by imperfect and
  incomplete information.
• Derive conceptual models of real target systems,
  identifying the key parameters.

22
Engineering Professors Conference EPC Generic
Ability to statements - B

• Ability to transform conceptual models into
  determinable models.
• This means the ability to
• Construct determinable models over a range of
  complexity to suit a range of conceptual models.
• Use mathematics and computing skills to create
  determinable models by deriving appropriate
  constitutive equations and specifying appropriate
  boundary conditions.
• Use industry standard software tools and
  platforms to set up determinable models.
• Recognise the value of Determinable Models of
  different complexity and the limitations of their
  application.
• Ability to use determinable models to obtain
  system specifications in terms of parametric
  values.
• This means the ability to
• Use mathematics and computing skills to
  manipulate and solve determinable models and use
  data sheets in an appropriate way to supplement
  solutions.
• Use industry standard software platforms and
  tools to solve determinable models.
• Carry out a parametric sensitive analysis.
• Critically assess results and, if inadequate or
  invalid, improve knowledge database by further
  reference to existing systems, and/or improve
  performance of determinable models.

23
Engineering Professors Conference EPC Generic
Ability to statements
- C

• Ability to select optimum specifications and
  create physical models.
• This means the ability to
• Use objective functions and constraints to
  identify optimum specifications.
• Plan physical modelling studies, based on
  determinable modelling, in order to produce
  critical information.
• Test and collate results, feeding these back into
  determinable models.
• Ability to apply the results from physical models
  to create real target systems.
• This means the ability to
• Write sufficiently detailed specifications of
  real target systems, including risk assessments
  and impact statements.
• Select production methods and write method
  statements.
• Implement production and deliver products fit for
  purpose, in a timely and efficient manner.
• Operate within relevant legislative frameworks.
• Ability to critically review real target systems
  and personal performance.
• This means the ability to
• Test and evaluate real systems in service against
  specification and client needs.
• Recognise and make critical judgements about
  related environmental, social, ethical and
  professional issues.
• Identify professional, technical and personal
  development needs and undertake appropriate
  training and independent research.

24
Working with Level Descriptors

• They should be seen as helpful guides rather than
  dictates
• They are generic and may contain sections not
  appropriate to a particular programme
• They may not cover all possible learning that is
  relevant to the programme.
• The words become more meaningful if you look at
  descriptors at the previous and next level
• Look at the relationship between descriptors at
  the same level as they do not function
  independently of each other
• Use them to provide an appropriate vocabulary to
  describe learning

25
A Module Specification from a Clean Sheet!
Generic Level Descriptors
Identify Aim of Module
Translate Level Descriptors into Subject
Descriptors
Develop the module and rethink it including the
initial learning outcomes
Write Learning Outcomes
Write Threshold and Grading Assessment Criteria
Develop Assessment Method to test achievement of
assessment criteria
Develop a teaching strategy to enable learners to
reach the learning outcomes / assessment criteria
26
Writing Module Specifications

• Clearly identify the intended level of the module
• Formulate clear and unambiguous intended
  threshold and possibly desirable learning
  outcomes for the module
• The threshold learning outcomes identify the
  essential learning to merit the award of the
  credits for this module
• Desirable learning outcomes can be included to
  provide guidance of learning above threshold
  which will be assessed to provide grading
• Identify assessment criteria that encourage
  learning at the appropriate level
• Threshold assessment criteria should specify how
  satisfactory performance of the threshold module
  learning outcomes can be demonstrated
• Grading-related assessment criteria are used to
  provide incentive for higher achievement above
  threshold performance

27
Writing a Module Description in Reality!
Generic Level Descriptors
Existing Aims, Objectives and Syllabus Content of
Module
Translate Level Descriptors into Subject
Descriptors
Review Aim of Module
Existing Assignments and Laboratories
Rewrite Objectives as Learning Outcomes
Develop the module and rethink it including the
initial learning outcomes
Write Threshold and Grading Assessment Criteria
Modify Existing Assessment Methods to test
achievement of assessment criteria
Modify current teaching strategy to enable
learners to reach the learning outcomes /
assessment criteria
28
Writing Learning Outcomes

• A well written learning outcome is likely to
  contain
• A verb that indicates what the learner is
  expected to be able to do at the end of the
  period of learning
• Word(s) that indicate on what or with what the
  learner is acting. If the outcome is about a
  skill then the word may describe the way the
  skill is performed
• Word(s) that indicate the nature (in context or
  terms of standard) of the performance required as
  evidence that the learning was achieved

29
Examples of Learning Outcomes

• The learner is expected to be able to
• demonstrate understanding of the purpose and
  operation of the hardware and software components
  present in personal computers, microprocessors
  and embedded processor applications.
  (Level 1 Computer Systems)
• explain the physical basis of the operation of
  Metal Oxide Semiconductor Transistors (MOSFETs)
  and obtain the small signal model of a MOSFET
  
  (Level 2 Electronic Devices)

30
Features of Good Learning Outcomes

• Must be achievable by students within the time
  available and at the level of learning at which
  the students are. Be realistic!
• Written in terms of the learner being expected
  to be able or as intended learning outcomes
• Should specify areas of learning rather than
  specific curriculum
• There should be in the range of 4 to 10 learning
  outcomes per module too many makes statements
  of the assessment criteria unmanageable
• Should be written in a language that is
  understood by all and is unambiguous
• Each intended learning outcome should represent a
  major achievement expected by students at the end
  of the module
• Learning outcomes must be assessable by a
  reasonable and manageable form of assessment
  within the time allocated to the module
• Achievement of each threshold learning outcome is
  essential in order to pass the module

31
Vocabulary for Writing Learning Outcomes and
Assessment Criteria - A

• Verbs which require evidence of knowing
• Be aware of, define, describe, extract, identify,
  know, label, list, match, measure, name,
  organise, outline, present, recall, recognise,
  recount, relate, repeat, select, state,
  underline, write.
• Verbs which require evidence of comprehension
• Clarify, classify, compare, comprehend, contrast,
  convert, defend, describe, discuss, distinguish,
  estimate, exemplify, explain, express, extend,
  find, formulate, generalise, give examples of,
  identify, illustrate, indicate, infer, interpret,
  judge, justify, name, paraphrase, perform,
  predict, present, report, represent, restate,
  rewrite, select, summarise, translate,
  understand.
• Verbs which require evidence of knowledge /
  understanding
• Apply, arrange, assess, change, choose, compute,
  construct, demonstrate, discover, draw (up),
  exemplify, explain how, find, give examples,
  illustrate, manipulate, modify, operate, order,
  practice, predict, prepare, produce, relate,
  select, show, solve, use, verify

32
Vocabulary for Writing Learning Outcomes and
Assessment Criteria - B

• Verbs which require evidence of analysis
• analyse, break down, calculate, categorise,
  compare, conclude, contrast, criticise, devote,
  diagnose, differentiate, distinguish between,
  divide, elucidate, evaluate, examine, identify,
  illustrate how, infer, justify, outline, point
  out, precis, question, recognise, relate,
  resolve, select, separate, subdivide.
• Verbs which require evidence of synthesis
• account for, alter, argue, build up, combine,
  compile, compose, conclude, create, derive,
  design, develop, devise, engender, enlarge,
  explain, formulate, generalise, generate,
  integrate, manage, modify, order, organise, plan,
  prepare, present, produce, propose, put together,
  rearrange, reconstruct, relate, reorganise,
  report, restate, revise, select, structure,
  suggest, summarise, synthesise, teach, tell,
  write.
• Verbs which require evidence of evaluation
• appraise, assess, choose, compare, conclude,
  contrast, criticise, defend, describe how,
  determine, discriminate, estimate, evaluate,
  judge, justify, measure, question, rate, value.

33
Examples of Learning Outcomes

• Discuss and comment on the following threshold
  learning outcomes
• Describe the structure of telecommunications
  networks (Level 1)
• Understand the ideas of differentiation and
  integration (Level 1)
• Understand the principles of human cognitive
  systems and motor performance when operating
  interactive computer systems (Level 2)
• Demonstrate oral and written communication skills
  (Level 2)
• Appreciate the use of the z-transform in digital
  signal processing (Level 3)
• Improve at working in software engineering teams
  (Level 3)
• Describe the principal characteristics of human
  vision, hearing and speech relevant to
  audiovisual communication and their exploitation
  in image, video, and audio compression (Level 3)

34
Purposes of Assessment

• To provide a licence to proceed to the next stage
  of the programme or to graduation
• To classify the performance of the student in
  rank order
• To improve student learning by providing feedback
  on his or her strengths and weaknesses
• To motivate the student to learn
• To provide information for future selection or as
  a licence to practice
• To evaluate the course and improve teaching
• These may overlap or conflict!
• A common error is to assume that the results of
  an assessment task used for one purpose are
  appropriate for another purpose

35
Some Principles of Assessment

• Assessment shapes learning
• therefore change assessment to change learning
• Match the assessment tasks to the learning
  outcomes
• Match the assessment criteria to the task and the
  learning outcomes
• Keep the assessment criteria simple
• Be fair, reliable and valid in your marking
• Provide meaningful and timely feedback
• Do we apply these principles?

36
Some Assessment Methods

• Formal Examinations
• Knowledge and Understanding
• Analytical Ability
• Problem Solving
• Communication Skills
• Progress tests
• Knowledge and Understanding
• Simple Problem Solving
• Formal Practical Experiments
• Following Instructions and Recording Results
• Practical Ability
• Written Reports or Software Documentation
• Projects
• Requirement Analysis, Research, Problem Solving /
  Synthesis
• Practical Development
• Oral and Written Communication
• Possibly Group Working
• Other Coursework Assignments

37
Common Weaknesses in Assessment

• Tasks do not match the stated outcomes
• Criteria do not match the tasks or outcomes
• Criteria not known to and/or not understood by
  the students
• Overuse of one method of assessment
• Overload of students and/or staff
• Insufficient time for students to do the
  assignments
• To many assignments with the same deadline
• Insufficient time for staff to mark examination
  or assignment
• Absence of well defined criteria so consistency
  is difficult to achieve
• Unduly specific criteria which create a
  straightjacket for students
• Inadequate or superficial feedback provided to
  students
• Wide variation in marking between modules and
  within assessors
• Variation in assessment demands of different
  modules

38
Designing Assessments

• Some questions to be considered
• What are the learning outcomes to be assessed?
• What are the capabilities / skills either
  implicit of explicit, within the learning
  outcomes?
• Is the method of assessment chosen appropriate to
  the outcomes and skills?
• Is the method relatively efficient in terms of
  student time and staff time?
• What alternatives are there and what are their
  advantages and disadvantages?
• Does the specific assessment task match the
  outcomes and skills?
• Are the marking schemes or criteria appropriate?

39
Assessment Criteria a definition

• Not to be confused with assessment methods or
  tasks
• Assessment Criteria provide a clear indication of
  how achievement may be demonstrated
• Often specified with respect to each learning
  outcome, they describe what a learner is expected
  to do in order to demonstrate that the learning
  outcome has been achieved.
• Assessment Criteria may be used in three ways
• To confirm achievement of threshold standards
• To define what is expected in order to achieve
  each of the grades being awarded
• To specify a template of characteristics or
  qualities against which the students performance
  of the assessment task will be judged

40
Writing Assessment Criteria

• Consider the learning outcome being tested
• Consider the assessment task set
• Brainstorm requirements for, or attributes of,
  successful performance of the assessment task
• If necessary specify the range to clarify
  contextual factors and the level
• Focus on what is essential and categorise the
  requirements or attributes into clearly worded
  criteria
• Check that the criteria are measurable or
  assessable in valid and reliable ways and that
  the criteria are clear and unambiguous
• Repeat steps 3, 4 and 5 until you are fully
  satisfied
• Publish the assessment criteria with the
  assessment task and identify the intended
  learning outcome or outcomes that the task is
  assessing

41
An Example

• The learner is expected to be able to
• demonstrate understanding of the purpose and
  operation of the hardware and software components
  present in personal computers, microprocessors
  and embedded processor applications.
  (Level 1 Computer Systems)
• Assessment Task?
• Hardware Laboratory Test and evaluate the
  operation of computer hardware components by
  constructing a serial adder on a logic
  patch-board using available TTL devices.
• Assessment Criteria
• Connect correctly the following functional
  elements on the patch-board provided EX-OR, Half
  Adder, Full Adder, 4 to1 Multiplexer, D-type
  bistable carry store, 4-bit register, 4-bit
  counter, and links modules to form a 4-bit serial
  adder.
• Test each module and record results
• Complete the multi-choice test and score better
  than 70

42
Assessment Criteria Exercise 4 U

• Final Year Individual Project 30 credits
• Aim The project provides the student with the
  opportunity to apply knowledge and practical
  skills gained during the degree programme to the
  solution of a problem agreed with the supervisor.
  
• Form Groups of 3 or 4
• Write a Threshold Learning Outcome for this
  module
• Develop an Assessment Task to address this
  learning outcome
• Write Assessment Criteria for this task
• Repeat 1, 2 and 3 for a desirable graded learning
  outcome!

43
Report Back and Discussion

• Each group to report their
• learning outcomes
• assessment tasks
• assessment criteria
• Comments
• Discussion

44
(No Transcript)
45
Achievement of Module Threshold Learning Outcomes

• Peter Noakes
• Department of Electronic Systems Engineering
• University of Essex

46
Current Developments at Essex

• Internally Funded Project
• The objective is to develop module specifications
  and structures where the threshold and extended
  learning outcomes, and associated assessment
  methods and criteria are clearly defined for
  students.
• As a result a students satisfaction of threshold
  learning outcomes should be readily determined
• Initially concentrating of Year 1 Modules offered
  by ESE
• Working with colleagues in Departments of
  Computer Science and Biological Sciences

47
The Module Description

• For each module Staff and Students will have a
  clear view of
• the threshold learning outcomes
• their method of assessment and
• the assessment criteria used.
• the extended (desirable) learning outcomes
• the assessment activities used for grading
• the assessment criteria used for grading above
  threshold
• the expected time commitment for the activities
• This should ensure that a graduating student
  attains his or her maximum potential

48
Project Proposals

• In future a module description will identify
  separately the threshold and extended (desirable)
  learning outcomes and syllabus
• The threshold syllabus defines the topics and
  associated skills, their method of assessment and
  associated assessment criteria, that will be used
  to demonstrate threshold level achievement for
  the module
• The extended syllabus defines the topics, their
  method of assessment and associated assessment
  criteria, that will be used to demonstrate
  understanding and application of their knowledge
  above the threshold level for the module allowing
  grading above Third class.
• The threshold aspects of the module will be
  taught conventionally with clear guidance to
  required reading, and supported by regular
  formative on-line MCQ testing
• The extended aspects of the module will be taught
  by a combination of special topic lectures and
  directed self study with associated supporting
  problem classes

49
Module Specifications

• Module specifications to be captured by filling
  form to provide entry to a central database
• Data to be extracted for various audiences by
  running different reports
• Linked to marks database to provide clear
  abstraction of satisfactory threshold achievement

50
Module Assessment Current Approach

• Largely based on End of Year Examinations
• All have an in-term Multi-choice Progress Test
• Many modules have a Practical Laboratory
  (Hardware / Software / CAD / Database / Networks
  / Microprocessor / Matlab / Web) Assessment in
  various ways including log book, demonstration,
  presentation, report, oral, OMR test
• Some have Assignments electronic submission,
  paper submission, demonstration, oral
• Some modules include Projects demonstration,
  presentation, report, oral
• Module mark generated by forming a weighted
  aggregate of the marks awarded to individual
  elements
• A 40 overall aggregate does not ensure that all
  learning threshold outcomes have been satisfied!

51
Proposed Module Assessment

• Each module has 100 marks available
• Progress Test threshold material 10 marks
• Examination 2 hour paper 2 parts
• Part A say 8 to 20 questions on threshold
  material to produce say 20 marks
• Part B 3 to 5 on extended material answer 2 or
  3 for full marks applying knowledge, problem
  solving producing say 40 marks
• Laboratory or Assignment Coursework contribute
  say 20 marks on threshold material and say 10
  marks on extended material.

52
Graphical Representation
Module Marks
100
Part B examination
90
80
Degree Graded Assessments
70
60
Practical Activity
50
Part A examination
40
30
Threshold Assessments
Practical Activity
20
10
Progress Test