Assessing Programme Outcomes: Challenges and Possibilities

1
Assessing Programme OutcomesChallenges and
Possibilities
Workshop series to Help You Prepare Your OBA
Project Proposal 17 Dec 07 230 pm 500 pm
KP Kwan ? 2766 6320 ? etkpkwan_at_inet.polyu.edu
.hk
2
Workshop outline

• OBA funding Guidelines for application
• Meaning and purpose of Programme Learning
  Outcomes Assessment (PLOA)
• Challenges and difficulties in assessing
  programme outcomes
• Overview of possible PLOA strategies or methods
  to prove and improve programme effectiveness
• Ideas for developing your OBA project proposal on
  enhancing PLOA

3
OBA funding from LTC 2007-08

• Support collaborative effort in implementing OBA
• Preference for projects focusing on
• Designing effective TLA methods to facilitate
  students to achieve learning outcomes
• Assessing professional and generic programme
  outcomes
• Reviewing the alignment and mapping of subjects
  to programme outcomes
• Priority areas
• Funding limit 400K per project
• Deadline 31 Jan 2008

4
Selection criteria

• Alignment with PolyU goals and priorities in
  implementing OBA in student learning
• Outcomes/deliverables
• Impact
• Collaboration and engagement of front line
  teachers

5
Activity 1Why are you interested?

• In turn, briefly explain
• WHY you are interested in programme learning
  outcomes assessment PLOA
• WHAT you want to get out of this workshop.

Sharing
6
Purpose and meaning of Programme Learning
Outcomes Assessment PLOA
7
Why bother with PLOA?

• To improve student learning
• Integral component of Outcome-Based Education
• Evidence and feedback for improvement
• To prove or demonstrate programme quality and
  effectiveness
• Professional bodies or accreditation agencies
• UGC Quality Assurance Committee Audit (2010)
• To showcase programme and graduate quality to
  appeal to
• Prospective and current students
• Prospective employers
• The community at large

8
What is PLOA?

• Meaning of PLOA
• a systematic process of collecting broader
  evidence or data
• about how well graduates are achieving the
  intended programme learning outcomes, and
• interpreting and using the results to improve
  student learning
• Effectiveness/success defined in terms of
  outcomes actually achieved rather input or
  process indicators
• Focus on overall programme effectiveness rather
  than assessing performance of individual students

9
PLOA in a nutshell
State the PLOs that we want our graduates to
achieve on completing the programme

• Say what we do
• Do what we say
• Prove it
• Improve it

Design teaching, learning and assessment
activities such that students are able to achieve
the aspired PLOs
Collect and present evidence to demonstrate the
extent to which our graduates are actually
achieving the PLOs
Using outcomes evidence to improve the
effectiveness of the programme in facilitating
students achievement of the PLOs
10
Clarifying terminology

• Assessment
• the systematic collection, review and use of
  information for the purpose of improving student
  learning. Marchese, 1997
• Direct or indirect
• Quantitative or qualitative
• Not limited to formal tests or examinations
• Student learning outcomes
• the effect on students of the educational
  process that is, the knowledge, skills,
  behaviours and attitudes that are imparted to
  students through their learning experiences.
  UGC QAC Audit Manual 2007

11
Legitimate outcomes evidence
Direct Indirect
Based on direct assessment of students work or behaviours that demonstrate mastery of learning outcomes Based on subjective view or perception of students attainment of learning outcomes
12
Legitimate outcomes evidence
Direct Indirect
Based on direct assessment of students work or behaviours that demonstrate mastery of learning outcomes Based on subjective view or perception of students attainment of learning outcomes
Quantitative Qualitative
Numeric in nature (e.g. marks, ratings, etc.) Descriptive and categorical in nature (e.g. interviews, open-ended feedback, etc.)
13
Outcomes of PLOA
Programme learning outcomes PLO assessment methods Assessment findings Improvement actions
PLO 1
PLO 2
PLO 3
PLO 4
14
Example
Intended programme learning outcome Students
should be able to identify, analyse and solve
technical problems ABET Criterion 1 Item f

• Outcome assessment methods
• Students self assessment of competence with
  respect to specific programme outcome
• Specific final exam questions included in
  ECET207, 231, 307, 309, 417, 483 and BMET320 to
  measure this outcome Target 70 of students
  scored 70 or higher on each question.

Purdue University Dept of Electrical Computer
Engineering 2007 Annual Assessment Report
15
Example contd

• Assessment findings
• 80.2 of students strongly agreed/agreed that
  they can perform tasks associated with this
  outcome.
• On exam questions targeting this criterion, only
  39.8 of students scored a 70 or better.

• Improvement actions/changes
• The low result was traced to very poor
  performance in ECET284, and was addressed in the
  course during the next semester.
• Problem solving steps will continue to be
  stressed, especially in foundational courses.

Purdue University Dept of Electrical Computer
Engineering 2007 Annual Assessment Report
16
Activity 2An audit of your PLOA activities

• Refer to worksheet 1
• List the most important learning outcomes of your
  programme (PLOs)
• For each of the PLOs, identify the outcomes
  assessment methods and evidence that are already
  in place that you can draw upon to evaluate the
  overall programme effectiveness in achieving the
  outcome
• Reflect on your current PLOA activities and
  explain to your partner
• Problems you have observed
• Gaps in your PLOA that you have identified

Think-pair-share
17
Challenges and difficulties
18
Problem of measurement
Not everything that can be counted counts, and
not everything that counts can be counted.

Albert Einstein
19
Limitation of subject assessments

• Subject grades alone do NOT provide adequate
  information for PLOA
• Graded on students performance w.r.t.
  subject-specific content and outcomes
• Seldom assess broader programme outcomes that cut
  across subjects or require integration of
  learning from different subjects
• Generic competencies not assessed and reported
  separately, if assessed at all
• Reflected short-term learning rather long-term
  learning outcomes or impact of programme
• Some outcomes are difficult to be measured via
  conventional teacher-marked subject assessments

20
Benchmarking and standard

• Criteria for successWhat explicit criteria and
  standard would you use to indicate programme
  success in reaching the intended PLOs?
• BenchmarkingHow to demonstrate/prove that the
  level of achievement and competencies of your
  graduates regarding the PLOs are at par with
  international standard and what is expected for a
  beginning professional in your field?

21
Other problems/difficulties

• Accountability-driven
• Too many outcomes
• Lack of systematic outcomes evidence
• Mostly indirect survey data
• Data not available or presented in ways for those
  who need to analyse and apply it
• Exit outcomes measures rather than value-added
  outcomes measures
• Lack of involvement of front-line staff
• Workload, time and resources for outcomes
  assessment

Just another routine bureaucratic chore !
22
Best practices in PLOA

• Improvement-oriented
• Owned, driven and shared by frontline academic
  staff
• Must include direct measures of students
  performance
• Use both direct and indirect evidence from
  multiple sources
• Involve stakeholders in benchmarking standard and
  evaluating programme outcomes
• Focus more on high-priority outcomes
• Not doing MORE, but doing it DIFFERENTLY and
  EFFECTIVELY

23
Developing a PLOA Plan
24
6-step approach to developing a PLOA

1. Define programme mission and objectives
2. Articulate the intended programme learning
   outcomes
3. Choose appropriate outcomes assessment measures
   and methods
4. Explain how the outcomes evidence will be
   collected, analysed and reported what, when, how
   and by whom
5. Describe how the results will be used to improve
   learning
6. Establish timeline for implementation

25
Possible PLOA strategies for collecting direct
evidence of learning outcomes

• Course-embedded assessment
• Capstone experience/project
• Clinical/professional performance appraisals
• Portfolio assessment
• Standardised tests or exams (e.g. Major Field
  Achievement Tests, GRE, IELTS, CLA, CCTST, ATTA,
  etc.)
• Licensure exams (e.g. HKICPA Exam)
• External examiner or expert panel review (of
  student work samples)

26
Subject grading vs PLOA
Subject grading Outcomes assessment



27
Subject grading vs PLOA
Subject grading Outcomes assessment
Focus on performance of each individual student Focus on aggregate performance of students


28
Subject grading vs PLOA
Subject grading Outcomes assessment
Focus on performance of each individual student Focus on aggregate performance of students
According to mastery of subject-related knowledge and skills Based on evidence of attainment of specific programme outcome (assessment rubrics)

29
Subject grading vs PLOA
Subject grading Outcomes assessment
Focus on performance of each individual student Focus on aggregate performance of students
According to mastery of subject-related knowledge and skills Based on evidence of attainment of specific programme outcome (assessment rubrics)
Need to differentiate between different levels of performance (from A to F) Sufficient to distinguish between broad categories(Achieved - Not achieved) (Inadequate - Adequate - Proficient)
30
Assessing for grading and PLOA
Subject grade for each student for students
acad achievement record
Grading of students work for subject assessment
Subjectteacher
31
Assessing for grading and PLOA
Subject grade for each student for students
acad achievement record
Grading of students work for subject assessment
Subjectteacher
Evidence of aggregate performance of students on
particular PLOs for Dept or Programme review
Subjectteacher
Re-assessing students work for outcomes
assessment
ORDept review panel, student peers, employers
32
Possible PLOA strategies for collecting indirect
evidence of learning outcomes

• Students self-assessment of learning gains
  (e.g. the PolyU SAARD administered by SAO)
• Surveys and interviews of employers,
  alumni/graduates, current students
• Students engagement in curricular, co-curricular
  and extra-curricular activities (e.g. community
  service learning)
• Institutional data (e.g. completion/progression
  rate, employment statistics, etc.)
• Standardised surveys
• National Survey of Student Engagement
• College Student Experience Survey
• Course Experience Questionnaire

33
Example of acomprehensive PLOA plan
PLOA activities /measures PLO 1 PLO2 PLO3 PLO4 Who is responsible?
Capstone project X X Project supervisors
Embedded test item in subject ProfStudy301 X Lecturer of ProfStudy301
Clinical/professional performance appraisal X X X X Clinical educators or WIE supervisor
Language competency assessment X English Language Centre
Exit survey and interviews X X X X Programme leader
Employer survey X X X SAO
34
Activity 3Planning for enhancing PLOA

• Refer to the PLOA problems or gaps you identified
  in Activity 2
• What new or changed PLOA measures or processes
  will be most appropriate for addressing the
  problems or gaps?
• Explain briefly your project ideas to your
  partner
• What is the problem/gap?
• What outcomes assessment activity will be
  added/modified? How?
• Who will be involved?
• How will you evaluate its usefulness or impact?

Think-pair-share
35
Developing Your OBA Project Proposal
36
Good PLOA project proposals

• Designed to address important gaps/problems in
  current PLOA measure or practice
• Based/modelled on international best practices
• System-wide and sustainable impact
• Participatory and collaborative (within
  Programme, Department or Faculty/School)
• Involvement of key stakeholders
• Systematic project evaluation plan
• Effective plan to disseminate deliverables/experie
  nce

37
Ideas for potential projects

• Develop a comprehensive PLOA plan
• Design and pilot innovative PLOA activities or
  measures, e.g.
• Course-embedded assessment
• Capstone course/projects
• Portfolio assessment, etc.
• Refine/Design processes to collect and use
  outcomes evidence for improving programme
  outcomes
• e.g. assessment committee, PLOA report,
  outcomes-based programme review
• Benchmarking graduates competencies
• Involving employers and professionals in
  benchmarking standards and assessing graduates in
  key PLOs

38
Other ideas

• Other ideas and suggestions?
• Further questions?

39
Consultation and support

• KP Kwan EDCx6320 TU612etkpkwan_at_inet.poly
  u.edu.hk

40
Useful references

• Programme-Based Review and Assessment Tools and
  Techniques for Program Improvement University
  of Massachusetts Amherst Available
  athttp//www.umass.edu/oapa/oapa/publications/on
  line_handbooks/program_based.pdf
• California Assessment Institute
  Resourceshttp//cai.cc.ca.us/Resources/index.htm
  
• Allen, M. (2004). Assessing Academic Programs in
  Higher Education. Anker Publishing Co. Ltd.
• Walvoord, B.E. (2004). Assessment Clear and
  Simple A Practical Guide for Institutions,
  Departments, and General Education. Jossey-Bass.