QAC Seminar

1
QAC Seminar Introduction to QAC Audit
HKBU 12 Mar 2008
Trevor Webb Assistant Secretary-General QAC
Secretariat
Quality Assurance Council
2
Seminar Objectives

• To provide participants with
• Background to the QAC and an
• Overview of the QAC audit process
• Outline of the responsibilities of audit panel
  members
• Introduction to the audit methodology and to the
  auditing of teaching and learning
• Explanation of methods of gathering information,
  and audit and interviewing techniques
• Opportunity to identify issues and raise
  questions

Quality Assurance Council
3
Overview of QAC

• QAC Structure
• The Council
• The Secretariat
• Auditors and Audit Panels
• Primary roles of the QAC
• Conduct Quality Audits of Teaching Learning
• Promote QA QE / spread good practice
• Rationale for establishing the QAC

Quality Assurance Council
4
Progress To Date

• Audit Manual (http//www.ugc.edu.hk/eng/qac/index.
  htm)
• Auditor Briefing Notes
• Initial Audit Cycle
• Register of Auditors
• Auditor Induction Professional
  Development
• CUHK and HKBU Audits

Quality Assurance Council
5
QAC Quality Audits 1

• QAC Quality Audits
• will NOT question institutional autonomy
• will NOT validate or re-accredit institutions
• (self-accrediting status is recognised)
• will NOT be excessively intrusive
• will NOT be confrontational or trying to find
  fault
• will NOT be about surprises
• will NOT compare institutions
• will NOT determine funding of institutions

Quality Assurance Council
6
QAC Quality Audits 2

• QAC Quality Audits are
• about self-evaluation
• about peer review
• about working with institutions
• about partnership
• about dialogue
• for institutions and students
• a constructive but robust process
• about helping institutions to improve

Quality Assurance Council
7
Roles Responsibilities of Auditors

• Auditors are expected
• to be familiar with the QACs approach/methodology
  
• to read thoroughly the Institutional Submission
  (IS) and the Supplementary Material
• to provide written comments on the IS before the
  Initial Meeting of the Audit Panel
• to participate fully in meetings, contributing to
  the discussion
• to avoid applying pre-conceived templates to
  their consideration of the audited institution
• to contribute to, and agree, the Panels Report
• to work as a Team
• (QAC Audit Manual Annex B.3)

Quality Assurance Council
8
QAC Approach to Audit

• QAC Audits are about
• Fitness for Purpose in teaching learning,
• where the
• Quality of Student Learning is the
• Centre-piece (focal point),
  with an emphasis on
• Quality Enhancement, i.e. continuous improvement
• QAC Audit Manual Section 3.1

Quality Assurance Council
9
QAC Audit Methodology Focus Areas 1

• Audit Panels will use the ADRI methodology
• Approach (What is the institutions purpose?)
• Deployment (How does the institution achieve
  its purpose?)
• Results (What evidence does the institution
  have that its purpose is being achieved?)
• Improvement (What processes are in place for
  improvement)
• in relation to 11 core Focus Areas (to be
  revisited later in the Seminar)

Quality Assurance Council
10
QAC Audit Methodology Focus Areas 2

• Audit Panels examine
• how an institutions practices in each focus area
  contribute to student learning quality
• whether an institutions objectives, procedures
  and outcomes are informed by appropriate external
  reference points
• a sample of programmes in depth, as test cases
• (QAC Audit Manual Sections 3.3, 3.4 and 3.5)

Quality Assurance Council
11
The QAC Approach

• Fitness for Purpose
• where institutions have different purposes which
  reflect their missions and role statements they
  have agreed with the UGCnot a review against
  pre-defined standards.

Quality Assurance Council
12
QAC Audit Scope Focus

• Student learning quality is the focal point of
  the audit system
• The QAC has selected a set ofactivities common
  to all institutions, as the focus areas of
  audit.taken together, the focus areas
  effectively define the scope of QAC audit. (QAC
  Audit Manual, Section 3.4)

Quality Assurance Council
13
Scope of QAC Audit Activities

• All HK first degree programmes and above, however
  funded, that are provided by UGC- funded
  institutions
• Includes provision by continuing and community
  education arms
• Includes self-financing and joint programmes
  leading to a HK award
• Includes teaching and learning aspects of
  research degree programmes.

Quality Assurance Council
14
QAC Audit Focus Areas

• Articulation of appropriate objectives
• Management planning accountability
• Programme development and approval processes
• Programme monitoring and review
• Curriculum design
• Programme delivery, resources, mode, learning
  env.
• Experiential / out of classroom learning, i.e.
  external engagement (e.g. OS exchange,
  work-integrated-learning, Professional and
  Industry linkages, etc.)
• Assessment
• Teaching quality and staff development
• Student participation
• Activities specific to research degrees

Quality Assurance Council
15
The QAC Audit Process
Preparation Panel Selection (QAC) Self-Review
Institutional Submission (Auditee)
Preliminary Phase The Initial Meeting Preliminary
Offshore Visits (Auditors)
The Audit Visit (The main event)
The Audit Report (QAC) Follow-Up (Auditee QAC)
16
Preparation

• Initiation of the process
• Nomination of additional focus areas
• Panel selection
• Self-review
• Institutional Submission

Quality Assurance Council
17
Self-review Institutional Submission

• Self- review is an opportunity for the
  institution to analyse and reflect on its
  objectives for student learning, on its
  procedures, standards and benchmarks,
  andmeasures (Audit Manual, Section 5.4)
• Self-review also forms the basis of the
  Institutional Submission to the audit
• QAC does not prescribe approach, but there are
  some key issues that should be addressed.

Quality Assurance Council
18
Key Self-review Questions

• What student learning outcomes are we trying to
  achieve? (graduate attributes, standards etc.)
• How are we trying to achieve our desired student
  learning outcomes? (policies procedures etc.)
• How do we know we are achieving them? (what
  measures and benchmarks, etc.)
• How do we respond to what the evidence tells us?
  (what feedback mechanisms, impact on programme
  standards etc.)

Quality Assurance Council
19
Institutional Submission

• QAC does not prescribe a detailed format, but.
• Auditee is expected to submit a draft
  Institutional Submission to QAC six months prior
  to Audit Visit and the final three months prior
• Two components
• Main Submission
• Supplementary Materials key documents referred
  to in Main Submission.

Quality Assurance Council
20
Main Submission

• 12,000 words
• Section 1 Introduction/Structures
• Section 2 Overview of Teaching Learning
  QA System
• Section 3-N Focus Areas
• Section N1 Conclusion
• References
• Should be Stand-alone

Quality Assurance Council
21
Preliminary Phase

• Initial Meeting of the Panel (about 9 wks before
  AV)
• - Refresh on QAC audit procedures
• - Review of Institutional Submission to identify
  major issues questions
• - Selection of programmes for sample review
• - Identification of further information/documenta
  tion required
• - Planning of Audit Visit
• - Decisions on whether to visit provision
  outside HK
• - Decisions on who to meet during the Audit
  Visit
• Preliminary Visit to Auditee to plan the Audit
  Visit (about six weeks beforehand)
• Other visits, such as to operations outside HK,
  as required.

Quality Assurance Council
22
The Audit Visit (s)

• Main Visit length 3-5 days
• Panel typically operates as a group, but may
  split for some parallel sessions
• An Audit Visit Programme and Worksheets are
  prepared to provide a common reference for the
  panel and to ensure that the necessary issues are
  covered.

Quality Assurance Council
23
Audit Visit Usual Sessions

• Opening meeting with President/Vice-Chancellor
• Staff (various groups, senior through to junior)
• Students (including those taking sample
  programmes)
• Key boards and committees
• Range of stakeholders (Council Members
  Employers)
• Panel deliberation sessions
• Sessions on request
• Call-back sessions
• Exit meeting
• (QAC Audit Manual, section 7 and Annex A.2)

Quality Assurance Council
24
The Audit Report

• Format
• Executive Summary
• Findings for each Focus Area
• - Commendations
• - Affirmations
• - Recommendations
• Programmes sampled (but not named)
• Institutional response (Appendix)

Quality Assurance Council
25
The Audit Report (cont.)

• Concern by QAC over fairness, accuracy and
  balance
• Produced by the Audit Co-ordinator
• - exposure draft sent to auditee
• - feedback considered by AC and Chair
• Brief institutional response
• Consideration by QAC
• Publication
• Consideration by the UGC

Quality Assurance Council
26
Follow-Up

• 18 Month Progress Report
• - Action (s) taken
• - Person responsible
• - Date by which action is expected to be
  completed
• - Means by which the institution will assess
  effectiveness
• Consideration by QAC
• Consideration by UGC

Quality Assurance Council
27
Any Questions? Discussion
Quality Assurance Council
28
Audit Method Components

• Approach to audit (Fitness-for-purpose v)
• Audit Scope Focus (Areas of Focusv)
• Triangulation of data information sources
  (Institutional Submission, supporting documents,
  sampling of programmes, meetings
• Document Analysis and Interpretation of
  Information - ADRI (Identification of key quality
  issues and lines of enquiry)
• Identification and Reporting of audit findings
  (to be covered in depth in the pre-audit
  Intensive Workshop)
• Auditing, interviewing, questioning, and meeting
  organisation tools techniques required for
  effective performance of the above.

Quality Assurance Council
29
The Critical Questions
A
D
I
R

1. What are the institutions objectives and on what
   basis does it set these objectives? (A)
2. How does the institution go about achieving its
   objectives? (D)
3. How does the institution measure achievement of
   its objectives and what evidence is there that
   the objectives are being achieved? (R)
4. What processes are in place for improvement (and
   how do these processes feed back to the setting
   and reporting of achievement of the objectives)?
   (I)

Quality Assurance Council
30
Managing Meetings

• How might participants be feeling on entering the
  meeting room?
• How can the panel help participants feel more at
  ease?
• How can the panel ensure that all participants
  are involved in the discussion?

Quality Assurance Council
31
Questioning Techniques 1

• In general, auditors should avoid
• asking complex or multiple questions up-front (a
  more straight-forward question, with a follow-up
  question/(s) is generally more effective)
• telling anecdotes or making speeches
• detailing the situation in their own
  organisation and,
• offering gratuitous advice.

Quality Assurance Council
32
Questioning Techniques 2

• Creating an atmosphere for genuine dialogue
  during the Audit Visit is extremely important
• Panel members should try to act as colleagues and
  peers of the participants, rather than acting as
  inspectors
• To this end, the questioning and discussion must
  be fair and polite. It must also be rigorous and
  incisive, as the audit report must reflect both
  the institutions achievements and its areas for
  improvement (Commendations, Affirmations, and
  Recommendations)
• Panel members should ensure that all participants
  are engaged in the discussion, for the
  encouragement of an organisational culture of
  continuous quality improvement.

Quality Assurance Council
33
Any Questions? Opportunity for final discussion
of any Seminar topics.
Quality Assurance Council
34
Thank you for your participation
Quality Assurance Council