Pengalaman Sebagai Penilai: Pengukuran dan CQI

1
Pengalaman Sebagai Penilai Pengukuran dan CQI

• Prof. Ir. Dr. Abdul Wahab Mohammad
• Fakulti Kejuruteraan

2
Tujuan

• Untuk memberi perspektif pengalaman sebagai
  seorang penilai Lembaga Jurutera Malaysia dalam
  konteks (i) Pengukuran dan (ii) CQI
• Menghubungkait dengan keperluan MQF dan MQA CODE
  OF PRACTICE INSTITUTIONAL AUDIT

3
Outcome Based Education (OBE)

• Outcome-Based Education means clearly focusing
  and organizing everything in an educational
  system around what is essential for all students
  to be able to do successfully at the end of their
  learning experiences. This means starting with a
  clear picture of what is important for students
  to be able to do, then organizing the curriculum,
  instruction, and assessment to make sure this
  learning ultimately happens (Spady, 1994).

4

Malaysian Qualifications Framework (MQF)

• A unified system and reference point for all
  national qualifications
• Uses nationally endorsed principles, criteria,
  guidelines competency standards that set the
  boundaries for naming, positioning and linking
  qualifications
• Qualifications are awarded by certified providers
  acting under different Acts and mechanisms
• Providers operate within or outside the formal
  education system including individually-driven
  life long learning efforts and verifiable
  workplace training and experiences.

5
(No Transcript)
6
Features of the MQF

• clear criteria for defining each qualification
  underpin standards facilitate mutual
  recognition
• Provide flexible learning pathways systematic
  credit transfer for progression within across
  educational sectors, including RPL within/outside
  the country
• Not a rigid system- accommodates new
  qualifications that arise according to needs
  growth of knowledge

7
Learning Outcomes

• Abilities in various domains of learning that
  students are expected to demonstrate as evidence
  of competency.
• MQF Domain of Learning
• Mastery of body of knowledge (depth, breadth and
  relative difficulty of specific content)
• practical or psychomotor skills (range and
  complexity),
• scientific method, critical thinking, problem
  solving, autonomy in decision-making
• communication skills, leadership and team work
• information management and life long learning
  skills
• personal attributes, ethics, shared values and
  professionalism
• social responsibility and accountability
• Entreprenerial and managerial skills

8
Learning Outcomes
PSYCHOMOTOR/ PRACTICAL/ TECHNICAL SKILLS
Credits
Competency level
SOCIAL SKILLS RESPONSIBILITY
KNOWLEDGE
PROFESSIONALISM, VALUES, ATTITUDES, ETHICS
9
3. Learning Outcomes

• The LEVELS are distinguished from each other
  principally by learning outcomes
• Other distinguishing characteristics may include
• minimum entrance requirement,
• typical duration
• cumulative duration in full time equivalent
  (part-time courses)

10
(No Transcript)
11
Bachelor Degree

• Bachelor degree prepare students for public
  career, entering post-graduate programmes and
  research as well as highly-skills career. It also
  prepares an individual to take full
  responsibility an autonomy in making professional
  decision. The bachelors degree is awarded to an
  individual who are able to
• Demonstrate knowledge and understanding of
  fundamental principles for a specific discipline
  obtained from higher level text-books
• Apply knowledge and understanding using methods
  that demonstrate professionalism in work
• Argue and solve problems within the respective
  study discipline
• Use the technique to find and apply data for
  decision making by taking into account the
  relevant social, scientific and ethics issues
• Communicate and deliver effectively information,
  ideas, problems and solutions to expert and
  non-expert
• demonstrate team work and suitable interpersonal
  skills for the job
• demonstrate self-study skills to pursue advanced
  life-long learning with higher autonomy

12
MISSION GOALS
VISION
TEACHING LEARNING LEARNING OUTCOMES
Design of the programme content structure, T/L
methods, assessment methods
graduates
students
Continuous Quality Management governance
structure administration of resources to
design, deliver, monitor enhance programmes
RESOURCES Entry criteria, student support
services, academic faculty, administrative staff,
support staff, physical facilities, money, ICT,
community stakeholder engagement, research
links to education
13
Continuous Quality Improvement (CQI)

• Simply put, it is the concept that there is
  always room for improvement. Within a company, it
  is the commitment to constantly improve
  operations, processes and activities in order to
  meet customer requirements in an efficient,
  consistent and cost effective manner
• Industry Canada website at www.ic.gc.ca

How do we apply CQI within our learning
environment?
14
(No Transcript)
15
CQI

• What are you currently doing?
• What works well?
• What does not work well?
• What are you planning for the future?

16
Code of Practice for Institutional Audit
(COPIA) and Code of Practice for Programme
Accreditation (COPPA)
Entrusted with and committed to ensuring quality
in Higher Education
17
CODE OF PRACTICE INSTITUTIONAL AUDIT

• Section 1 An Overview of Quality Assurance in
  Higher Education
• Section 2 Guidelines on Criteria and Standards
  for Higher Education Institution
• Section 3 The Quality Audit
• Section 4 Institutional Audit
• Section 5 The Panel of Auditors
• Section 6 Guidelines for the
• Section 7 Institutional Audit Report

18
(No Transcript)
19
SELF-ASSESSMENT
Self-assessment in an organizational setting,
refers to a comprehensive, systematic and regular
review of an organization's activities and
results referenced against certain
criteria/benchmark. The Self-Assessment process
allows the organization to discern clearly its
strengths and areas in which improvements can be
made and culminates in planned improvement
actions which are then monitored for progress
20
AREAS OR ASPECTS OF STANDARDS

• Vision, mission, educational goals and learning
  outcomes
• Curriculum design and delivery
• Assessment of students
• Student selection and support services
• Academic staff
• Educational resources
• Programme monitoring and review
• Leadership, governance and administration
• Total continual quality improvement.

21
(No Transcript)
22
CRITERIA

• Each area is sub-divided into specific criteria,
  which are operationally defined and serve as
  performance indicators of quality.
• The criteria reflect three critical factors
  input, performance and quality management
  factors.
• Each criterion is operationally defined at two
  levels of attainment benchmarked and enhanced
  standard

23
STANDARDS

• The standard defines the expected level of
  attainment for each criterion and serves as a
  performance indicator.
• Standards are specified at two levels of
  attainment.
• The use of two levels is to acknowledge that
  institutions are at different stages of
  development and to emphasise that quality
  improvement is a continuous process.

24
Benchmarked Standard The standards must be met
and compliance demonstrated during institutional
audit. Benchmarked standards are expressed as a
must.
Enhanced Standard The standard is in accordance
with the international and national consensus on
good practices in higher education. Institutions
should be able to demonstrate achievement of some
or all of these or that initiatives to do so have
or will be taken. Achievement of these standards
will vary with the stage of development of the
institutions, their resources and policies.
Standards for enhanced standard are expressed by
a should.
25
LEARNING OUTCOMES

• The quality of the institution is ultimately
  assessed by the ability of its graduates to carry
  out their expected roles and responsibilities in
  society.
• This requires a clear definition of the
  competencies that are expected to be achieved at
  graduation and these should reflect the level of
  competencies expected in the Malaysian
  Qualifications Framework.

26
Outcomes for Bachelor Degree

• Bachelor degree prepare students for public
  career, entering post-graduate programmes and
  research as well as highly-skills career. It also
  prepares an individual to take full
  responsibility an autonomy in making professional
  decision. The bachelors degree is awarded to an
  individual who are able to
• Demonstrate knowledge and understanding of
  fundamental principles for a specific discipline
  obtained from higher level text-books
• Apply knowledge and understanding using methods
  that demonstrate professionalism in work
• Argue and solve problems within the respective
  study discipline
• Use the technique to find and apply data for
  decision making by taking into account the
  relevant social, scientific and ethics issues
• Communicate and deliver effectively information,
  ideas, problems and solutions to expert and
  non-expert
• demonstrate team work and suitable interpersonal
  skills for the job
• demonstrate self-study skills to pursue advanced
  life-long learning with higher autonomy

27
Benchmarked standards

• The institution must formulate educational goals
  consistent with institutions vision, and
  mission.
• The institution must define the specific
  competencies that students should demonstrate at
  graduation.
• The competencies must include
• mastery of body of knowledge
• practical skills
• social skills and responsibility
• ethics and professionalism
• scientific method,
• critical thinking and problem solving
• communication skills and team work
• information management and lifelong learning and
• entrepreneurship and management.

28
Enhanced Standard

• The link between competencies expected at
  graduation and those required during career
  undertakings and/or further studies should be
  specified.

29
ASSESSMENT OF STUDENTS

• Student assessment is an important aspect of
  quality assurance because assessment methods
  drive student learning and the outcome of
  assessment is used in awarding qualifications.
• Hence methods of student assessment have to be
  clear and must support the learning outcomes.

30
Benchmarked standards

• Assessment principles, methods and practices must
  be compatible with learning outcomes and
  programme content.
• The process and methods of assessment must
  reflect the change to the programme outcomes from
  any review exercise.
• The assessment must encourage critical thinking,
  problem solving and methods to enhance
  intellectual skills.

31
Enhanced standards

• Assessments should be reviewed from time to time
  to achieve integrated learning

32
ASSESSMENT METHODS

• Benchmarked standards
• The frequency and methods of student assessment
  including the grading criteria and all awards
  must be documented and communicated to students
  on commencement of the programme.
• The assessment methods must comprise of summative
  and formative evaluations as well as theory and
  practical assessments (where relevant).
• A variety of methods and tools must be used
  appropriately for the given learning outcomes and
  competencies such as communication, problem
  solving, teamwork and self-directed learning.
• The institution must have mechanisms to ensure
  the validity, reliability and fairness of the
  assessment system.

33

• Enhanced standards
• Methods of assessing should include external
  assessors.
• The institution should evaluate and document the
  reliability and validity of all the assessment
  methods used and from time to time, review and
  introduce new assessment methods.

34
Forms of Assessment

• Formative Assessment The collection of data and
  the feedback of the results on an ongoing basis
  (G. Rogers J. Sando, 1996)
• Summative Assessment Designed to produce
  information that can be used to make decisions
  about the overall success of the project or
  process. (G. Rogers J. Sando, 1996)

35
Effective Assessment Programs

• Characteristics include
• Aligned with the institutions mission
• Has faculty ownership and responsibility
• Supported institution-wide
• Based on clear and measurable outcomes
• Uses multiple measures
• Provides useful feedback to all parties
• Leads to improvement
• Established process to evaluate assessment program

36
(No Transcript)
37
MANAGEMENT OF STUDENT ASSESSMENT

• The management of the assessment system is
  intrinsically linked to the institutions
  responsibility as a body that confers
  qualifications and other awards of national and
  international standing.
• The robustness and security of the processes and
  procedures related to student assessment are
  important in inspiring confidence in the quality
  of the degree that is awarded by the institution.

38

• Benchmarked standards
• The institution must publish its grading,
  assessment and appeal policies and its practices
  must be consistent with the policy.
• The institution must have mechanisms to ensure
  the security of all documents and records related
  to assessment

39

• Enhanced standards
• The curriculum or programme committee should have
  mechanisms to review and implement new methods of
  assessment.
• Students representatives, academic staff and
  other stakeholders should be involved in
  improving the system of student assessment.
• Representatives of relevant stakeholders that may
  include staff and students should be involved in
  assessment review exercises.

40
PROGRAMME MONITORING AND REVIEW

• Educational programmes are improved by evaluating
  the structure and process of education that
  include
• administrative structure,
• leadership and governance,
• the learning environment and culture of the
  institution
• specific curriculum components such as the
  syllabus, teaching methodologies and student
  performance as well as general learning outcomes
  such as career choice and performance in further
  training.

41
PROGRAMME MONITORING AND REVIEW

• Feedback needs to be obtained from multiple
  sources to strengthen the evidence-based platform
  of educational quality in higher education.
• Evidence is gathered from students and
  graduates feedback, performance in examinations,
  longitudinal study of graduate performance and
  perception of significant stakeholders within and
  outside the institution on the strengths and
  weaknesses of graduates and programme relevance.
• The sources of information include students,
  graduates as well as stakeholders in the
  community and amongst employers, educational and
  government agencies, professional organisations
  and postgraduate educators.

42
PROGRAMME MONITORING AND REVIEW

• Measures of student performance would include
  information on the average study duration,
  assessment scores, pass and failure rates at
  examinations, success and dropout rates,
  students and graduates report about their
  course experience, as well as time spent by
  students on areas of special interest.
• Evaluation of student performance in examinations
  can reveal very useful information. If student
  selection has been correctly done, a high failure
  rate in a programme indicates something amiss in
  the curriculum content, teaching-learning
  activities or assessment system.
• The programme committee needs to monitor the pass
  rate in each course and investigate if the rate
  is too high or too low.

43
MECHANISMS FOR PROGRAMME EVALUATION

• Benchmarked standards
• The institution must establish mechanisms and
  resources for programme evaluation and for
  monitoring the implementation and student
  progress.
• The mechanisms for programme evaluation must
  include the provision of benchmarked data through
  the use of valid and reliable methods such as
  involving experts in education and external
  assessors.
• Feedback received must be reviewed by the
  appropriate committee(s) and the information
  channelled to those who are responsible for
  programme development and implementation for
  further actions and dealt with appropriately.
• Programme evaluation must address the structure
  and process (e.g. teaching-learning methods) of
  education.

44

• Enhanced standards
• The institution should have its own internal
  validation or accreditation and re-validation or
  re-validation processes and mechanisms for all
  its programmes.
• The institution should proceed only after careful
  review by the appropriate bodies, approval by the
  governing board, and any necessary review and
  approval by the relevant authorities.

45
TEACHER AND STUDENT FEEDBACK

• Benchmarked standard
• Teachers and students must have appropriate
  channels for informing issues to the authority.
  Effective means of both teacher and student
  feedback must be systematically sought.

• Enhanced standards
• Teachers and students should be given feedback on
  the actions that have been done to address the
  issues that they have disclosed.
• Teachers and students should be actively involved
  in the planning of programme evaluation and its
  results be used for programme improvement.

46
STUDENT PERFORMANCE

• Benchmarked standard
• Various aspects of student performance must be
  analysed in relation to the mission of the
  institution, the curriculum and the outcomes of
  the programmes.

• Enhanced standards
• Student performance and progression should be
  analysed taking into consideration student
  background and conditions as well as entrance
  qualifications.
• Analysis of performance should be used to provide
  feedback to the committees responsible for
  student selection, curriculum planning and
  student counselling

47
INVOLVEMENT OF STAKEHOLDERS

• Benchmarked standards
• Programme evaluation must involve the governance
  and administration of the institution, members of
  the institution, employers, where appropriate
  representatives of the community, education and
  government agencies and professional
  organisations.
• For professional programmes, the professional
  bodies must be involved in programme evaluation.

• Enhanced standards
• For non-professional programmes the relevant
  industry should be invited to participate in
  programme evaluation.
• The range of stakeholders should have access to
  results of the course and programme evaluation,
  and their views on the relevance and development
  of the curriculum be considered.

48
TOTAL CONTINUAL QUALITY IMPROVEMENT

• Benchmarked standard
• The institution must establish dynamic policies,
  procedures and mechanisms for regular reviewing
  and updating of its structure, functions,
  strategies and core activities to assure quality
  and must rectify documented deficiencies.
• The institution must provide evidence of the
  review system, the result of the conducted
  review, the steps undertaken to implement the
  changes and evidence of achievement.
• There must be a relation of quality assurance
  with the achievement of the key performance
  indicators.

49

• Enhanced standard
• The process of continual improvement should be
  based on prospective studies and analyses and
  should lead to the revisions of the policies and
  practices of the institution in accordance to
  past experience, present activities and future
  perspectives.

50
SYSTEM PROGRAMME OUTCOMES
Input from Stake holders
Direct Measures Indirect Measures
Benchmarking

• The following changes to the PO system
• Statement has been improved from 12 to 15 to
  incorporate technoprenuer etc
• Specific performance criteria have been
  identified
• Rubrics have been improved and finalised
• Implementation at various level
• Contribution from outside the faculty as well
  (PPP, Kolej kediaman)

CQI
Who are our stakeholders? UKM, BEM, IEM,
MQA Industrial Advisory Panel Alumni, Students,
Faculty Members
Who, How and When?
Direct Measures (Summative) Assessment of
quizzes, tests and exams Use of rubric for soft
skills Assessment of reports, lab sheets
Indirect Measures Exit survey, Course
survey Alumni and industry survey Kajian Halatuju
Kejuruteraan 2006 External Assessors
Input from Stakeholders Annual meeting with
Industrial Advisory Board, Alumni Department and
Faculty review meeting
Input from Industry Specifically through
industrial training visit Input from guest
lecturers, industrial visit
Who are involved Deanery, Quality Coordinator,
Head of Department, Head of Programme, Industrial
Training Unit, Individual lecturers
How and When? OBE Committee, BJK, Dept and
Faculty Meeting, Faculty Retreat, IAB Meeting
51
SYSTEM CURRICULUM/COURSE OUTCOMES
Direct Measures Indirect Measures
External Benchmark

• The following changes to the PO system
• Department has to decide whether any significance
  changes have been made
• Performance has been improved
• Student feedback are accounted for

Input from Lecturers
Curriculum/ Course Outcomes
EAC Requirements
Input from External Assessors, IAB
Who, How and When?
Direct Measures (Formative) Assessment of
quizzes, tests and exams Use of rubric for soft
skills Assessment of reports, lab sheets, FYP, etc
Indirect Measures Exit survey Course
survey External Assessors Industry input Student
Feedback
The curriculum revision in 2005 have taken into
account input from EAC, External Assessors, IAB,
Lecturers as well as other benchmarking efforts
52
Kesimpulan

• Proses pengukuran dan penilaian dan CQI adalah
  berkait rapat di dalam perlaksanaan MQF
• Panduan mengenai 9 aspek

53
Sekian, Terima kasihSoalan dan Perbincangan
adalah dialu-alukan