The literacy/ABET situation

1
Constructing an integrated model for the quality
assurance of school based assessment in
preparation for the National Senior Certificate

2
Introduction

• International trend in assessment is towards a
  combination of both external and internal
  assessment.
• Internal assessment is often put in place to
  collect evidence on learning outcomes that do not
  easily lend themselves to pen and paper tests.
• Can also include learning outcomes that are
  traditionally assessed by external examinations.
  
• Reliability of internal or school based
  assessment is always in question.
• Statistical approaches to quality are
  restrictive, but are adopted because they are
  easy to implement.

3
Challenges Associated with School Based
Assessment

• The level of teacher preparation.
• Variation in the scoring of assessment tasks
  among teachers.
• Increase in both teacher and learner work load.
• Variance in SBA can result from learners being
  assisted by their teachers, peers or parents or
  due to the difference in the difficulty of
  classroom based tasks.
• Different approaches to SBA among different
  schools.

4
Current Status of SBA in South Africa

• Quality education is a national priority in South
  Africa.
• Emerging from an apartheid education, Government
  has embarked on overhauling the entire
  curriculum.
• As from 2008, a new certificate, the National
  Senior Certificate (NSC) will be issued at the
  end of Grade 12.
• SBA constitutes 25 and external examinations
  constitute 75 of the final assessment.
• National system of education with the national
  Department of Education responsible for policy
  development and policy compliance and the nine
  provincial education departments, responsible for
  the implementation of policy.

5
Current Status of SBA in South Africa

• Umalusi, the Quality Assurance Council, is
  responsible for the final standard and quality
  of assessment, leading to the issue of the
  certificate.
• Compulsory inclusion of SBA as part of the final
  exit certificate was mandated in 2001.
• Given the concerns regarding the reliability of
  SBA, a model of statistical moderation was
  adopted.
• SBA marks are adjusted per school, per subject
  within a certain range of the adjusted
  examination marks.
• Provincial Education Departments (PEDs), are
  responsible for internal moderation.
• Effectiveness of internal moderation systems
  varies from one province to the other.

6
Model for Quality Assurance of SBA

7
Conceptual Framework for Quality Assurance of SBA

• Basic Principles
• Quality Assurance must focus on the quality of
  each component of the assessment and not just the
  end-product.
• Quality assurance must focus on what happens
  before, during and after the assessment takes
  place
• Before Establishing appropriate circumstances
  for assessment to take place
• During Verifying that assessment procedures and
  judgments are appropriate.
• After Retrospective analysis to see whether
  assessment procedures or assessor judgments need
  to be improved.
• Model based on the Input, Process , Output (IPO)
  model, which states there is a causal link
  between the inputs, processes and outputs.

8
Conceptual Framework for Quality Assurance of SBA

• Basic Principles
• Point of departure is the establishment of
  clearly defined standards that are accepted and
  understood by all participants in the assessment
  process.
• Definition of standards must be followed by
  appropriate support, training and guidance.
• Implementation must be accompanied by monitoring
  and evaluation, leading to feedback.

9
Conceptual Framework for Quality Assurance of SBA

FEEDBACK

• INPUT
• Policies
• Resources
• Structures
• Procedures

• PROCESS
• Designing
• Administration
• Evaluating

• OUTPUT
• Final Assessment Result

STANDARDS
10
Conceptual Framework for Quality Assurance of SBA

• Fundamental Pillars
• Setting of Standards
• Curriculum Standards
• Performance Standards
• Assessment System Standards
• Moderation Standards
• Support and Guidance
• To Managers
• To Subject Advisors
• To Educators
• Monitoring and Evaluation
• Evaluation of the Assessment System
• Monitoring the implementation of the assessment
  process
• Moderation of the Assessment Outcome
• Retrospective Analysis

11
Setting of Standards

• The DoE and Umalusi have the joint responsibility
  to provide standards for good assessment.
• Standards for assessment must relate to the
  input, process and output of assessment.
• (a) Curriculum Standards
• Contained in the National Curriculum Statements
  (NCS) and are articulated as learning outcomes,
  assessment standards and the prescribed content.
• Proposed that the curriculum standards
  differentiate between knowledge and skills that
  are best measured through written examinations
  and those are best measured through SBA.
• Performance Standards
• Assists educators in judging how well learners
  have performed with reference to a specific
  curriculum standard.
• Exemplars of learner performance assist in
  establishing a common standard among assessors.
  

12
Setting of Standards

• Assessment System Standards
• Comprises a description of the policies,
  regulations, structures and procedures that make
  up assessment.
• This includes design and production of
  assessment instruments conduct of assessment
  awarding of marks and evaluation, feedback and
  review.
• Moderation Standards
• Minimum requirements for internal and external
  moderation must be established and adhered to by
  all providers of assessment.

13
Support and Guidance

• Support and Guidance to Managers
• Managers include managers at national,
  provincial, district and school levels.
• In most cases these managers are educators who
  have little or no management training or
  experience.
• Training should relate to contents of the policy
  and management of the policy.
• Support and Guidance to Subject Advisors
• Subject advisors are the arbitrators and
  mediators of policy.
• Support to subject advisors should include
• Policy and procedures relating to SBA.
• Content areas of the subject.
• Forms of assessment relating to SBA
• Subject Advisory role.
• Appropriate support material.
• Support and Guidance to Educators
• Success of SBA implementation is determined by
  the competency of educators and their commitment
  to this mode of assessment.
• Pre-service and In-service training.
• Development of item banks
• Exemplars of learner evidence

14
Monitoring and Evaluation

• Final leg in the quality assurance process.
• PEDs must take responsibility for the schools and
  districts in the province.
• The DoE will provide support,guidance and
  exercise national oversight over all PEDs.
• Umalusi will take responsibility for the final
  quality assurance .

Umalusi
DoE
9 PEDs
Districts
Schools
15
Monitoring and Evaluation

• (a) Evaluation of the Assessment System (Input)
• Includes an evaluation of all policies,
  regulations, structures, processes and tools used
  in SBA.
• District to evaluate the schools under their
  control.
• Provincial Head office to evaluate districts, by
  conducting sample school evaluations.
• DoE will evaluate the PEDs, by conducting sample
  district evaluations
• Umalusi will finally evaluate the assessment
  bodies.
• Umalusi report will serve as the final report on
  the status of the assessment systems of the nine
  PEDs.

16
Monitoring and Evaluation

• (b) Evaluation of the Implementation of the
  Assessment Process
• Focuses on the actual implementation of the
  assessment process.
• Covers the following aspects
• Ensuring SBA is implemented at schools.
• Ensuring compliance with policy.
• Ensuring that assessment tasks are designed in
  accordance with subject assessment guidelines.
• Ensuring that assessment tasks are administered
  in accordance with the principles of good
  assessment.
• Ensuring that the assessment evidence is marked
  according to assessment guidelines.
• Ensuring that marks are computed according to
  policy.

17
Monitoring and Evaluation

• (c) Moderation of the Assessment Outcome
• IPO model pre-supposes that if the input and the
  process have been thoroughly quality assured, the
  outcome should demonstrate a high reliability.
• Assessment tasks that mirror the examinations
  must be separated from the other SBA tasks (tasks
  that measure skills).
• SBA assessment tasks that mirror the examinations
  can be statistically moderated to the
  examinations, allowing for a 5 10 tolerance
  range.
• The other skills based tasks must be moderated
  through qualitative processes, which should occur
  at the four tiers of organisation
• School Level.
• Cluster level.
• Provincial Level
• National level
• Face moderation limited to skills based tasks.
• Feedback provided at each level of the moderation
  process

18
Monitoring and Evaluation

• (d) Retrospective Analysis
• Review of the
• assessment procedures
• assessment judgments
• assessment outcomes.
• Improvement of areas of weakness.
• Feedback

19
Challenges

• Making quality an integral part of each process
  and phase.
• Provision of appropriate human resources at each
  level.
• Effective training of educators.
• Common understanding of clearly articulated
  standards.
• Disparate resource and competency levels across
  schools based on their previous history

20
Conclusion

• Statistical moderation does not have to be the
  sole determinant in the quality assurance of SBA.
• Statistical moderation has a role to play in the
  correlation and moderation of assessment
  constructs that are similar.
• Qualitative data from the inputs, process and
  outputs of assessment will provide useful
  information in the monitoring and evaluation of
  the final assessment outcome.
• Clearly defined standards should be the starting
  point in the quality assurance process.