Assessment Professional Learning Module 5: Making Consistent Judgements

1
Assessment Professional Learning Module 5
Making Consistent Judgements
2
Assessment OF learningoccurs when teachers use
evidence of student learning to make judgements
on student achievement against goals and
standards.

• Assessment FOR learning
• occurs when teachers use inferences about student
  progress to inform
• their teaching.

• Assessment AS learning
• occurs when students reflect on and monitor their
  progress
• to inform their future
• learning goals.

3
The Victorian Essential Learning Standards place
an imperative on us to make high quality
judgements about student learning.What does
that mean we should do?
4
Quality and trustworthiness in evidence of
learning requires both valid (fair, accurate,
appropriate) and consistent (reliable)
assessment judgements.
5
Valid

• Everyday synonyms
• - truthful
• - appropriate
• - justified
• - convincing
• - accurate
• - legally acceptable
• - fair

For valid assessment fairness, accuracy and
appropriateness matter.
6
Validity is a matter offairness, appropriateness
and accuracy

• Are the tasks constructed, presented and
  conducted so that all students have an equal
  chance of demonstrating their learning against
  the standards?
• Do the tasks represent all the valued learning
  you want the students to have undertaken? Do they
  assess a sample of all the important concepts or
  all the important standards?
• Are the assessment tasks probing the students
  thinking to the depth you want - or are they able
  to get away with a superficial understanding?

7
Validity is a matter offairness, appropriateness
and accuracy

• Are the assessment tasks monitoring what you
  think they are (and not some intervening
  prerequisite skill, or conceptual understanding)?
• In particular, do the assessment tasks require
  specific knowledge or skills which some students
  may not have, and which have not been explicitly
  taught?
• (e.g. how to read, draw a graph or google?)

8
Assessment is valid if it

• assesses appropriate content and standards
• assesses the important standards
• provides information which is useful for some
  valuable purpose (for/as/of)
• is assessed with sufficient accuracy
• is fair to all students.

9
Consistency is a matter of reliability

• How confident can you be that judgements that you
  have made are not significantly affected by
  chance factors such as
• how the student was feeling on the day?
• how the assessor was feeling on the day?
• who the assessor was?
• luck in being assessed on some things and not
  others?
• luck in that the mode of assessment suited the
  student particularly well (or didnt)?

10
Examples of everyday use of the term reliability

• A set of kitchen scales is reliable if ...
• A clock is reliable if
• A train is reliable if
• A friend is reliable if
• A full-back is reliable if
• An assessment task is reliable if (what?)

11
Definition of consistency

• The consistency of an assessment is the
  reliability with which it assesses whatever it
  assesses.
• It is the reproducibility that is the focus.

12
The assessor matters in making consistent
judgements.We are human and our judgements may
be influenced by many factors besides the actual
standard of work. INTER-rater reliability
if another assessor judged the work, would the
student be awarded the same result?
INTRA-rater reliability if the same assessor
judged the work on another day would the result
be the same?
13
Consistent assessment

• An assessment is consistent if the scores that
  students get are reliable
• from one occasion to the next
• from one form of assessment to another
• from one assessor to another.

14
The difference between consistency and validity

• Consistency (or reliability) is a technical
  question. Will work be judged in the same way
  every time, regardless of who assesses it, or
  how, or when?
• Validity has both a technical element (does the
  assessment accurately judge what it says it
  judges?) and a philosophical one (is the
  assessment appropriate and fair?).
• Validity is an indication of the value and
  correctness of what we do.

15
Consistency, alone, is not sufficient.Assessment
must also be valid (fair, appropriate and
accurate).
16
Each step in the assessment process involves

• the possibility of error
• value-driven decisions
• assessment processes that may disadvantage
  some types of students.
• Use collaborative planning processes,
  transparency of assessment tasks, criteria,
  rubrics and variety to maximise valid and
  consistent judgements.

17
Strategies and protocols

• For valid and consistent judgements of student
  learning progression, we need to put in place
  strategies such as
• common assessment tasks planned collaboratively
• shared design processes for assessment tasks
  and for rubrics
• comparing work with exemplars (e.g. assessment
  maps)
• cross-marking of sample assessment tasks
• moderation and consistency protocols.

18
Critical stages of the Moderation Process

• Developing a common understanding of the
  standards to be assessed and the purpose of
  assessment
• Drafting the assessment task requirements
• Drafting the criteria - or marking scheme
• Sample assessing
• Final assessing.
• Activity 5-4A further defines this process.

19
The final assessing stage of moderation may
involve

• assessors meeting together
• sharing of assessed work samples
• discussion of queries
• cross-marking each others work samples
• discussion of borderline or special
  circumstance cases.

20
When deciding HOW to make more valid and
consistent judgements efficiency issues must
also be considered

• time efficient (yours and your
  students!)
• learning efficient (maximises learning)
• teaching efficient (reduces wastage)
• cost efficient
• ??? other factors

21
Making consistent judgements is important.
But consistency is not enough - we have to make
valid judgements as well.