Fairness as an issue in school-based assessment

1
Fairness as an issue in school-based
assessment
University of Hong Kong Faculty of Education
Inaugural Seminar SeriesEnglish Language
School-based Assessment Integrating Theory and
Practice, Jan 9th 2006

• Liz Hamp-Lyons, Honorary Professor
• Faculty of Education, University of Hong Kong

2
Changing assessment cultures

• The official adoption in Hong Kong of the UK
  Assessment Reform Groups (1998) distinction
  between assessment for learning, and assessment
  of learning has stimulated the beginnings of a
  major paradigm shift
• Hong Kong schools are moving from a culture of
  testing to a learning and assessment culture
  (Hamp- Lyons 1999 in press).

3
Changing assessment cultures in Hong Kong

• http//cd.emb.gov.hk/basic_guide/BEGuideeng0821/ch
  apter05.html
• Based on the beliefs that every student is
  unique and possesses the ability to learn, and
  that we should develop their multiple
  intelligences and potentials there should be a
  change in assessment practices and schools should
  put more emphasis on 'Assessment for Learning' as
  an integral part of the learning, teaching and
  assessment cycle

4
Changing assessment cultures in Hong Kong

• In other words, teachers should use
  assessments (e.g. as simple as effective verbal
  questioning, observation of student behaviour)
  and provide immediate feedback to enhance student
  learning in everyday classroom lessons. The focus
  is on why they do not learn well and how to help
  them to improve rather than just to use
  assessments to find out what knowledge students
  have learned.

5
Theory and practice

• The HKU Faculty of Educations SBA team is
  working on an evolving but coherent programme of
  research and evaluation, professional
  development, and system-wide support (Davison
  2005)
• Overall, the teams work in SBA aims to carry out
  both basic research (addressing fundamental
  theoretical problems in language assessment) and
  applied research (responding to the needs of
  education communities)

6
The HKCEE English SBA initiative

• SBA is integrated into the teaching and learning
  process, with teachers involved at
• all stages of the assessment cycle
• ? planning the assessment programme
• ? identifying and developing appropriate
• assessment tasks
• ? overseeing the assessment process
• ? making final judgments

7
Fairness theory and practice

• In theory, fairness in an assessment for learning
  context means that every learner has the highest
  quality opportunity possible for demonstrating
  that/what s/he has learned
• However, theory of what fairness is remains
  underdeveloped and fragmented different kinds of
  theorists take different positions

8
Contexts for fairness

• In practice, fairness in an SBA context means
  that the right and responsibility of being fair
  lies with the teacher (see slide 4)
• However, in practice different stakeholders have
  different understandings and different
  expectations of what makes something fair

9
Fairnessthe Hong Kong context

• Research studies have found that assessment
  innovation was severely constrained by
  traditional school culture and by teacher,
  parental and student expectations (Cheung Ng
  2000 Carless 2001 Adamson Davison 2003
  Davison Tang, 2003 Hamp-Lyons, Chen Mok
  2000).
• Many teachers reported they cannot assess,
  only mark. They feel unable to make a
  difference in teaching and learning, to respond
  to individual needs, because of community
  expectations of convergence and commonality.
  Teachers feel their assessment processes are
  expected to change, without fundamental purposes
  being explicitly challenged. Such role conflict
  results in increasing stress and a decline in
  perceived teacher expertise.

10
Teachers understandings of fairness

• R How? How is it fair?
• T1 It can help the students better. The teachers
  are here to help. It is more encouraging because
  it lasts for a period.

11
Teachers understandings of fairness

• T1 All the ideas are good but still there are
  loop-holes. The only thing Im afraid is what if
  I am the assessor and my relatives are going to
  be the students, so how can we check or ensure
  that my relatives will not be assessed by me and
  there will be absolute fairness. I think its a
  concern to the public.

12
Teachers understandings of fairness

• T2 I found HK students really concerned about
  the issue of fairness as well, so I mean.
  evaluation is going to have to be as standardized
  as possible, I think it is going to be a big
  problem for a lot of teachers and students to
  try, you know, not just among different schools
  but even within one school, different classes
  how do you standardize it in a way thats fair to
  everyone?

13
Teachers understandings of fairness

• R What are you trying to standardize?
• T2 Wow, just the assessment criteria, because as
  you said it does count. And I found the HK
  students as soon as they know these are real
  marks and these marks are really counted for
  something, then they get motivated right away
  first of all we have to go through some kind of
  learning curricula adjustment to get the teachers
  and the students used to this sort of style and I
  mean that will be expected.

14
Teachers understandings of fairness

• T2 such a large window where is not so fair,
  right, if you know what Im trying to say, the
  differences between the schools and different
  banding of schools and then even schools like
  ours, the difference between the classes.

15
Fairnessthe system?

• T3 Yeah, I think, you know, I think basically
  the EMB whenever they need to promote certain
  changes to the education system, they should
  really think of, you know, is there really a need
  to change? If there is a need to change, think of
  how to make the changes smoother from one system
  to another and also what they can do, what they
  can do for both the teachers and the students
  because you know, the teachers are working on the
  front line, right?

16
T4 As I have mentioned earlier, I would say as a
teacher, a school teacher, examination in the
school usually shares two purposes. One is
assessing them the other is teaching... In fact
every time I teach them, not just assessing them.
Because after their examination, I give them some
feedback, right after the examination, group
after group. It really helps them to improve and
so thats more still I think the SBA component
is better because it can encourage students to do
better. In a way, I think it will affect the
atmosphere of the school, the learning
atmosphere. The students think that if they learn
harder, I mean they can get some marks in
advance. And I think it is the right direction.
17
The need for managed change

• Teachers cannot be expected to change their
  culture of assessment without the support of
  their panel, the school Principal, the EMB/CDI
  and the HKEAA
• Teachers also need the support of targeted
  professional development in new assessment
  cultures
• Messages about the purpose and value of change
  also need to reach parents and the public

18
Constraints on teachers concepts of fairness

• These constraints stem partly from the inadequate
  attention that is paid to educational assessment
  in professional training (not only in Hong Kong
  but almost world-wide), but also
• From outside influences as Pryor and Lubisi
  (2002) report in research on AfL in Ghana and
  South Africa, worry is extrinsic rather than
  intrinsic.

19
What is fairness?

• Stobart (2005) makes the point that fairness is
  fundamentally a sociocultural, rather than a
  technical, issue. He says that
• Fair assessment cannot be considered in isolation
  from both the curriculum and the educational
  opportunities of the students.
• Fairness is a justice that goes beyond acting in
  agreed upon ways and seeks to look at justice of
  the arrangements leading up to and resulting from
  those actions. (p.1)

20
What the 2007 English SBA does

• With school-based assessment and assessment for
  learning, the teachers role is foregrounded.
• The teacher is mediator in the assessment
  process.
• An assessment that puts a teacher and a known
  student together, that encourages mediation, also
  encourages diagnostic, interactive and
  collaborative characteristics in the assessments,
  even those for summative purposes. 
• The guidelines booklet Introduction to the
  School-based Assessment Component booklet (SBA
  Consultancy Team, 2005, commissioned by the
  HKEAA) marks a shift in the assessor's role from
  that of an external "tester" with no
  responsibility to the student other than to make
  a reliable judgment, to that of a mediator and
  facilitator whose responsibility is to see that
  every student has the best possible opportunity.

21
In assessment for learning, the teacher becomes a
mediator and facilitator who is

• sensitive to different learners levels and needs
• able to make effective use of questioning,
  prompts and probes to evaluate as well as promote
  student learning
• able to help lower proficiency students to
  demonstrate their full range of oral skills
• able to challenge higher proficiency students.
• This form of assessment is commonly known as
  dynamic assessment (Anton 2003 Lantolf and
  Poehner 2004) or educative assessment (Wiggins
  2005), and is grounded in Vygotskys theory of
  the Zone of Proximal Development (ZPD) (1978).

22
Areas for clarification-1

• In a questionnaire to SBA teacher trainers, the
  question on which they were least agreed among
  themselves was
• To be fair to students, all students should be
  given the same amount of time for their task.
• (SD 1.61 on 6-point scale)

23
Views of fairness
Fairness in the current HKU examination system Fairness in the SBA component
Treat all students equally (vs. equitably) Treat each student individually (i.e. unequally)
Same task, same input, same conditions, same length of time for all Different arrangements for different students according to their language proficiency
Major aim To give a consistent grade to each student in an invariable settings for selection purposes Major aim To provide opportunity for all students to demonstrate their best. Learning culture for all ? sense of achievement.
24
Why is equal not fair?

• Within assessment for learning, fair assessment
  is defined as assessment that provides the best
  opportunities for all, that provides checks and
  balances to help all participants in the system
  feel that they and their peers are acting fairly
  or being assessed fairly

25
Why is equal not fair?

• Gender or age factors
• Confidence vs shyness
• Self-centred vs other-centred
• equality is a myth, a goal that is never
  attained research shows that all oral language
  assessors vary according to candidate
  characteristics (e.g. Brown 2003 Iwashita 2004)

26
Areas for clarification-2

• The other questions where SBA teacher trainers
  views were very varied was
• The main goal of the standardization session is
  to check the accuracy of the teachers marking.
• (SD 1.58 on 6-point scale where 1Disagree)

27
Standardization is not about accuracy---a

• standardisation is one of the areas of checks
  and balances provided in the HKEAA English SBA
  system
• Teachers meet together, look at/listen to/discuss
  student oral samples, the tasks students have
  done, etc
• They share their scores and talk about why they
  think a sample is at a level on each domain

28
Standardization is not about accuracy---b

• Sharing and talking are key processes that lead
  to understanding and to common ground
• Therefore a teacher may choose to adjust a score
  after such a discussion
• But the focus of the meetings is collegial
  support and sharing, not checking each others
  accuracy

29
Changing practice, changing theory

• Following Black et al (2003) the HKEAAs English
  SBA innovation is changing theory by changing
  practice
• Notions of fairness are being expanded and
  redefined
• The overpowering influence of the narrow concept
  of reliability is being replaced by a more
  appropriate one -- trustworthiness
• but the system has much to do to explain this to
  all levels of the community

30
In conclusion

• Messick (1999) discusses issues of
• Access
• Assessments that restrict the access to only some
  of a student population are not fair
• Quality
• Assessments that are not of validated high
  quality are not fair
• Student development
• Assessments that do not provide means for
  students to develop in the knowledge/skill
  assessed are not fair
• Public policy
• Assessments that exist within an inadequate
  public policy environment are unlikely to be fair

31
The fundamental fairness

• Overall the purpose of assessment is to improve
  standards, not merely to measure them. (Review
  of secondary education in England, 1993-1997.
  Government Office for Standards in Education
  OFSTED 1998).