Nick Saville

1
Investigating the impact of language assessment
systems within a state educational context
Nick Saville Bridging the gap between theory
and practice EALTA Krakow May 2006
2
Investigating the impact of language assessment
systems within a state educational context
Nick Saville Bridging the gap between theory
and practice EALTA Krakow May 2006
3
Outline

• Background - a personal perspective
• 1980s
• Bachman early 1990s
• The literature on washback/impact
• early work and recent progress
• gaps? where next?
• Analysis of three case studies what can be
  learnt?
• Towards a comprehensive model of impact
• Applying the model in a state educational context
• the Asset Languages Project

4
Background

• The 1980s a personal perspective
• assessment in Italian universities
• entrance exams in Japan
• the influence of TOEIC/TOEFL e.g. in Japan/Korea
• developing Cambridge exams
• Tests affect individuals and society!
• How can this be managed better?
• What is needed to do a better job?

5
Background 1987-1990 Japan
Considerations in developing fair tests
V
Test
Practicality?
R
The art of the possible
6
Practicality
V
Practicality in Language Testing an
educational management model
Test
R
P

• Main argument test development is a form of
  educational innovation - and needs to be managed
  as such
• ... achieving a balance between the purpose of
  the test, its validity for the purpose, the
  required reliability for the purpose and the
  constraints imposed by the context is essentially
  the task facing the test designer .
• Saville (1990), University of Reading - based on
  test development project Japan, 1987-9

7
Practicality

• Aspects of Practicality within a context and
  educational setting
• Acceptability
• Applicability
• Availability
• Difficulty
• Economy
• Interpretability
• Relevance
• Replicability

a principled approach to Practicality should
provide the test designer with the means of
approaching test development so that a suitable
balance can be achieved without overlooking
factors which cause possible solutions to fall
down in practice.
V
Test
R
P
8
Putting the test into context
The aim is not only to encourage good testing
practice, but to prevent bad tests being produced
.... ... a bad test is not only one with low
reliability and dubious validity but also one
which has a damaging backwash on the curriculum.
Saville 199011-13
A logical consequence . is that ethicality will
be achieved as a result .. . this is because
any test which is produced should be appropriate
to the educational context in which it is to be
used and the effect on learners and institutions
will be a major consideration.
V
Test
R
P
9
Putting the test into context
V
Test
R
P
10
Impact Ripples
V
Test
R
P
11
Impact Ripples
Local Impact micro level
V
I
Test
R
P
12
Impact Ripples
Wider Impact (macro level)
V
I
I
I
I
Test
P
R
13
Usefulness as overall validity

• U V R I P
• Bachman - Cambridge 1990/91

14
Usefulness as overall validity

• U V R I P
• Bachman and Palmer, 1996 U Cv A I R
  I P

Developing useful tests, fit for
purpose Balancing the test qualities
15
Starting to develop a model

• 1993 1995
• Using VRIP to develop and revise exams
• e.g. IELTS 1995
• The IELTS impact project

16
The literature on washback/impact

• Readings in the language testing literature
• Hamp-Lyons (1989)
• Wall and Alderson (1993) Does washback exist?
  Etc..
• Bailey (1996)
• Hamp-Lyons (1997)
• Watanabe (1997)
• Cheng and Watanabe (eds) (2004)
• Recent PhD studies and subsequent books based on
  research conducted in the 1990s
• Cheng (SILT 21 - 2005)
• Wall (SILT 23 - 2005)
• Green (2004 SILT forthcoming 2007)
• Hawkey SILT 24 (forthcoming - 2006)
• Current work in Lancaster, ETS, UCLA, Cambridge
  etc.

17
The literature on washback/impact

• So
• Impact is relatively new in the field of language
  assessment - an extension on the notion of
  washback and related to ethicality
• It is now considered to be of growing importance
• It is part of a validity argument and evidence
  needs to be provided
• Broadly speaking there is consensus
• impact deals with wider influences and includes
  the macro contexts - tests and examinations in
  society
• washback is an aspect of impact related to the
  micro contexts of the classroom and the school
• BUT
• The dynamics between the micro and macro contexts
  mean that this is a complex rather than a simple
  relationship
• - a complex dynamic system

18
The literature on washback/impact

• And currently
• there is no comprehensive model of test or
  examination impact within educational contexts
• impact has not yet been fully integrated into an
  approach to test development and validation in a
  systematic way

19
Three case studies 1995 to 2004

• Case 1 - the world-wide survey of the impact of
  IELTS
• a starting point for the work and the original
  model for what has followed
• a conceptualisation of impact and
  design/validation of suitable instruments to
  investigate it
• Case 2 - the Italian PL2000 project
• an application of the model within a macro
  educational context
• an initial attempt at the applying the approach
  on a limited basis within a state educational
  context
• Case 3 - the Florence Learning Gains Project
• an extension and re-application of the model
  within in a single school context
• at the micro level focusing on individual
  stakeholders within a single language teaching
  institution

20
Learning from the case studies

• What can be learned using these specific impact
  projects as meta-data?

21
Learning from the case studies

• Three key factors of contemporary educational
  systems need to be accounted for
• the nature of complex dynamic systems
• the roles that stakeholders play within such
  systems
• the need to see assessment projects as
  educational innovations within the systems and to
  manage change effectively

22

• the nature of complex dynamic systems

23
2. the roles that stakeholders play
24
Hybrid Model of the Diffusion / Implementation
Process
3. the need to see assessment projects as
educational innovations and to manage change
effectively

• See Wall (2005)
• a case study using insights from testing and
  innovation theory
• E.g. Henrichsen (1989)

Antecedents
Process
Consequences
25
Learning from the case studies

• When applied to language assessment two key
  factors also need to be accounted for
• the nature of language itself as a
  socio-cognitive phenomenon (the latest views on
  validity)
• the nature of the test development and validation
  process
• from conception to routine data collection and
  analysis
• Impact research, therefore is no different from
  any other kind of validation activity........

26
1. A SOCIO-COGNITIVE FRAMEWORK
Messick Bachman Kane Mislevy Weir etc.
27
A SOCIO-COGNITIVE FRAMEWORK
The testing system
Construct
28
The contexts
Learning contexts
Testing contexts
Use of results contexts
29
Impact
30
2. Model of the Test Development Process
31
Model of the Test Development Process
Identifying stakeholders and their needs
Linking these needs to the requirements of test
usefulness - including predicted impact
- theoretical
- practical
Long term, Iterative Processes - a key feature of
validation
32
Involvement of the stakeholder constituency

• E.g. during test design and development
• presentation and consultation to do with
  specifications and detailed syllabus designs
• professional support programmes for institutions
  and individual teachers/students etc. who plan to
  use the examinations
• training and employment of suitable personnel
  within the field to work on all aspects of the
  examination cycle to be question/item writers,
  to act as examiners, etc.

33
After an examination becomes operational

• Procedures also need to be in place to routinely
  collect data which allows impact to be estimated
  
• e.g.
• who is taking the examination
• (i.e. a profile of the candidates)
• who is using the examination results and for what
  purpose
• who is teaching towards the examination and under
  what circumstances
• what kinds of courses and materials are being
  designed and used to prepare candidates
• what effect the examination has on public
  perceptions generally
• (e.g. regarding educational standards)
• how the examination is viewed by those directly
  involved in educational processes
• (e.g. by students, examination takers, teachers,
  parents, etc.)
• how the examination is viewed by members of
  society outside education
• (e.g. by politicians, business people, etc.)

34
Towards a comprehensive model

• How can these considerations be combined to
  produce a comprehensive, integrated model?

35
Next phase applying the model

• Asset Languages within the UK educational context

36
Contacts
www.assetlanguages.org.uk
saville.n_at_cambridgeESOL.org www.cambridgeesol.
org/rs_notes