Designing and Developing Useful Language Tests

1
Designing and Developing Useful Language Tests

• Lyle F. Bachman
• Department of Applied Linguistics TESL
• University of California, Los Angeles

2
Uses of Language Tests

• to make inferences about individuals language
  ability
• to make predictions about individuals ability to
  use language in contexts outside the test itself
• to make decisions about individuals

3
Correspondences between Language Test Performance
and Language Use
4
(No Transcript)
5
Language Ability of the Language User
Language Ability of the Test Taker
6
(No Transcript)
7
A Framework of Task Characteristics

• Language Use Task
• an activity that involves individuals in using
  language for the purpose of achieving a
  particular goal in a particular situation
• (Bachman Palmer, 1996, p. 44)

8
A Framework of Task Characteristics

• Target Language Use (TLU) Domain
• a set of specific language use tasks that the
  test taker is likely to encounter outside of the
  test itself, and to which we want our inferences
  about language ability to extrapolate
• (Bachman Palmer, 1996, p. 44)

9
A Framework of Task Characteristics

• Task Characteristics
• Setting the physical circumstances under which
  either language use or language testing takes
  place. Includes the physical characteristics of
  the setting, the participants involved, and the
  time of the task.

10
A Framework of Task Characteristics

• Task Characteristics
• Rubric the context for the task includes those
  characteristics that provide the structure for
  the task, and constrain how language users or
  test takers are expected to respond to these tasks

11
A Framework of Task Characteristics

• Task Characteristics
• Input material contained in the task, which
  test takers need to process in some way, and to
  which they are expected to respond.

12
A Framework of Task Characteristics

• Task Characteristics
• Expected response the language use that is
  expected, given the way in which the rubric, or
  context, for the task is configured, and the
  particular input that is provided

13
A Framework of Task Characteristics

• Task Characteristics
• Relationship between input and response
• Reactivity the degree of reciprocity, or
  interaction involved,
• Scope the amount and range of input that needs
  to be processed in order to respond, and
• Directness the extent to which the response can
  be made by using information in the input by
  itself, or whether the language user or test
  taker must also rely on information in the
  context or in his or her own real world knowledge.

14
Describing Target Language Use Tasks

• 1. Identify the TLU tasks in the relevant domain.
• 2. Select TLU tasks as a basis for designing test
  tasks.
• 3. Describe these tasks in terms of their task
  characteristics.

15
Describing Target Language Use Tasks

• 1. Identify the TLU tasks in the relevant domain.
• 2. Select TLU tasks as a basis for designing test
  tasks.
• 3. Describe TLU tasks in terms of their task
  characteristics.

16
TLU Domain
PR Department of a Phone Company
Task 1 writing promotional material
Task 2 phone discussions with clients, co-workers
Task 4 responding to customer complaints in
writing
Task 3 responding to customer complaints on
the phone
Task 5 giving complaint information to
co-workers by phone
Task 6 giving complaint information to co-workers
in writing
17
Developing Test Tasks

• Modify TLU task types for use as test tasks
• Create original test tasks

18
(No Transcript)
19
Developing Test Tasks

• Modify TLU task types for use as test tasks
• Create original test tasks

20
(No Transcript)
21
Example Test Development Project

• Project setting
• telephone company needs to hire new employees

22
Describing Target Language Use Tasks

• 1. Identify the TLU tasks in the relevant domain.
• 2. Select TLU tasks as a basis for designing test
  tasks.
• 3. Describe these tasks in terms of their task
  characteristics.

23
Example TLU Domain
PR Department of a Phone Company
24
Selecting TLU Tasks for Use in Designing Test
Tasks

• Some TLU tasks may be essential to performing the
  job

25
Example TLU Domain
PR Department of a Phone Company
Task 1 writing promotional material
Task 2 phone discussions with clients, co-workers
Task 3 responding to customer complaints on
the phone
Task 5 giving complaint information to
co-workers by phone
26
Selecting TLU Tasks for Use in Designing Test
Tasks

• Some TLU tasks may not require the areas of
  language ability we want to measure.
• Some TLU tasks may not be appropriate for all
  test takers.

27
Describing Target Language Use Tasks

• 1. Identify the TLU tasks in the relevant domain.
• 2. Select TLU tasks as a basis for designing test
  tasks.
• 3. Describe these tasks in terms of their task
  characteristics.

28
TLU Domain
PR Department of a Phone Company
Task 1 writing promotional material
Task 2 phone discussions with clients, co-workers
Task 4 responding to customer complaints in
writing
Task 3 responding to customer complaints on
the phone
Task 5 giving complaint information to
co-workers by phone
Task 6 giving complaint information to co-workers
in writing
29
Developing Test Tasks

• Modify TLU task types for use as test tasks.
• Create original test tasks.

30
Developing Test Tasks

• Modify TLU task types for use as test tasks
• Create original test tasks

31
(No Transcript)
32
Developing Test Tasks

• Modify TLU task types for use as test tasks
• Create original test tasks

33
(No Transcript)
34
Bachman-Palmer Approach to Language Test
Development

• Provides test developers with a
• theoretically grounded and principled
• basis for developing and using
• language tests.

35
Bachman-Palmer Approach to Language Test
Development

• Provides test developers with an
• understanding that will enable them
• to make their own decisions about
• either selecting or developing a
• language test that will be useful for
• their particular situation.

36
Bachman-Palmer Approach to Language Test
Development

• Requires test developers to consider the
• fundamental correspondence between
• language test performance and non-test
• language use.

37
(No Transcript)
38
Using Needs Analysis to Identify TLU Tasks

• 1. Identify stakeholders.
• 2. Develop procedures for gathering information
  about tasks.
• 3. Gather information about the domain and tasks.
• 4. Analyze tasks in terms of their task
  characteristics.
• 5. Make an initial grouping of tasks into
  categories.

39
Test Usefulness

• Principle 1 Overall test usefulness, rather
  than individual qualities is to be maximized
• Principle 2 Individual qualities cannot be
  evaluated independently, but most evaluated in
  terms of their combined effect on overall
  usefulness

40
Test Usefulness

• Principle 3 Test usefulness and the appropriate
  balance among the different qualities can only be
  determined for a given specific testing situation.

41
Qualities of Usefulness

• Reliability
• Construct Validity
• Authenticity
• Interactiveness
• Impact
• Practicality

42
Describing Target Language Use Tasks

• 1. Identify the TLU tasks in the relevant domain.
• 2. Select TLU tasks as a basis for designing test
  tasks.
• 3. Describe these tasks in terms of their task
  characteristics.