PILOTING CROATIAN EXPERIENCE Tamara Kramaric Maras

1
PILOTINGCROATIAN EXPERIENCETamara
Kramaric Maras
2

• However well designed an examination may be, and
  however carefully it has been edited, it is not
  possible to know how it will work until it has
  been tried out on students.
• Alderson, Clapham and Wall.1995.Language Test
  Construction and Evaluation, CambridgeUniversity
  Press

3
(No Transcript)
4
CHALLENGES

• establishing a system
• number of pilot tests and items
• piloting tests or sets of items
• pre-testing before piloting
• finding the suitable population (size, time)
• administering the test
• interpreting the statistics
• establishing the control group

5
PLANNING

• how many pilot tests a year?
• where?
• population size and level?
• the most appropriate dates?
• budget?

6

• 2 pilot tests a year
• Slovenia , BiH
• May/November
• 35-60, L1, L2, L3
• bilateral activity, host country responsible for
  costs of accomodation/food
• a detailed rationale

7
PILOT TEST CONSTRUCTION AND ADMINISTRATION

• what should the pilot test look like?
• more items than in the real test?
• pre-piloting before the pilot?
• test administration
• student preparation and motivation

8

• resembling the real test as much as possible
  (number of tasks/items, layout, task types,
  rubrics, time)
• pre-piloting on available population in FLS
• administering the test ourselves
  standardisation session
• students need to understand the mutual benefits

9
INTERPRETATION OF STATISTICS

• satisfactory values of FV/DI?
• ALCPT as a factor
• anchor items

10

• L2 FV 70 - 90
• DI 0,2-0,4
• L3 FV 40 - 60
• DI 0,5 -1
• (bi-level tests of reading/listening)
• (population of 1/3 L3, 2/3 L1, L2)

11
CONTROL GROUP

• not enough candidates in FLS for L2 /L3 piloting
  in Croatia
• who else could be involved in piloting?

12

• establishing a control group
• volunteers (L2/L3 or ALCPT 80), cadets
• importance of motivation
• updating and extending the control group

13
BENEFITS OF PILOTING

• main validation process
• standardisation through moderation
• standardisation sessions (oral and written tests)
• new ideas
• backwash on teaching/learning

14

• ????