THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE (CNaVT)

1
THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE
(CNaVT)
Language testing and culture
B.C.vanOel_at_uva.nl
Universiteit van Amsterdam The Netherlands
Katholieke Universiteit Leuven Belgium
TBLT Honolulu September 2007
2
Question
Do you congratulate someone with or on their
birthday?
3
Answer
Neither! Its a trick question. We dont use to
congratulate or congratulations in connection
with birthdays in English, except in very special
cases (18, 21, 100, etc.) we normally just say
Happy Birthday or Many happy returns,
instead. And we dont shake hands as we do so.
However, we can congratulate people on getting
a job, winning an award, or getting married, for
instance. And then a handshake is often part of
the ritual. With is never used with to
congratulate or congratulations its always
to congratulate (someone) on and
congratulations on ..
4
The Certificate of Dutch as a Foreign Language

• CNaVT (Certificaat Nederlands als Vreemde Taal)
• administration in more than 40 countries around
  the world from Madrid to Jakarta and from
  Warsaw to Rio de Janeiro
• 2500 candidates each year

5
Assess as direct as possible

• construct on the basis of needs analysis
• authenticity 6 exams represent 6 different TLU
  domains
• task-based exam tasks represent real life
  language use situations

6
Valid assessment
Uiterwijk Vallen (Language Testing 2005,
22) () the question is does the test measure a
particular construct for the different groups in
a valid way? In other words, does it measure, for
various subgroups, what is claims to measure?
7
The effect of culture
Marquardt Gillam (Language Testing 1999,
16) The most frequent problem related to
misinterpretation of assessment results is
related to cultural () differences between test
items, examiners and their () candidates.
8
Fairness to candidates
McNamara Roever (Language Testing. The Social
Dimension) An interpretative argument is only as
strong as its weakest link. It is the social
responsibility of testers to report efforts to
estimate the impact of these factors and to
control them.
9
Item bias analysis

• statistical procedures to find out if an item is
  shows DIF
• research the factors responsible for the DIF
• determine if the cause of DIF is construct
  (ir)relevant, leading to bias

10
CNaVT item bias research

• PTIT exam 2007
• 130 items, spread over 2 versions, 9 tasks per
  version
• Facet-analysis facet country
• Mantel-Haentzel analysis

11
CNaVT item bias research

• Facets
• empirically 1324 bias terms item country
• 68 interactions showed DIF
• 56 disadvantageous to subgroup
• 12 advantageous to subgroup

12
CNaVT item bias research
13
CNaVT item bias research
Example 1 Country Indonesia Item Question in
form What sort of object have you lost?
14
CNaVT item bias research
Example 2 Country Russia Item Question from
friend in letter () But I often think back
with nostalgia about my holiday in your country.
() And you? What did you do during your
holidays? Did you travel? Where to? Or did you
stay home and what did you do? ()
15
Our issues concerning DIF analysis

• Categorisation on the basis of nationality,
  region? No strict boundaries. Or what is
  culture?
• Determination that culture is the causing factor
  of variance so many intervening factors (for
  example socio-economical status)
• Determination that DIF is part of the construct gt
  tension

16
What we do

• awareness of our construct what (concerning
  culture) is relevant, what is irrelevant
• generate input on possible cultural bias from
  different sources in different stadia
• In case of suspicion of irrelevance DIF analysis

17
To conclude
More research Further define our
construct Communication to stakeholders Fairness
to candidates!!!
18
THE CERTIFICATE OF DUTCH AS A FOREIGN LANGUAGE
(CNaVT)
www.cnavt.org
B.C.vanOel_at_uva.nl
Katholieke Universiteit Leuven
Universiteit van Amsterdam
TBLT Honolulu September 2007