B

1

Into Europe - European Standards in Language
Assessment Conference 9-10 February 2006
The BGF Linking Experience

• Bánóczy Erika
• Benke Eszter
• Budapesti Gazdasági Foiskola
• Nyelvvizsga és Továbbképzo Központ

2
Harmonizing national examinations with the
Common European Framework of Reference (CEF)

• Background to the project
• The origins of the existing level system
• The Strasbourg project
• Case Studies (WIP)
• Objectively scored tasks
• Subjectively scored tasks
• Examiners familiarity with the CEF
• LSP and the CEF

3
Background
Language examination levels according to the
Accreditation manual (1999), p.36.
4
Background
Language examination levels according to the
Accreditation manual (1999), p.33.

• Intermediate language proficiency corresponds to
  the intermediate level as suggested by the
  Council of Europe intermediate level (Vantage
  Level) and extends into the lower part of
  advanced level (Operational Proficiency).

5
Language examination levels accredited in 2000
6
Background
Modified language examination levels according to
the new Accreditation manual (2004), p.25.
7
Background
Language examination levels according to the
Accreditation manual (2004) p.8.

• The state-accredited examination system intends
  to harmonize the theory and practice of national
  and international (European) language
  examinations. This effort is manifest in the
  transferability of the Hungarian three-level
  system and the more detailed and up-to-date level
  system of the Council of Europe.
• (Work towards harmonization is still in progress.)

8
Levels to be harmonized
?
Advanced
Intermediate
Elementary
?
?
9
Aim of the BGF project

• Piloting the Manual
• Empirical validation of original levels
• Training of experts
• Modification of existing levels
• Benchmarking existing performance samples
• Creating new tasks based on new levels

10
Seminal documents and materials

• Relating Language Examinations to the Common
  European Framework of Reference for Languages
  learning, teaching, assessment. Manual (2003)
• Reference Supplement (2004)
• Nyelvvizsgák illeszkedése a Közös Európai
  Referenciakerethez (2005)
• Case Studies
• Calibrated performance samples

11
Procedures to relate examinations to the CEF

• Familiarisation with the CEF
• Specification
• Standardisation
• Empirical validation
• Reporting the results

12
Work completed so far

• 3 stages
• English project
• participants (anchor persons)
• procedures
• German project
• large-scale examiner training
• time and materials used

13
Breakdown of working hours in the first stage (1)
14
Breakdown of working hours in the first stage (2)
15
Technical equipment used
16
Objectively scored tasks

• Familiarisation with CEF and DIALANG
• scales
• Study of calibrated sample tasks (if and where
  available)
• Standard setting modified Angoff method
• Collation and tabulation of results
• Comparison of qualitative and quantitative data

17
Standard-setting grid for a reading comprehension
task
18
Data collection 1
19
Data collection 2
20
Data analysis

• Method applied
• Comparison of qualitative and quantitative data

21
(No Transcript)
22
Problematic item
Item No CEF level Facility value logit
item 11 B2 85 -1.09
Alpha if item deleted

• .787

Alpha .771
23
External validation

• Criterion test
• BEC Vantage level
• Content analysis to establish comparability
  (ALTE content analysis checklist)
• Correlation
• comparison of statistical data

24
Results descriptive statistics
BEC Vantage (N107) NYTK Intermediate (N 106)
Mean SD S.E. mean 11.18 2.98 .288 10.84 3.422 .332
Mean facility value 61.99 54.24
Cronbach alpha 0.66 0.65
25
Results - correlation

• r0.59 (plt 0.01)
• after correction for attenuation r0.9
• considerable similarity between empirical and
  intuitively set levels of difficulty

26
Subjectively scored tasks

• Oral proficiency tests
• watching videotaped calibrated performance
  samples
• benchmarking own samples
• comparison of CEF based and original scores
• Writing
• benchmarking own samples
• comparison of CEF based and original scores

27
Writing selection of criteria
28
Results so far

• benchmarked intermediate performance samples in
  six languages
• accumulated experience and emerging expertise
  fairly thorough familiarity with the CEF level
  system

29
Survey on familiarity with the CEF

• 82 respondents
• 35 English
• 21 German
• 10 French
• 10 Spanish
• 4 Italian
• 2 Russian
• 18 yrs average teaching experience
• questionnaire
• 15 statements
• 5 point Likert-scale

30
Familiarity with the CEF - questionnaire

• I have heard of the CEF before.
• I know the Hungarian version of the CEF.
• I know the language version of the CEF I teach.
• I have my own copy of the CEF.
• I know the level system used in the CEF.
• I am familiar with the descriptors used in the
  CEF.

31
Use of the CEF - questionnaire

• I use the CEF for defining proficiency levels.
• I use the CEF in curriculum design.
• I use the CEF letter level system when choosing
  course books.
• I use the diagnostic self-assessment scales.

32
The benchmarking workshop and the future -
questionnaire

• I find the sorting tasks useful in getting
  familiar with the CEF levels.
• The CEF descriptors are easy to apply.
• It is possible to realistically assess students
  language proficiency with the help of the CEF
  scales.
• I find it feasible to harmonize our existing
  system of levels with the CEF levels.
• The CEF levels are becoming more and more common
  in the professional discourse/communication of
  those involved in education (teachers, students,
  parents, employers etc.).

33
Top 3

• I find the sorting tasks useful in getting
  familiar with the CEF levels. ( 4.1)
• I know the level system used in the CEF. (
  3.9)
• I find it feasible to harmonize our existing
  system of levels with the CEF levels. (
  3.8)

34
Bottom 3

• I use the CEF in curriculum design. ( 2.3)
• I have my own copy of the CEF. ( 2.3)
• I use the diagnostic self-assessment scales. (
  2.2)

35

• ESP and the CEF

36
ESP and the CEF

• Some intriguing issues
• L(SP) competence
• definitions of LSP
• Swales, 1985
• Strevens, 1988
• EAP-EOP (Robinson, 1991)
• general-specific continuum (Dudley-Evans St
  John, 1998)

37
ESP and the CEF

• Some intriguing questions
• Which ESP/LSP definition is our system based on?
• How are ESP/LSP skills different from EGP skills?

38
ESP and the CEF

• General purpose language testing vs LSP testing
• authenticity of task
• interaction between language knowledge and
  specific purpose content knowledge
• (Douglas, 2000)

39
ESP and the CEF

• specific aspect of the specifications (specific
  specifications)
• specific aspect of the operationalisation of the
  construct
• specific aspect of the tasks

40
ESP and the CEF

• special
• job
• field
• vocation
• terminology

41
ESP and the CEF

• Table 1. Common Reference Levels global scale (
  p.24)
• B2 Can understand the main ideas of complex text
  on both concrete and abstract topics, including
  technical discussions in his/her ?eld of
  specialisation.

42
ESP and the CEF

• Table 2. Common Reference Levels self-assessment
  grid (pp. 26-27)
• A2 Spoken Production
• I can use a series of phrases and sentences to
  describe in simple terms my present or most
  recent job.
• B1 Spoken Interaction
• I can enter unprepared into conversation on
  topics that are familiar
• (e.g. work).
• C1 Reading
• I can understand specialized articles even when
  they do not relate to my field.
• C2 Reading
• I can read with ease virtually all form of
  written language such as specialised articles.

43
ESP and the CEF

• Sustained monologue (p.59)
• A2 Can describe his/her family, living
  conditions, educational background, present or
  most recent job.
• Creative writing (p.62)
• B1 Can write about everyday aspects of his/her
  environment, e.g. a job in linked sentences.

44
ESP and the CEF

• Overall listening comprehension (p.66)
• B2 Can understand standard spoken language, live
  or broadcast, on both familiar and unfamiliar
  topics normally encountered in personal, social,
  academic or vocational life.
• C1 Can understand enough to follow extended
  speech on abstract and complex topics beyond
  his/her own
• ?eld

45
ESP and the CEF

• Listening as a member of a live audience (p. 67)
• C2 Can follow specialised lectures and
  presentations employing a high degree of
  colloquialism, regional
• usage or unfamiliar terminology.
• Overall reading comprehension (p.69)
• A2 Can understand short, simple texts on familiar
  matters of a concrete type which consist of high
  frequency everyday or job-related language.
• B1 Can read straightforward factual texts on
  subjects related to his/her ?eld and interest
  with a satisfactory level of comprehension.
• C1 Can understand in detail lengthy, complex
  texts, whether or not they relate to his/her own
  area of speciality, provided he/she can reread
  dif?cult sections.

46
ESP and the CEF

• Reading for information and argument (p.70)
• B2 Can obtain information, ideas and opinions
  from highly specialised sources within his/her
  ?eld. Can understand specialised articles outside
  his/her ?eld, provided he/she can use a
  dictionary occasionally to con?rm his/her
  interpretation of terminology.
• Reading instructions (p.71)
• B2 Can understand lengthy, complex instructions
  in his ?eld, including details on conditions and
  warnings,
• provided he/she can reread dif?cult sections.
• C1 Can understand in detail lengthy, complex
  instructions whether or not the instructions
  relate to his/her own area of speciality

47
ESP and the CEF

• Overall listening comprehension (p.66)
• B1 Can understand straightforward factual
  information about common everyday or job related
  topics
• B2 Can understand standard spoken language, live
  or broadcast, on both familiar and unfamiliar
  topics
• normally encountered in personal, social,
  academic or vocational life.
• C1 Can understand enough to follow extended
  speech on abstract and complex topics beyond
  his/her own ?eld

48
ESP and the CEF

• Understanding a native speaker interlocutor
  (p.75)
• C1 Can understand in detail speech on abstract
  and complex topics of a specialist nature beyond
  his/her own ?eld
• C2 Can understand any native speaker
  interlocutor, even on abstract and complex topics
  of a specialist nature beyond his/her own ?eld

49
ESP and the CEF

• A2 B1 job, work,
• B2 C1 own field, speciality,
  specialised,
• C1 C2 beyond own field

50
Useful/beneficial experience

• Internal validity of tests is vital for external
  validation
• continuous internal validation
• Ongoing local and global validation

51
Useful/beneficial experience

• Apparent need for
• intensive familiarization
• adequate number of experts
• a keen eye for the differences between the
  original and the translated versions of the CEF
• harmonization within and across languages
• Further challenges
• bilingual tasks
• LSP examinations

52
C2 Mastery
C1 Effective Operational Proficiency
B2 Vantage
B1 Threshold
A2 Waystage
A1 Breakthrough
Cooperation and sharing of experience to make it
common.
53

• May we all continue to learn from each other.

54

• Thank you.
• banoczy_at_bgf.nyelvvizsgak.hu
• benke_at_bgf.nyelvvizsgak.hu