Developing Descriptors

1

• Developing Descriptors
• Brian North
• www.eurocentres.com
  www.eaquals.org
• bjnorth_at_eurocentres.com
  bnorth_at_eaquals.org

2
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

3
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

4
Conceptualisation

• CEFR
• CEFR descriptors observable, functional outcomes
  competence descriptors also mainly observable
  proficiency
• Interaction (BICS) / Production (CALP)
• Illustrative videos of 16-18 yr olds difficulty
  with BICS C2
• LoS more complex than modern languages
• Language aspects / non-language aspects
• Discourse emphasis genres cognitive skills
• Developmental linked to cognitive growth
• Far less known about LoS than modern languages
• 20 years experience with descriptors 1975-1995
• 20 years developing descriptive scheme 1975-1995

5
Characteristics of LoS or (C) ALP

• We/you know it involves more
• specific, formal, abstract
• explicit, detailed, conventionalised (
  expectations)
• cohesive and structured (e.g. sequencing)
• coherent (goal-oriented)
• planning, self-monitoring, internal feedback,
  editing
• rhetorical skills and structures, strategies
• BUT
• How much is really known about academic
  discourse?
• Reception of exposition by the teacher
• Interaction in class
• Production by the teacher
• To what extent are skills transversal a common
  core?

6
(No Transcript)
7
Need for Collaboration Research

• Vollmer
• Pooling expertise and materials
• Corpus of curricula and examination papers
• Classroom observation and research
• Interviews with teachers
• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Analyse textbooks

8
Need for Collaboration Research

• CEFR Preparatory Work
• Clarify concept 1975 (Threshold) 1992
  (Proposal)
• Experience with descriptors (BN 1983-93)
• Classroom discourse analysis (BN 1984-9)
• Involvement of stakeholders (Working Party
  1992-6)
• CEFR Project Design
• Analyse and align existing systems
• Interactive definition of categories with
  Authoring Group
• Swiss National Research Project
• Involvement of teachers in qualitative validation
  - Workshops

9
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

10
Key Questions I

• Relationship to
• content standards
• European Qualifications Framework
• CEFR
• Categories to be described
• Transversal categories as in Table 5 of ERDLE
  proposal (p52)
• Subcategories of Recep, Inter, Prod, Interp,
  Evaluation, Mediation?
• Cognitive skills strategies from Situation
  analysis (Beacco et al)
• What else?
• Style
• concrete-salient features (CEFR-style) / abstract
• Length including assumptions Can make a
  complaint B1
• broad-holistic / atomistic-analytic / both
  (Fleming)

11
Key Questions II

• Thresholds to be described
• expected language proficiency levels
• types of discourse
• stages of cognitive development
• strategies
• How to deal with difficult parts (non-language)
  e.g. Bildung
• consideration of others
• critical thinking, sound judgement and courage to
  express it?
• flexibility in thinking and argumentation

12
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

13
Construction

• Creating a classified bank of descriptors
• Collate / deconstruct all source systems
• Eliminate doubles, redundancy
• Identify gaps
• Editing and drafting
• Confirm style
• Harmonise use of verbs (not done in CEFR
  English!)
• Harmonise formulations
• Create variations (for missing levels)
• Author missing categories
• Organisation
• Classify with serial numbers
• Translation to key languages / check translations
  with plurilinguals

14
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Set Summarise developing proficiency

15
Qualitative Validation

• Analysis of teachers discussing proficiency
• Video of two learners
• Who is better? Why? Justify your choice
• Repertory grid analysis of categories teachers
  use to compare quality
• Sorting descriptors into categories
• Pile of (maximum 60) descriptors
• Set of (maximum 4) envelopes labelled with the
  relevant categories
• Discard envelope
• Tick ones that are clear, relevant and useful
• Sorting descriptors into levels
• Pile of (maximum 15) descriptors for same
  category
• Set of (CEFR 6) envelopes labelled with levels
• Discard envelope / Tick ones that are clear,
  relevant and useful

16
Quantitative Validation - Purpose

• To construct a scale from the descriptors for the
  core construct
• To bolt onto / link to this scale sets of
  descriptors for categories that prove to be less
  core areas
• To find out/confirm what level specific
  descriptors are
• To discover which descriptors do not work
• To confirm communality of the interpretation of
  the descriptors across
• Languages
• Regions / countries / systems
• Educational sectors

17
Quantitative Validation Steps

1. Identify good/best descriptors from the pool
   after the qualitative validation
2. Confirm the supposed level of these descriptors
3. Create a set of overlapping checklists of c50
   descriptors (like ELP checklists) each checklist
   targeted at a level
4. Define a rating scale Yes/No 0-4 for the
   descriptors
5. Identify classes at approximately the right level
   for each checklist
6. Arrange teacher assessment and/or self-assesment
   with the checklists
7. Collect minimum 150 examples of each checklist
8. IRT Rasch Model Rating Scale Analysis to build
   scale
9. Eliminate descriptors with 80 or 20- (Rasch
   problem)

18
Anchor Design CEFR (North 2000)
19
Recommended Design (after De Jong)
Data Collection
20
Vertical Scale of Descriptors
21
Vertical Scale of Descriptors
22
Extending the Core Scale I
23
Quantitative Validation - Prerequisites

• Construct is well-defined common understanding
  of what is being described/rated/scaled
• Descriptors are well-formulated, clear and
  relevant
• Teachers/learners are capable of making
  judgements about the areas concerned
• There is a solid anchor design in the data
  collection

24
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

25
Setting Thresholds between Levels

• Marking out equal intervals on the scale
• Identifying jumps in content described, gaps
  between clusters of descriptors
• Comparing to original scale author intention
• Comparing to Waystage, Threshold, Eurocentres,
  Cambridge exam levels
• Fine-tuning for equal intervals
• Checking for consistency, coherence

26
CEFR 3.6 Salient Characteristics A2

• The majority of descriptors stating social
  functions
• greet people, ask how they are and react to news
  
• handle very short social exchanges
• discuss what to do, where to go and make
  arrangements
• Descriptors on getting out and about
• make simple transactions in shops, banks etc.
• get simple information about travel and services

27
CEFR 3.6 Salient Characteristics B1

• Maintain interaction and get across what you want
  to
• give or seek personal views and opinions
• express the main point comprehensibly
• keep going comprehensibly, even though pausing
  evident, especially in longer stretches
• Cope flexibly with problems in everyday life
• deal with most situations likely to arise when
  travelling
• enter unprepared into conversations on familiar
  topics

28
CEFR 3.6 Salient Characteristics B2

• Effective argument
• account for and sustain opinions in discussion by
  providing relevant explanations and arguments
• explain a viewpoint on a topical issue giving the
  advantages and disadvantages of various options
• Holding your own in social discourse
• interact with a degree of fluency and spontaneity
  that makes regular interaction with native
  speakers possible
• adjust to changes of direction, style and
  emphasis
• A new degree of language awareness
• make a note of "favourite mistakes" and monitor
  speech for them

29
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency

30
Appendix
United States No Child Left Behind
2001-7
31
US No Child Left Behind 2001-7

• States have a legal duty to provide the support
  to ensure that every child is proficient in the
  academic language they need to be successful at
  school.
• Must test this.
• Must be at least a grade above and a grade below
  proficient. (not just the usual US master /
  non-master)

32
US No Child Left Behind 2001-7

• No overall framework or common reference points
• Testing-led dozens of consortia
• No time for research
• No systematic definition of the construct ALP
• Confusion with English Language Arts (
  creative writing for native speakers)
• or
• Elaborated from language used in subject content
  standards
• No definition of proficient 15 significantly
  different interpretations
• Some states 3, some 4, some 5 grades all
  different names, numbers, concepts
• CHAOS

33
Need for Collaboration Research

• Vollmer
• Pooling expertise and materials
• Corpus of curricula and examination papers
• Classroom observation and research
• Interviews with teachers
• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Analyse textbooks

34
US Experience

• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Analyse textbooks

35
Analysing Aligning Standards

• Assumptions in Subject Standards
• Elementary School observe, analyse, compare,
  describe, record
• Middle School identify, recognise, compose,
  explain
• High School recognise, describe, explain
• (Bailey and Butler 2003)
• Extracting the language features embedded in the
  content standards presented significant
  challenges .
• Bailey, Butler and Sato (2005) have been
  successful developing standards-standards
  linkages that involve both language and content
  standards BUT procedures to establish such
  linkages remain to this day in their infancy
• (Chaloub-Deville 2008)

36
US Experience

• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Analyse textbooks

37
Classroom Research

• From 2001
• Analysis of functions in science classrooms
• Teachers
• Students
• Repair strategies
• (Bailey and Butler 2003)
• BUT
• All tests produced before any research results
  were available even in consortia aware of the
  problem
• (Chaloub-Deville 2008

38
US Experience

• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Analyse textbooks

39
Teacher Expectations

• Students must learn acceptable ways of presenting
  information to the teacher not usually
  explicitly taught
• Very little study
• Teachers are rarely explicitly aware of their
  language expectations
• Dropped the idea of teacher interviews because
  anecdotal unreliable information
• (Bailey and Butler 2003)

40
US Experience

• US No Child Left Behind (Bailey Butler)
• Analyse and align existing content and language
  standards
• Need for observation and research on Teacher Talk
• Need for empirical analysis of performance of
  mother tongue and second language students
• Interviews with teachers (expectations Recep
  Production)
• Textbooks

41
US No Child Left Behind 2001-7

• No overall framework or common reference points
• Testing-led dozens of consortia
• No time for research
• No systematic definition of the construct ALP
• Confusion with English Language Arts (
  creative writing for native speakers)
• or
• Elaborated from language used in subject content
  standards
• No definition of proficient 15 significantly
  different interpretations
• Some states 3, some 4, some 5 grades all
  different names, numbers, concepts
• CHAOS

42
Stages in Developing Descriptors

• 1. Conceptualisation
• Clarifying the construct. What are we describing?
• Collecting relevant example, systems
• Deciding categories in the descriptive scheme
• Clarifying key questions
• 2. Construction
• Creating the descriptor pool
• Editing, drafting filling gaps in the
  description
• 3. Validation
• Qualitative through iterative workshops with
  teachers
• Quantitative through IRT scaling of use in
  assessment
• 4. Interpretation
• Set thresholds between levels
• Summarise developing proficiency