Standardisation of judgements: reading and writing

1
Standardisation of judgements reading and writing

• Ülle Türk
• University of Tartu/Estonian Defence Forces
• ulle.turk_at_ut.ee/ylle.tyrk_at_mil.ee

2
Reading
3
Training with standardised samples for reading

• Phase 1 Illustration a first assessment of the
  level of one text and its corresponding tasks and
  items
• At what CEFR level can a test taker already
  answer the following item correctly?
• Phase 2 Controlled practice different texts
  with their corresponding tasks and items are
  assessed under the guidance of the co-ordinator
• Phase 3 Individual assessment of the remaining
  items in chunks of 4-6 items
• The session continues until a spread of results
  of one and a half levels is achieved.

4
Benchmarking local samples of reading

• Individual rating followed by (pair) discussion
  leading to group consensus
• Discussion of spread across individual ratings
  and iteration until suitable agreement (spread
  equal to one and a half levels) is reached
• Ratings of the CEFR standardised samples should
  be analysed statistically in order to
• Confirm the relationship to the levels and
• Calculate intra-rater reliability (consistency)
  and inter-rater reliability

5
Task 4 Items 14

• Text Characteristics
• Text source magazine
• Authenticity authentic
• Discourse type mainly argumentative
• Domain public
• Topic other/politics
• Nature of content fairly extensive abstract
  content
• Text length 350 words
• Vocabulary extended vocabulary
• Grammar wide range of complex structures
• Text likely to be comprehensible by learner at
  CEF level C1

6
Items 1 2

• Item 1
• Item type Selected response/multiple choice
• Operations Infer implicit main idea/gist
• Item level estimated A2/B1
• Key B
• Per cent correct 90.5
• Discrimination (item/total correlation) .31

• Item 2
• Item type Selected response/multiple choice
• Operations Make explicit inferences about the
  writers mood
• Item level estimated C2
• Key B
• Per cent correct 36.1
• Discrimination (item/total correlation) .45

7
Items 3 4

• Item 3
• Item type Selected response/multiple choice
• Operations Make inferences about implicit main
  idea/gist
• Item level estimated C2
• Key B
• Per cent correct 43.3
• Discrimination (item/total correlation) .20

• Item 4
• Item type Selected response/multiple choice
• Operations Make inferences about implicit detail
  
• Item level estimated B1
• Key C
• Per cent correct 83.6
• Discrimination (item/total correlation) .43

8
Task 5 Items 58

• Text Characteristics
• Text source magazine
• Authenticity authentic
• Discourse type mainly expository
• Domain public/personal
• Topic other/contemporary life
• Nature of content mainly abstract content
• Text length 418 words
• Vocabulary extended vocabulary
• Grammar mainly complex structures
• Text likely to be comprehensible by learner at
  CEF level C1

9
Items 5 6

• Item 5
• Item type Selected response/multiple choice
• Operations Infer implicit detail
• Item level estimated B2
• Key B
• Per cent correct 65,9
• Discrimination (item/total correlation) .52

• Item 6
• Item type Selected response/multiple choice
• Operations Evaluate implicit opinion
• Item level estimated C2
• Key C
• Per cent correct 49,2
• Discrimination (item/total correlation) .32

10
Items 7 8

• Item 7
• Item type Selected response/multiple choice
• Operations Make inferences about implicit main
  idea/gist
• Item level estimated C1
• Key C
• Per cent correct 52,1
• Discrimination (item/total correlation) .36

• Item 8
• Item type Selected response/multiple choice
• Operations Evaluate implicit attitude/conclusion
  
• Item level estimated B2
• Key A
• Per cent correct 74,4
• Discrimination (item/total correlation) .24

11
Writing
12
Training with standardised samples for writing

• Phase 1 illustration with about three
  standardised performances 60 minutes
• Phase 2 controlled practice with 35
  standardised performances 60 minutes
• Phase 3 individual assessment of 35
  standardised performances 60 minutes
• The session ends when the vast majority of
  participants agree on the level, with the spread
  not exceeding one and a half levels.

13
Benchmarking local samples of writing

• Individual rating and group discussion of high,
  middle and low performances 60 minutes
• Individual rating of about five more performances
  60 minutes

14
Sample task 1

• Your English Teacher has asked you to write a
  story.
• Your story must have the following title
• The most important day of my life
• Write your story on your answer sheet.

15
Example answer 1

• The most important day of my life.
• During a lifetime there is so many days you could
  call, the most important day of your life. It
  could be the day you chose wich school you are
  going to, or what you want to work with the rest
  of your life. Another important day is when you
  get married, or you chose where to live. But most
  of all it must be a very important day when you
  give birth to a child. I think that changes
  everything you have been doing until then. Than
  you have to realice that somebody are more
  important than yourself.

16
Sample task 2

• This is an extract from a letter which you
  receive from the World Opinion Organisation.
• We are carrying out an international survey on
  families around the world and the importance of
  family relationships. Please write us a report
  for our survey outlining the typical family
  situation in your own country and suggesting how
  you think the situation might change in the
  future.
• Write your report. Write approximately 250 words.

17
Sample task 3
You have read the extract below as part of a
newspaper article on the loss of national and
cultural identity. Readers were asked to send in
their opinions. You decide to write a letter
responding to the points raised and expressing
your own views. We are losing our national and
cultural identity. Because of recent advances in
technology and the easy availability and speed of
air travel, different countries are communicating
more often and are therefore becoming more and
more alike. The same shopping malls and fast food
outlets can be found almost everywhere. So can
the same types of office blocks, motorways, TV
programmes and even lifestyles. How can we
maintain the traditions that make each nation
unique? Write your letter. Do not write any
postal addresses.
18
Sample task 3

• You recently entered a competition and have just
  received this letter from the organiser. Read the
  letter, on which you have made some notes. Then,
  using all the information in your notes, write a
  suitable reply.
• Write a letter of between 120 and 180 words in an
  appropriate style on the opposite page. Do not
  write any postal addresses.

19
(No Transcript)
20
Sources

• Manual for relating language examinations to the
  Common European Framework of Reference for
  Languages a preliminary pilot version
  http//www.coe.int/T/DG4/Portfolio/?LEM/documen
  ts_intro/Manual.html
• The CEFR Grid for Writing, v. 2.
• The Final Grid for analysing reading and
  listening items www.ling.lancs.ac.uk/cefgrid