Developing an automated assessment tool for children

1
Developing an automated assessment tool for
childrens oral reading

• Leen Cleuren
• March 5 2007

2
Overview

• SPACE project
• General objectives
• Development of Chorec
• Development of an automated assessment tool
• Doctoral research project (DRP)
• General objectives
• Development of a new computerized reading test
  battery

3
SPACE General objectives

• Explore the benefits of speech recognition
  technology for the assessment of word decoding
  skills
• Automated assessment
• No examiner-bias

4
SPACE General objectives

• Explore the benefits of speech recognition and
  speech synthesis technology for the development
  of a remedial reading tool
• Individual practice possible
• Personally adapted appropriate feedback

5
SPACE General objectivesin other words...

• Looking for a reading tutor that can make a
  diagnosis
• Demands for the speech recognizer
• Tracking the childs progress during reading
• Accurately detecting and classifying oral reading
  errors and strategies

6
SPACE General objectivesin other words...

• Looking for a reading tutor that can read to or
  read along with the child can intervene whenever
  reading errors occur can give appropriate
  feedback
• Demands for the speech synthesizer
• Naturally sounding and highly intelligible speech
• Being able to give different kinds of feedback

7
SPACE Development of Chorec

• ASR within the context of reading assessment and
  instruction is a very challenging task
• e.g.
• Articulatory competences of young children can
  differ
• Oral reading can be fraught with reading errors

8
SPACE Development of Chorec

• To improve the speech recognizer's ability to
  accurately detect reading errors
• statistical characterization of reading behavior
  is necessary
• a model that contains information on the nature
  and prevalence of likely reading errors is needed
  
• ? Chorec Childrens Oral REading Corpus
• Database of recorded and annotated childrens
  oral reading and oral reading errors and
  strategies

9
SPACE Development of Chorec

• Recordings
• 256 regular elementary school children (grade
  1-4)
• 150 children with known reading disabilities
  (elementary school age)
• Words, pseudowords, stories
• Annotations
• Different annotation layers containing different
  information

10
Speech signal 2 microphones
strand
stAnt
Reading strategy
Reading error
11
SPACE Development of ChorecClassification of
reading stragies

• correct reading
• correct direct word recognition within the 1st
  trial
• repeating a directly recognized word once or
  more
• partially or completely (in)correctly spelling
  out a word before correctly synthesizing it
• incorrectly direct word recognition in the first
  trial but reading it correctly in the final trial

To read doos Child reads d...oo...s doos
b...oo... doos
To read doos Child reads doos
To read doos Child reads doos ... doos
To read doos Child reads boos ... doos
12
SPACE Development of ChorecClassifcation of
reading strategies

• incorrect reading
• incorrect direct word recognition within the 1st
  trial
• partially or completely (in)correctly spelling
  out and incorrectly synthesizing a word or not
  synthesizing it at all
• direct recognition of a word within the first
  trial but correcting it wrongly on a second
  trial
• omission or insertion of a word
• asking for a complete or partial prompt of a
  word before carrying on reading it

To read doos Child reads boos
To read doos Child reads b...oo...s boos
d...oo...s ...
To read doos Child reads doos ... boos
To read De doos staat op de tafel. Child reads
De doos staat op tafel. De doos staat op de
grote tafel.
13
SPACE Development of ChorecClassification of
errors

• paragraph level
• omission or repetition of a whole line or
  sentence
• erroneous insertion of a word
• change of word order
• substitution of a word by a synonym or
  semantically related word
• sentence level
• omission or repetition of a part of a sentence
• word level
• wrong decoding strategy
• wrong direct word recognition strategy
• grapheme level
• sequential errors, substitution errors
• deletion errors, insertion error

14
Girl, 1st grade, regular elementary school, 34
syl. words
15
SPACE Development of an automated assessment
tool

• Work done by ESAT
• Demo

16
Overview

• SPACE project
• General objectives
• Development of Chorec
• Development of an automated assessment tool
• Doctoral research project (DRP)
• General objectives
• Development of a new computerized reading test
  battery

17
DRP General objectives

• Development of a new computerized test battery
  for the assessment of childrens word decoding
  skills
• Analysis of the quantitative and qualitative
  development of reading errors and strategies in
  elementary school children with and without
  reading disabilities

18
DRP Development of a test battery

• Achievements
• Research objectives
• Participants
• Data collection
• Speed-accuracy trade-off problem

19
DRP Development of a test batteryAchievements

• Development of a computerized reading tutor
  assessment platform (see demo)
• Development of a test battery to assess
  elementary school childrens word decoding skills

20
DRP Development of a test batteryAchievements

• Word and pseudoword reading test (WRT PWRT)
• 3 lists of (pseudo)words 1 syl., 2 syl., 34
  syl.
• Words oog, water, omdraaien
• Pseudowords eem, ulen, ometuif
• Story reading test (SRT)
• 9 graded text stories AVI 1 AVI 9

21
DRP Development of a test batteryResearch
objectives

• Standardization of the WRT and PWRT
• Speed-accuracy trade-off problem
• Reliability assessment and validation of the WRT
  and PWRT
• Looking for an alternative measure to capture
  reading fluency

22
DRP Development of a test batteryParticipants

• 256 regular elementary school children (grade
  1-4)
• 124 boys
• 132 girls
• Mothertongue Dutch
• No doubling or passing over

23
DRP Development of a test batteryData
collection

• Questionnaire for parents and teachers
• Teacher
• Reading instruction method used?
• Childs AVI-level?
• Childs school history?
• RD present?
• Parents
• RD present?
• Childs name, birth place, birth date,
  nationality, (previous) residence
• Languages spoken by the child
• Chorec audio recordings children reading WRT,
  PWRT, SRT
• Administration of One-Minute-Test, Klepel,
  AVI-test

24
DRP Speed-accuracy trade-offDistribution of
speed (2nd grade)

• No class. var.
• ? Lognormal distrib.
• Mean 106 ms
• Std. Dev. 53 ms

25
DRP Speed-accuracy trade-offDistribution of
speed (2nd grade)
p lt 0.05 R² 0.12
WRT
PWRT
26
DRP Speed-accuracy trade-offDistribution of
speed (2nd grade)
p lt 0.05 R² 0.42
1LG
2LG
34LG
34LGP
2LGP
1LGP
Not significant1LG-2LG, 34LG-2LGP,
1LGP-2LG
27
DRP Speed-accuracy trade-offDistribution of
speed (2nd grade)
p gt 0.05
Boys
Girls
28
DRP Speed-accuracy trade-offDistribution of
speed (2nd grade)

• No interaction between
• Sex
• Task

p gt 0.05
29
DRP Speed-accuracy trade-offDistribution of
correct (2nd grade)

• No class. var.
• Mean 32
• Std. Dev. 8

30
DRP Speed-accuracy trade-offDistribution of
correct (2nd grade)
p lt 0.05 R² 0.31
PWRT
WRT
31
DRP Speed-accuracy trade-offDistribution of
correct (2nd grade)
1LG
2LG
34LG
p lt 0.05 R² 0.61
1LGP
2LGP
34LGP
Not significant1LG-2LG 34LG-1LGP
32
DRP Speed-accuracy trade-offDistribution of
correct (2nd grade)
p lt 0.05 R² 0.02
Boys
Girls
33
DRP Speed-accuracy trade-offDistribution of
correct (2nd grade)

• No interaction between
• Sex
• Task

p gt 0.05
34
DRP Development of a test batterySpeed-accuracy
trade-off

• WRT PWRT partially speed and partially
  accuracy tests
• Speed test without time limit
• Accuracy is important too
• 1LG -0.29, 1LGP -0.44
• 2LG -0.45, 2LGP -0.43
• 34LG -0.59, 34LGP -0.25

r - 0.65
35
DRP Development of a test batterySpeed-accurac
y trade-off

• Speed-accuracy trade-off!
• Very fast with low accuracy
• Very slow with high accuracy
• To perform as well as possible various
  strategies possible
• Optimize speed
• Optimize accuracy
• Optimize both
• ? We need a measure that captures both!

36
DRP Development of a test batterySpeed-accurac
y trade-off

• Score 1 total response time/correct Score
  2 total response time/ (1-error)

37
DRP Development of a test batterySpeed-accurac
y trade-off

• Alternative item response models?
• We have information on reading performance at the
  level of the word
• paard clown huis
• start of utterance
• stimulus presentation

38
DRP Development of a test batterySpeed-accuracy
trade-off

• p chance to be
• correct
• ? skill level

p
1 item
39
Thank you!

• Questions?