Are Accommodations Used for ELL Students Valid?

1
Are Accommodations Used for ELL Students Valid?

• Jamal Abedi
• University of California, Davis
• National Center for Research on Evaluation,
  Standards and Student Testing (CRESST)

2
Validity of accommodations

• Results of several studies have indicated that
  some accommodation strategies help both ELL and
  non-ELL students
• These accommodations do more than just level the
  playing field because they provide unfair
  advantages to the recipients
• For some accommodations there is enough research
  to judge the appropriateness and validity of the
  accommodations for ELL students

3
Validity of Accommodations

• Research has shown that providing an English
  dictionary (Abedi, Courtney, Leon, 2003 Abedi,
  Lord, Kim, Miyoshi, 2000) and extra time
  (Abedi, Lord, Hofstetter, Baker, 2000 Hafner,
  2001 Thurlow, 2001) affects performance of all
  students (see also, Maihoff, 2002 Thurlow Liu,
  2001)
• That is, the results of accommodated and
  non-accommodated assessment may not be aggregated
  
• Research also suggests that translation of
  assessment tools into students native language
  may not produce desirable results if the language
  of instruction and assessment is not aligned
  (Abedi, Lord, Hofstetter, Baker, 2000)
• The purpose of accommodation for ELL students is
  to reduce the impact of language factors in order
  to provide a fair assessment of the content
  knowledge for these students

4
This particular study addresses the following
four questions

• 1. Do accommodation strategies help reduce the
  performance gap between ELL and non-ELL students
  by removing language barriers? (Effectiveness)
• 2. Do accommodation strategies impact performance
  of non-ELL students on content-based assessment?
  (Validity)
• 3. Do student background variables impact
  performance on the accommodated assessments?
  (Differential impact)
• 4. Are the accommodations easy to apply?
  (Feasibility)

5
Methodology

• A total of 1,854 Grade 4 students and 1,594 Grade
  8 students in 132 classes at 40 school sites
  participated in this study
• The English proficiency designation of students
  was determined based on school records
• Out of 3,448 students, 1,712 (49.7) students
  were identified as being ELL
• Three language categories were targeted for this
  study Spanish, Chinese, and other Asian
  languages

6
Methodology (continues)

• Of the 1,712 ELL students, 1,614 (94.3) students
  had Spanish language background
• A science test of multiple-choice and open-ended
  questions was administered in four forms (1)
  original, (2) either a Customized English
  Dictionary, (3) a Bilingual/English Glossary, or
  (4) a Linguistic Modification of the test that
  addressed the challenge of understanding the
  English lexicon and, possibly, its syntax
• These three forms also allowed additional time
  (50)
• In addition to the science test, an English
  language reading battery was also administered to
  determine students reading proficiency levels

7
Methodology (continues)

• The linguistic complexity of the science
  itemsbut not the science contentwas simplified
• The reading measure was important because
  students at different levels of reading
  proficiency may benefit differently from any
  accommodation
• The study also included a student background
  questionnaire, an accommodation follow-up
  questionnaire, and teacher and school
  questionnaires
• The questionnaire included language background,
  country of origin, and length of time in the
  United States, and also asked students to
  self-assess their proficiency both in English and
  in their home language

8
Methodology (continues)

• The accommodation follow-up questionnaire asked
  students whether the dictionary/glossary helped
  them during the science test and how the language
  in the test could have been made easier to
  understand
• To control for teacher, school, and class
  effects, test materials and accommodations were
  distributed randomly among students

9
Null Hypotheses

• H01 In the science assessment, ELL students do
  not benefit from any of the accommodations used
  in this study. (Effectiveness)
• H02 Accommodation does not impact performance of
  non-ELL students on the science test. (Validity)
• H03 Student background variables do not impact
  performance on accommodated science assessment.
  (Differential impact)

10
Methodology (continues)

• We created a latent composite of all the three
  components that measured English reading
  efficiency/proficiency (1) Fluency section of
  the Language Assessment Scales (LAS), (2) one
  intact block of the 1994 NAEP Reading assessment
  (released items), and (3) an experimental word
  recognition section.
• We used this latent composite as a covariate
• Each open-ended science and reading item was
  scored independently by two raters
• Interrater reliability indices (percent of exact
  and within one-point agreement, P.M. correlation,
  intraclass correlation, and kappa, and alpha
  coefficients) were computed
• For the open-ended science items in Grade 4,
  percent of agreement ranged from a low of 69 to
  a high of 97. The kappa coefficient ranged from
  a low of .42 to a high of .94

11
Findings

• The internal consistency coefficient for the
  entire set of Grade 8 reading items was .78 for
  Grade 4, the overall internal consistency
  coefficient for reading was .82
• Students with Spanish as a primary home language
  performed slightly higher on the Linguistic
  Modification accommodation than under the
  Standard condition.
• Students whose primary home language is neither
  English nor Spanish benefited the most from the
  Linguistic Modification version of the test

12
Item-level analysis

• The higher the level of the linguistic
  complexity, the larger the performance difference
  between ELL and non-ELL students
• The larger the performance difference between ELL
  and non-ELL students, the more linguistic
  modification of test items helped reduce the
  performance gap
• Out of 30 science items, students performed
  better on 22 items under the Linguistic
  Modification condition

13
Findings (continue)

• Neither ELL nor non-ELL Grade 4 students
  benefited from any of the three accommodation
  strategies that were used
• We believe this may be because language demand in
  textbooks and tests may be less in lower than in
  higher grades
• The lack of significant impact on Grade 4 non-ELL
  students is an encouraging result because it
  suggests that the accommodation did not alter the
  construct under measurement

14
Results (continues)

• The Linguistic Modification accommodation was
  shown to have a significant impact on the ELL
  students performance in grade 8
• Accommodation helped ELL students to increase
  their performance while the accommodated
  performance of non-ELL students was unchanged

15
Summary Conclusion

• A non-significant impact of the linguistically
  modified test on the non-ELL group assures the
  validity of this accommodation
• Increase performance of ELL students (grade 8)
  under the linguistically modified test shows
  effectiveness of this accommodation
• As for feasibility, this accommodation requires
  up-front preparation, but is easy to implement in
  the field therefore, it is feasible for
  large-scale assessments

16
For more information contact

• Jamal Abedi
• jabedi_at_ucdavis.edu
• [phone]