ONR Advanced Distributed Learning

1

• ONR Advanced Distributed Learning
• Language Factors in the Assessment of English
  Language Learners
• Jamal Abedi
• University of California, Los Angeles
• National Center for Research on Evaluation,
  Standards, and Student Testing (CRESST)
• July 18, 2003

2
The No Child Left Behind Act mandates inclusion
of ALL students

• Goals 2000
• Title I and VII of the Improving Americas School
  Act of 1994 (IASA)
• However, language factors create a major obstacle
  in including English language learners (ELLS)
• Because of possible English language
  deficiencies, ELL students have been
  traditionally excluded from large-scale National
  and State assessments.

3

• CRESST Studies on the Impact of Language Factors
  on the Assessment of ELL Students

4
Study 1 Analyses of extant data (Abedi, Lord,
Plummer, 1995). Used existing data from NAEP
1992 assessments in math and science. SAMPLE
ELL and non-ELLs in grades 4, 8, and 12 main
assessment. NAEP test items were grouped into
long and short and linguistically complex/less
complex items.

• Findings
• ELL students performed significantly lower on the
  longer test items.
• ELL students had higher proportions of omitted
  and/or not-reached items.
• ELL students had higher scores on the
  linguistically less-complex items.

5

Study 2 Interview study (Abedi, Lord,
Plummer, 1997) 37 students asked to express
their preference between the original NAEP items
and the linguistically modified version of these
same items. Math test items were modified to
reduce the level of linguistic complexity.

• Findings
• Over 80 interviewed preferred the
  linguistically modified items over the
  original version.

6
The revised items need less time to respond

• Its easier to read, and it gets to the point,
  so you wont have to waste time.
• I might have a faster time completing that one
  cause theres less reading.
• Less reading then I might be able to get to
  the other one in time to finish both of
  them.
• Cause its, like, a little bit less writing.

7
The vocabulary in the revised items was more
familiar

• This one uses words like approximation, and
  this one uses words that I can relate
  to.
• It doesnt sound as technical.
• I cant read that word.
• Because its shorter and doesnt have, like,
  complicated words.

8
Study 3 Impact of linguistic factors on
students performance (Abedi, Lord, Plummer,
1997). Two studies testing performance and
speed. SAMPLE 1,031 grade 8 ELL and non-ELL
students. 41 classes from 21 southern California
schools.

• Findings
• ELL students who received a linguistically
  modified version of the math test items
  performed significantly better than those
  receiving the original test items.

9
Study 4 The impact of different types of
accommodations on students with limited English
proficiency (Abedi, Lord, Hofstetter, 1997)
SAMPLE 1,394 grade 8 students. 56 classes
from 27 California schools.

• Findings
• Spanish translation of NAEP math test
• Spanish-speakers taking the Spanish translation
  version performed significantly lower than
  Spanish-speakers taking the English version.
• We believe that this is due to the impact of
  language of instruction on assessment.
• Linguistic Modification
• Contributed to improved performance on 49 of the
  items.
• Extra Time
• Helped grade 8 ELL students on NAEP math tests.
• Also aided non-ELL students. Limited potential
  as an assessment accommodation.

10
Study 5 Impact of selected background
variables on students NAEP math
performance (Abedi, Hofstetter, Lord, 1998).
SAMPLE 946 grade 8 ELL and non-ELL students.
38 classes from 19 southern California schools.

• Findings
• Four different accommodations used
  (linguistically modified, a glossary only,
  extra time only, and a glossary plus extra time).
• The glossary plus extra time was the most
  effective accommodation.
• Glossary plus extra time accommodation
• Non-ELLs showed a greater improvement (16) than
  the ELLs (13).
• This is the opposite of what is expected and
  casts doubt on the validity of this
  accommodation.

11
Study 6The effects of accommodations on the
assessment of LEP students in NAEP (Abedi, Lord,
Kim, Miyoshi, 2000) SAMPLE 422 grade 8 ELL
and non-ELL students. 17 science classes from 9
southern California schools.

• Findings
• Some forms of accommodations may help the
  recipients with the content of assessment. For
  example, a dictionary defines all the words in a
  test, both content and non-content.
• A Customized Dictionary
• Easier to use than a published dictionary
• Included only non-content words in the test.
• ELL students showed significant improvement in
  performance.
• No impact on the non-ELL performance.

12

• Study 7
• Language accommodation for large-scale assessment
  in science
• (Abedi, Courtney, Leon, Mirocha, Goldberg,
  2001).
• SAMPLE 612 grades 4 and 8 students. 25
  classes from 14 southern California schools.

• Findings
• A published dictionary was both ineffective and
  administratively difficult as an
  accommodation.
• Different bilingual dictionaries had different
  entries, different content, and different
  format.

13

• Study 8
• Language accommodation for large-scale assessment
  in science
• (Abedi, Courtney, Leon, 2001)
• SAMPLE 1,856 grade 4 and 1,512 grade 8 ELL and
  non-ELL students.
• 132 classes from 40 school sites in four cities,
  three states.

• Findings
• Results suggested linguistic modification of
  test items improved performance of ELLs in
  grade 8.
• No change on the performance of non-ELLs with
  modified test.
• The validity of assessment was not compromised by
  the provision of an accommodation.

14

• Study 9
• Impact of students language background on
  content-based performance analyses of extant
  data (Abedi Leon, 1999).
• Analyses were performed on extant data, such as
  Stanford 9 and ITBS
• SAMPLE Over 900,000 students from four
  different sites nationwide.

Study 10 Examining ELL and non-ELL student
performance differences and their relationship to
background factors (Abedi, Leon, Mirocha,
2001). Data were analyzed for the language
impact on assessment and accommodations of ELL
students. SAMPLE Over 700,000 students from
four different sites nationwide.

• Findings
• The higher the level of language demand of the
  test items, the higher the performance gap
  between ELL and non-ELL students.
• Large performance gap between ELL and non-ELL
  students on reading, science, and math problem
  solving (about 15 NCE score points).
• This performance gap was zero in math
  computation.

15
Study 11 Research-supported accommodations for
English language learners in NAEP (Abedi,
Courtney Leon, 2002) SAMPLE 607 grade 4
students (46 ELLs and 54 non-ELLs) and 542
grade 8 students (47 ELLs and 53 non-ELLs)
Accommodations Computer testing, customized
dictionary, and extra time. A reading composite
score was used as a covariate. Student responses
to accommodation follow-up questionnaires and
background questionnaires were analyzed.

• Findings
• The computer testing was the most effective
  accommodation. It provided an alternative test
  item delivery and an easy-to-access gloss of
  non-math lexicon.
• The customized dictionary was also shown to be
  effective.
• Since non-ELLs who were accommodated performed
  the same as non-ELLs who were not
  accommodated, the two effective accommodations
  are deemed valid.

16

Study 12 Opportunity to Learn for English
Language Learners (Abedi, Courtney, Leon,
2002) SAMPLE 607 grade 4 students (46 ELLs
and 54 non-ELLs) and 542 grade 8 students (47
ELLs and 53 non-ELLs)

• Findings
• Student self-reported OTL correlated with their
  actual performance in math.
• Teacher-reported OTL (their indication that they
  taught the materials) did not correlate as
  high with the student performance.

17
Study 13 Issues and problems in
classification of students with limited English
proficiency (Abedi Leon, 2002)

• This study examined the validity of LEP
  classification scheme by analyzing extant
  data.
• LEP classification codes correlated poorly with
  test scores.
• In lower grades, low-performing ELLs tend to
  remain classified as LEP.
• There appears to be a tendency to reclassify
  these students in higher grades.
• Correlation between test scores and LEP
  classification varies substantially among
  districts.
• The results of longitudinal analyses indicated
  that in addition to language proficiency,
  student background variables were also predictors
  of LEP classification.

18
Study 14 Opportunity to learn for English
language learners OTL and language interaction
(Abedi, Courtney, Leon, 2002) This study
examines the differences, if any, in opportunity
to learn (OTL) between ELLs and their non-ELL
peers in grade 8 math. SAMPLE 700 grade 8
algebra students (in the 2-year track)

• Research Questions
• Do ELL students receive the same level of OTL as
  non-ELL students? (Observation/teacher
  interview/ student OTL questionnaire and field
  testing)
• Are the OTL factors influenced by student level
  of English language proficiency?
  (Observation/teacher interview/ student OTL
  questionnaire)
• Are the OTL factors for ELL students influenced
  by the teachers impression of the ELL
  students ability to learn? (Observation/teacher
  interview/ student OTL questionnaire)

19

• References to CRESST Studies

20
Reports
21
Abedi, Courtney Leon (2002) Research-Supported
Accommodations for English Language Learners in
NAEP. Los Angeles University of California, Los
Angeles, National Center for Research on
Evaluation, Standards, and Student
Testing. Abedi, J., Lord, C., Hofstetter, C.
(2001). Impact of Selected Background Variables
on Students NAEP Math Performance. National
Center for Education Statistics (NCES), Working
Paper, Publication (NCES 200111). Abedi, J.
(2001). Assessment and Accommodations for English
Language Learners Issues and Recommendations.
Los Angeles, National Center for Research on
Evaluation, Standards, and Student Testing.
Policy Brief 4. Abedi, J. Courtney, M. and
Leon, S. (2001). Language Accommodation for
Large-scale Assessment in Science Assessing
English Language Learners. Los Angeles
University of California, Los Angeles, National
Center for Research on Evaluation, Standards,
and Student Testing.
22
Abedi, J. Lord, C. Kim, C. Miyoshi, J (2001).
The effects of accommodations on the assessment
of LEP students in NAEP. National Center for
Education Statistics (NCES), Working Paper,
Publication (NCES 200113). Abedi, J.,
Courtney, M., Mirocha, J., Leon, S., Goldberg.
J. (2000). Language Accommodation for
Large-scale Assessment in Science. Los Angeles
University of California, Los Angeles, National
Center for Research on Evaluation, Standards,
and Student Testing. Abedi, J., Lord, C., Kim,
C., Miyoshi, J (2000). The effects of
accommodations on the assessment of LEP
students in NAEP. Los Angeles University of
California, Los Angeles, National Center for
Research on Evaluation, Standards, and Student
Testing. CSE Technical Report 537. Abedi, J.,
Leon, S., Mirocha, J. (2001). Students
performance differences in standardized
achievement tests and background factors
Analyses of Extant Data. University of
California, Los Angeles, National Center for
Research on Evaluation, Standards, and Student
Testing.
23
Abedi, I. Leon, S. (1999). Impact of students
language background on content-based
performance Analyses of extant data.
University of California, Los Angeles, National
Center for Research on Evaluation, Standards, and
Student Testing. Abedi, J., Hofstetter, C.,
Baker, E. Lord, C. (1998). NAEP math
performance and test accommodations
Interactions with student language background,
Draft Report. Los Angeles University of
California, Los Angeles, National Center for
Research on Evaluation, Standards, and Student
Testing. CSE Technical Report 536. Abedi, J.,
Lord, C., Hofstetter, C. (1997). Impact of
selected background variables on students NAEP
math performance. Los Angeles University of
California, Los Angeles, National Center for
Research on Evaluation, Standards, and Student
Testing. CSE Technical Report 478. Abedi,
J., Lord C., Plummer, J. R. (1997). Language
Background as a Variable in NAEP Mathematics
Performance. Los Angeles Center for the Study of
Evaluation, CSE Technical Report 429.
24
Publications
25
Abedi, J. (2002). Standardized achievement
tests and English language learners
Psychometrics and linguistics issues.
Educational Assessment (accepted for
publication in Educational Assessment). Abedi,
J., Hofstetter, C., Lord, C . (2002) Assessment
Accommodations for English Language Learners A
Review of Empirical Research and Policy Issues.
Review of Educational research (submitted for
publication). Abedi, J. (2002). Issues and
problems in classification of students with
limited English proficiency. Educational
Measurement Issues and Practice. (submitted for
publication). Abedi, J. (2002). Assessing and
Accommodations of English language learners
Issues, concerns and recommendations. Journal
of School Improvement. v3, n1, Spring 2002.
26
Abedi, J., Lord, C (2001). The Language Factor in
Mathematics Tests. Applied Measurement in
Education, 14, 3, June 2001. Abedi, J. (2000).
Loaded Questions? American Language Review, The
Magazine for Language Teaching Professional.
July/August 2000. Abedi, J., Lord, C.,
Hofstetter, C., Baker, E. (2000) Impact of
accommodation strategies on English language
learners test performance. Educational
Measurement Issues and Practice, 19, 3, pp.
16-26.