October 13, 2006

1
Item Analyses and Features Studies and Other
Findings

• October 13, 2006
• Jamal Abedi
• CRESST/University of California, Davis
• Seth Leon Jenny Kao
• CRESST/University of California, Los Angeles

2
Examining Differential Item Functioning in
Reading Assessments for Students with
Disabilities

• This study examined differences between students
  with disabilities and non-disabled students using
  DIF
• Data from Stanford Achievement Test (Ninth
  Edition) Reading Comprehension (RC) and Word
  Analysis (WA) in two different states were used
• Data were from the 1997-1998 school year

3
Focus on Reading

• In Site 2 there were 278,287 Grade 3 students
  (7.6 with disabilities) and 244,446 Grade 9
  students (9.9 with disabilities).
• In Site 4, there were 6,611 Grade 3 students
  (6.8 with disabilities), and 5,287 Grade 9
  students (9.9 with disabilities)

4
The following research questions guided
the study

• Do items on standardized Reading Comprehension
  (RC) and Word Analysis (WA) subscales exhibit
  Differential Item Functioning (DIF) for students
  with disabilities?
• Are more items that exhibit DIF for students with
  disabilities located in the second half of RC and
  WA subscales rather than in the first half?

5
Research questions (continued)

• 3. Do students with disabilities consistently
  under-perform on items located in the second half
  relative to items located in the first half, as
  compared to non-disabled students?
• 4. Do the results of DIF vary by grade (3 and 9)?

6
Results

• For grade 9, many items exhibited DIF
• Items that exhibited DIF were more likely to be
  located in the second half of the assessment
  subscales
• After controlling for reading ability, students
  with disabilities consistently under-performed on
  items located in the second half
• Results were seen in grade 9 for data from two
  different states.

7
Results (continued)

• In grade 3 there were fewer items that were shown
  to exhibit DIF for students with disabilities
  This was true for both RC and WA subscales
• This study has several limitations to the data
• There was no access to information regarding the
  testing accommodations that students with
  disabilities might have received
• No access to the type of disabilities

8
Examining Differential Distractor
Functioning in Reading Assessments for
Students with Disabilities

• In DDF analysis we examined the pattern of
  incorrect answers or distractors
• Data from Site 4 were used to examine Grade 3 and
  Grade 9 students responses to Stanford
  Achievement Test, Ninth Edition.

9
The following research questions guided this
study

• Do items on standardized Reading Comprehension
  (RC) and Word Analysis (WA) subscales exhibit
  differential distractor functioning (DDF) for
  students with disabilities?
• Does the differential distractor functioning for
  students with disabilities increase for items
  located in the second half of RC and WA
  subscales?
• Do the results of DDF vary by grade (from grade 3
  to grade 9)?

10
Results

• Results suggest that a substantial number of
  items from both the Reading Comprehension (RC)
  and Word Analysis (WC) subscales exhibit DDF for
  students with disabilities in grade 9.
• Results also suggest that items showing DDF were
  more likely to be located in the second half of
  the assessments rather than the first half of the
  assessments.
• Results also indicate that DDF was present for
  grade 9 test items, but not for grade 3 items.

11
Results (continued)

• Even when controlling for ability using only the
  items in the first half of the assessments, more
  grade 9 items exhibited DDF than grade 3 items
• Students with disabilities were less likely to
  choose the most common distractor chosen by their
  non-disabled peers
• Students with disabilities might be more randomly
  selecting one of the four response options rather
  than making an educated guess

12
Results (continued)

• Access to information regarding the type of
  accommodations students received (e.g.,extended
  time) would be helpful, but was not possible

13
Summary/Conclusion

• More items were identified as DIF and DDF in
  grade 9 than grade 3
• Students with disabilities selected less commonly
  used distractors
• In both DIF and DDF students with disabilities
  had more difficulties with the items in the
  second half of the test
• This may suggest the higher level of impact of
  fatigue and frustration on students with
  disabilities

14
Item Features Study Research Questions
Focus on Passage Order (Model 1)

• Test versions two for students with disabilities
  (1-2 2-1) three for non-disabled students (1-2
  and 2-1 and 1 chunked-2 not-chunked).
• The research questions are divided into three
  parts even though some of them may be
  inter-related.

15
I. Passage Order (1-2 versus 2-1)

• Does passage order affect students performance
  overall?
• Is student performance on the first passage
  affected by whether it appears first in the test
  (1-2) or last (2-1)?
• Is there a differential effect of passage order
  on students with disabilities as compared with
  non-disabled students?

16
II. Motivation Fatigue

• Do the students with disabilities in this study
  report lower motivation than non-disabled
  students?
• Does the performance of students who reported
  lower motivation have a relationship with passage
  order?
• Is there a relationship between passage order and
  students rating of fatigue?

17
III. Chunking (for non-disabled students
passages 1-2 both chunked)

• Is student performance affected by whether the
  test is presented as chunked or not chunked?
• Is there a relationship between students report
  of motivation and chunking?
• Is there a relationship between students report
  of fatigue and chunking?
• Note Fatigue questions will appear after each
  set of passages and corresponding items.
  Motivation questions appears once as post-scale.

18
Design

• Analysis of covariance
• Student ability will be controlled for using
  scores on their prior years state assessment, or
  by a quick reading efficiency battery (the CRESST
  TIMER test)
• Motivation scale will appear only as a post-test.
• Research questions relating to chunking will
  again be addressed with a similar analysis of
  covariance approach. Analysis will only be
  applied to the students without disabilities.

19
Item Features Study Research Questions
Focus on Chunking (Model 2)

• Will be tested on students with no disabilities
• Test versions two for everyone both consist of
  two long passages one version is chunked, one is
  not chunked.

20
I. Chunking

• Does chunking improve the performance of
  non-disabled students? (validity)
• Does chunking improve the performance of students
  with disabilities?
• If the performance of both groups improve, is
  there a differential effect for students with
  disabilities?

21
II. Motivation Fatigue

• Do the students with disabilities in this study
  report lower motivation than non-disabled
  students?
• Did the students who took the chunked version of
  the test report higher motivation than students
  who took the non-chunked version?

22
Motivation Fatigue (continued)

• Do the students with disabilities in this study
  report less fatigue than non-disabled students?
• Did the students who took the chunked version of
  the test report less fatigue than students who
  took the non-chunked version?

23
III. Passage Order

• Is there a difference in performance between
  students with disabilities and non-disabled
  students on the second passage after controlling
  for performance on the first passage?
• If so, does this difference in performance occur
  on both the chunked and non-chunked versions?

24
Limitations

• No interaction between passage order and chunking
  can be estimated
• The effect of chunking will only be examined on
  non-SD students to provide some validity evidence
• Small number of SDs threaten to power of the
  analyses

25
Examine Chunking on Students with
Disabilities (Model 3)

• Test SDs twice
• Test passage order and chunking in a
  counter-balance order
• This model allows examination of chunking on SDs
  as well as non-SDs
• Useful information can be provided, for example,
  chunking can be more effective for SDs

26
Your advice/suggestions

• How to resolve the sample size with the limited
  access to schools?
• Which model do you think would produce more
  useful results?
• Can you help with more school sites/access?
• Are you interested in a more involved role in
  this study?