Sampling Issues and Accessible Assessments

1
Sampling Issuesand Accessible Assessments
Christopher Johnstone National Center on
Educational Outcomes University of Minnesota
2
Big Picture Issues

• There is a population of students who may
  otherwise be succeeding in reading activities in
  schools, but may not be succeeding on large-scale
  assessments.
• This population probably includes students with
  learning disabilities, but may also include
  non-disabled students
• We still are developing our definitions of
  accessible.

3
LD-specific issues

• Students who receive services for specific
  learning disabilities are a heterogeneous group.
• Specific reading disabilities may be related to
  different aspects of the reading process (e.g.,
  decoding, comprehension, etc.)
• Reading issues may be change with grade level of
  students.

4
Research Sampling Issues

• Teacher questionnaire for grades 4 and 8 regular
  and special education teachers.
• To be distributed in regular schools, but
  over-sampling may occur in schools that have high
  populations of students with learning
  disabilities, speech language impairments,
  hearing impairments, and mental retardation.
• Survey may be used as a mechanism for recruiting
  students for future research.

5
Questionnaire Design

• Most items use Likert-scales for teacher ratings.
• Every item has space for teachers to comment if
  they wish.
• Questionnaire begins by asking a teacher to think
  of a student whose test score is likely to differ
  from otherwise demonstrated reading ability and
  to estimate how large that difference might be.

6
Perceived Causes of Challenges Situational
Factors

• Items ask if difficulty on assessments may be a
  result of
• Temporary sickness
• Vision or motor difficulties

7
General Testing Problems

• Test anxiety
• Motivation
• Environmental Distractions
• Distracters within Test (e.g., extraneous
  graphics, too many multiple choice options)
• the subject of another PARA study

8
Word-level Challenges

• Lack of understanding about phonemic principles
• Lack of generalization with phonemic principles
• Poor sight word vocabulary
• Poor fluency

9
Comprehension Challenges

• Poor vocabulary
• Background knowledge deficiencies
• Lack of knowledge about expository text structure
• Passage length (passages too long or not broken
  up into manageable chunks)
• Lack of personal interest in text
• Needs to use strategies not allowed on
  large-scale tests (e.g., highlighting)

10
Under-AssessedAttributes

• Independent Reading
• Persistence
• Effective use of information

11
Research Plan

• Pilot survey was reviewed by 21 experts in the
  fields of reading, assessment, and special
  education in January, 2006.
• Survey will be distributed and analyzed in
  Spring, 2006 if NOT used as a sampling tool for
  students.
• Survey may be used as a sampling tool for
  students for experimental research. If so, survey
  will be distributed in Fall, 2006.

12
Research Results

• Results will be used to inform general principles
  of accessible assessment (to be developed
  collaboratively between PARA and DARA).
• Results will also be used to inform the
  development of testlets or sample items to be
  used in experimental research.