Defining Reading Proficiency for Accessible Large Scale Assessments

1
Defining Reading Proficiency for Accessible Large
Scale Assessments
Principles and Issues Paper American Educational
Research Association April 10, 2006 Deborah R.
Dillon, University of Minnesota John
Sabatini, ETS
2
Overview of the Project

• A collaboration between two projects funded to
  conduct research and development on accessible
  reading assessments for students with
  disabilities that affect reading.
• Goals include producing research findings and
  assessment techniques that demonstrate how
  large-scale assessments can become more
  accessible and valid.

3
A Vision for Accessible Assessments for Students
with Disabilities

• Students deserve the opportunity to show their
  proficiency as readers to show what they know
  and are able to do this should be reflected in
  the way reading proficiency is defined.
• Expectations for students reading performance
  should not be lowered rather, accommodations
  should be built into the test itselfvia
  universal designallowing participation of the
  widest range of students more valid inferences
  about performance.
• Flexible expressions of reading will be necessary
  to allow students who may not be proficient on
  all components of reading to demonstrate the
  skills they do know.

4
First Task in the Project
To design accessible reading assessments we need
precise definitions of the constructs being
measured. Step one involves developing a
definition of reading proficiency. To
accomplish this task we synthesized information
and described (a) a set of principles that will
guide the research and development phases of our
projects, and (b) unresolved issues that need to
be addressed related to each principle.

5
Sources for the Development of the Three
Principles and Related Issues

• A review of existing definitions of reading
  proficiency (e.g., reports such as NRP, RAND,
  PISA, PIRLS)
• A panel of experts to provide input
• Focus groups

6
Overview of Principles

• Principle 1 Definitions of reading proficiency
  must be consistent with core NCLB provisions.
• Principle 2 Reading proficiency must be defined
  in such a way that flexible expressions of
  reading are allowed while preserving the
  essential nature of reading. This is crucial as
  we seek to make assessments accessible to
  students with a variety of disabilities.
• Principle 3 Definitions of reading proficiency
  must reflect both comprehension and foundational
  skills.

7
Principle 1 Definitions of reading proficiency
must be consistent with core NCLB provisions.

• Accessible large-scale assessments developed by
  NARAP must adhere to two requirements of NCLB
  to provide (a) a valid measure of proficiency
  against academic standards, and (b) individual
  interpretive, descriptive, and diagnostic reports
  for the full range of students with disabilities
  that affect reading.
• NCLB requires states to address areas of reading
  proficiency and component skills through
  grade-level definitions of content standards
  NARAP cannot impose a particular standardwe must
  allow for the variability among states.

8
Principle 1cont.

• Access to, participation in, and progress in the
  general curriculum is a foundational requirement.
  
• The 2009 NAEP assessment, which is required to be
  used by all states, focuses on grade-level
  cognitive targets, and defines these as the
  mental processes or kinds of thinking that
  underlie reading comprehension.
• Assessments used primarily for accountability
  should also provide useful information to
  educators as they plan instructional improvements.

9
Issues Under Principle 1

• How do the important reading skills vary as a
  function of grade level?
• How do we determine which measure may be
  appropriate for use at a specific grade level?
• How much can achievement levels vary and still
  meet the requirements of grade-level content?
• How are differences in reading achievement
  standards (e.g., modified or alternate
  achievement standards) developed and defined? How
  are these varying achievement standards reflected
  in definitions of reading proficiency?

10
Principle 2 Reading proficiency must
be defined in such a way that flexible
expressions of reading are allowed while
preserving the essential nature of reading. This
is crucial as we seek to make assessments
accessible to students with a variety of
disabilities.

• Proficient readers, including students with
  disabilities, may rely on any set of component
  proficiencies to read and may compensate for some
  skills they lack by drawing on othersdepending
  on their disability (e.g., a student with
  congenital deafness may comprehend what he/she
  reads, but need to deploy alternate strategies to
  understand sound-symbol relationships)
• Public views of what constitutes reading are
  reflected in various ways these are indicated in
  how states allow/do not allow different
  approaches to assessments in reading and if
  assessments are allowed without restrictions

11
Principle 2--cont.

• Federal statutes allow a range of options in
  types of assessments used and the achievement
  standards applied to students with disabilities.
  Some students can take assessments based on
  modified achievement standards or alternate
  assessments based on grade level standards or on
  alternate achievement standards.
• Alternate achievement standards must be aligned
  to grade-level content standards but may differ
  from grade level achievement standards in
  breadth, depth, or complexity.

12
Issues under Principle 2

• How broadly can we define what constitutes
  reading and still have the definition based on
  grade-level achievement standards?
• Can what constitutes as reading for
  standards-based assessments differ by disability
  category or by needed accommodation?
• When do the concepts of modified and alternate
  achievement standards apply to grade-level
  reading standards?
• How so students with disabilities compensate for
  weaknesses in specific reading proficiency
  components due to their disability or multiple
  disabilities?

13
Principle 3 Definitions of reading proficiency
must reflect both comprehension and foundational
skills.

• Panels of researchers and practitioners indicated
  that the NARAP core construct of a definition of
  reading should include both foundational reading
  skills and comprehension.
• NARAP is basing its work on the NCLB definition,
  which emphasizes foundational reading skills, and
  the 2009 NAEP framework, which stresses
  comprehension and indicates that students must
  apply foundational skills to comprehend a variety
  of texts for various purposes and situations.

14
Principle 3cont.

• It could be argued that if a child demonstrates
  proficiency on an assessment focused on
  comprehension (e.g., 2009 NAEP), that no other
  assessment is necessary conversely, foundational
  reading skills may not be developed or are in
  process for the population NARAP is studying.
• The overall goal is to develop flexible
  assessments to assess reading comprehension
  and/or foundational skills based on student
  performance.

15
Issues under Principle 3

• To what degree can component skills be measured
  independently?
• Comprehension is the primary goal for readers.
  If students are proficient in this area with
  accommodations, do we need to measure the
  foundational skills?
• If comprehension is our primary goal, should the
  comprehension score be weighted more heavily when
  foundational skills are also assessed?
• What feasible techniques are available for
  measuring foundational skills such as fluency or
  phonics knowledge in the context of large-scale
  assessment?

16
Issues under Principle 3cont.

• For students who do not achieve grade-level
  proficiency, what processes can be developed or
  applied to aggregate their performance on
  component skills into an overall measure of
  reading proficiency?
• Some components appear to be problematic for
  certain disabilities (e.g. phonemes and deaf
  students). Do we develop alternate definitions
  of proficient reading for these populations based
  on a better understanding of reading processes
  and performances?
• Are some skills less critical to measure than
  others?

17
Issues under Principle 3cont.

• If foundational skills were only assessed after a
  student had performed below proficient on
  comprehension, what proportion of ALL students
  (with and without disabilities) would be assessed
  on each of the foundational skills? Is this
  proportion small enough to assess students in
  small groups, individually, or via computer?
• Can some foundational skills be assessed together
  (e.g., decoding and phonemes)?
• If foundational skills are going to be measured
  only for students who are not proficient on an
  assessment of comprehension, can accommodations
  be allowed that invalidate the foundational
  skills (e.g., read aloud for decoding or extra
  time for fluency).
• If a student is not proficient on a measure of
  reading comprehension, should listening
  comprehension be assessed prior to measuring
  foundational skills?

18
Conclusions

• The three principles will guide NARAP in
  formulating the definitions of reading
  proficiency used within the studies.
• The definition of reading proficiency will be an
  organizing framework that supports states.
• Research will be conducted consistent with
  Principle 1 and addressing issues listed under
  Principles 2 3 several issues cannot be
  resolved through empirical research these
  require information from NARAP to impact policies
  on assessing students with disabilities that
  impact reading.

19
NARAP Web Site
The Principles Issues Paper is posted at
http//www.narap.info/