Identifying Students in Need of Modified Achievement Standards and Developing Valid Assessments

1

• Identifying Students in Need of Modified
  Achievement Standards and Developing Valid
  Assessments

2
Who are the students needing modified achievement
standards?

• .and thoughts for developing eligibility criteria

Sue Bechard and Judy Snow January 16,
2008 Washington D.C.
3
Who are the students needing Modified Achievement
Standards (MAS)?

• Can include
• 2 of the total student population who can be
  counted as proficient on MAS,
• students with disabilities, from any of the 13
  disability categories,
• students who are addressing grade level content
  standards on their IEPs, but are not expected to
  meet grade level achievement standards in the
  current year
• students who need less difficult test items,
  covering the same breadth of content.

US Dept of Ed., 2007
4
Determination of eligibility for MAS.

• Must consider.
• objective evidence
• multiple measures of progress over time
• IEP goals that are based on grade level content
  standards
• providing students the opportunity to show what
  they know and can do on an assessment that is
  based on grade-level academic achievement
  standards.
• Must not consider.
• a specific disability category
• racial or economic background

US Dept of Ed., 2007
5
Population Identification Issues

• On average, students with disabilities comprise
  approximately 10 of the total student
  population.
• If 10 - 1 9, what characteristics should be
  used to distinguish the students appropriate for
  the 2 option within this group?
• Are there test performance distinctions?
• Are there specific learning characteristics?
• Are there specific learning needs for access to
  the general curriculum?
• What are the modified expectations relative to
  grade level content which distinguishes this
  population?

6
Population Identification Issues (cont.)

• What is the purpose of the AA-MAS?
• To allow students to do better (AYP improvement)?
• To provide better information for instructional
  planning (data on what students know)?
• To increase student self-esteem?
• To provide greater alignment between cognition,
  expectations, instruction and assessment?
• Is an AA-MAS needed at every grade level/content
  area?

7
Selected results from prior research.

• Colorado Report from the HB 05-1246 Study
  Committee, December, 2005
• New England Compact (RI) Enhanced Assessment
  Grant, 2004-2006
• Montana General Supervision Enhancement Grant,
  2005-2007

8
CO report Low Performers Who Score in the Bottom
1/3 of Scale Scores

• Students who score in the bottom one third of
  scale scores on CSAP are almost twice as likely
  to be Black or Hispanic as students of other
  ethnicities.
• Only 60 of students with IEPs scoring at lowest
  possible scale scores were able to be matched
  with a test the following year thus, they may be
  more mobile than their counterparts who score at
  higher levels.
• For those students scoring in the bottom
  one-third of scale scores, and where a match the
  following year was able to be made, it was found
  that these students did make substantial
  longitudinal growth.

9
CO report Students with IEPs Who Do Not Make
Longitudinal Growth

• On the Colorado CSAP Reading Test, there were 250
  students (of 444,407) across grade levels that
  were determined to be Students in the Gap.
• On the CSAP Math Test, there were 658 students
  (of 444,910) that were determined to be Students
  in the Gap.
• The CSAP as currently administered may not
  reflect their academic achievements however, if
  appropriate accommodations and more intensive
  instruction were provided, these students too may
  make more gains.

10
Georgia EAG

• Also looked at snapshot vs. longitudinal growth
• Low Performing lowest performance level in at
  least one assessment
• Persistently Low lowest performance level for
  three consecutive years

Melissa Fincher, July 2007
10
11
Rhode Island (New England Compact - NEC) EAG

• Teacher judgments of class work were compared to
  test performances and revealed two gaps of
  students performing below proficient
• Performance gap
• The test may not reflect classroom performance.
  Teachers see students performing proficiently in
  class, but test results are below proficient.
• Information gap
• The test may not be helpful for instructional
  planning. Teachers rate students class work as
  low as possible and test results are at chance
  level. No information is generated on what
  students can do.

Parker Saxon, 2007 Bechard Godin, 2007
12
NEC EAG data sources

• State assessment data grade 8 mathematics
  results from two systems
• General large-scale test results
• Demographics (special programs, ethnicity,
  gender)
• Student questionnaires completed at time of test
• Accommodations used at time of test
• State special education data
• Disability classification
• Free/reduced lunch
• Attendance
• Classroom teacher data
• Individual interviews
• Judgments of all students classroom work

13
NEC EAG findings

• The Information Gap in grade 8 mathematics
  comprised 2.3-4.3 of the total population
• Included non-disabled students.
• Test performance
• Students mostly guessed on the test items.
• Most used multiple accommodations.
• Teacher perceptions
• These students operate below grade level in
  class.
• Teachers are not surprised by their low test
  results.
• There is a disconnect in what is tested vs. what
  is taught.
• These students need more supports in the
  classroom.
• Student perceptions
• Think the test is harder than their classroom
  work.
• They try hard on the test.

14
NEC EAG Special program status of students in
the Information Gap
Non-gap comparison students who performed at
chance, on the test but higher in the
classroom. The majority of students performing at
chance were students with IEPs.
15
NEC EAG Disability designations of students in
the Information Gap

• Learning disabilities
• Lower percentages of students with SLD were in
  the information gap than in the general
  population.
• Other disabilities
• deaf/blind
• multiple disabilities
• hearing impairments
• mild to moderate cognitive disabilities
• combinations of disabilities.

16
Montana GSEG, 2005

• Students in the Sample
• Grade 5 students statewide
• Census sample of
• Students with an IEP
• Who took Spring 2006 Grade 4 math CRT
• A few (13) who scored well on the Alternate
  Assessment also included, selected by scores and
  recommendations from IEP teams
• CRT-M 672 students, CRT 199 students

Montana Office of Public Instruction and Measured
Progress
17
MT GSEG, 2005 data sources

• Pilot test results
• Student survey
• Test administrator survey
• Standard setting results
• Recorded discussions of standard setting
  panelists
• Interviews with standard setting panelists

18
MT GSEG 05 Test Information Functions
19
MT GSEG 05 Performance Level Comparisons, CRT-M
vs. CRT
Up Three Levels 7 1 All 7 went to Advanced
Up Two Levels 120 18 All 120 went from below proficient to proficient or advanced
Up One Level 282 42 88 went from nearing to proficient, 96 went to advanced
No Change 234 35
Down One 23 3 11 went from proficient to nearing proficiency
Down Two 5 .7
Down Three 1 .1
20
MT GSEG 05 student and teacher surveys Difficulty

• Students taking the CRT-M found the test slightly
  less difficult when compared to classroom content
  than students taking the regular CRT.
• Teachers felt the modified test should be
  modified more to reach the students having with
  the greatest challenges.

21
MT GSEG 05 feasibility question Is the CRT-M a
better measure?

• Students answered more items right
• Student scores went up
• Students moved up to another proficiency level
• Validity indicators improved
• More data analysis and study to determine which
  CRT-M students benefited most

22
Selected considerations from current research.

• Montana ( NEC) EAG, 2007-2009 Adapting Test
  Items to Increase Validity of Alternate
  Assessments Based on Modified Achievement
  Standards
• Montana GSEG, 2007-2010
• Identifying Students in Need of Modified
  Achievement Standards and Developing Valid
  Assessments

23
MT EAG, 2007

• Focus on high school reading comprehension to
• determine the processing requirements of test
  passages and items (use coding strategy)
• describe the cognitive abilities and challenges
  of the target population
• conduct cognitive labs
• develop item modifications based on cognitive
  variables

24
Preliminary feedback from Expert Panel (01-11-08)

• Some noted cognitive variables
• Abstract reasoning that relies on information
  from entire passage
• Long passages that require sustained attention
• Limited experience with multiple meanings of
  vocabulary words
• Dense passages that require large amounts of
  working memory
• Location in the passage where necessary
  information is found to answer the question
• Irrelevant information in passage makes mapping
  and sorting difficult
• Emotional content difficult for students with ED
  to process
• Answers to questions not found in passage (e.g.
  reliance on prior knowledge)

25
Montana GSEG, 2007

• Focus on middle school reading and mathematics
  to
• Identify students in need of modified achievement
  standards (MAS).
• Determine what content knowledge the student is
  lacking to achieve proficiency
• Develop dynamic online assessment that provides
  scaffolding based on distractor selection .

26
Students who will be included in the MT GSEG
study samples

• Middle school reading and mathematics
• Sample for analyzing items and distractors All
  students who took the tests of interest,
  disaggregated
• Sample for cognitive labs convenience sample of
  48 students (24 per content area)
• Sample for pilot test Approximately 5 of the
  total population

27
MT CRT Grade 10 Reading Example, 2007
28
Implications of research for identification of
target population

• Use of performance data
• Longitudinal performance data
• Students who are so low performing, nothing is
  known about them
• Match between classroom performance and test
  performance
• Distractor analyses

29
Implications of research for identification of
target population (cont.)

• Use of other data
• Teacher judgment data
• Opportunity to learn variables
• Mobility
• Attendance
• Program placement
• Performance data analyzed by cognitive modeling
  information
• Data from standards-driven IEPs

30
Data collected for The Whole IEP Process (C.
Massanari)

• What is the desired outcome for this student?
• Three to four years from now
• Students desired post-school outcome

What are the skills and knowledge essential to
meeting the desired outcome?

• What are the expectations of the general
  curriculum relative to the students age/grade?
• Content
• Expectations for learning and demonstration of
  learning
• Extracurricular activities or events available

31

• How do skills and knowledge essential to meeting
  the desired outcome compare with the general
  curriculum, including content and expectations
  for learning?
• Where are the similarities/connections?
• Where are the differences?
• Where within the general curriculum, including
  extracurricular, are the opportunities for
  learning the needed skills and knowledge?

• What are the students present levels of
  performance?
• What skills and knowledge does the student
  already possess?
• What other strengths does the student present?
• What are the areas of challenge?
• What accommodations, modifications, or other
  supports have proven beneficial for this student?

• Given all the information we have discussed thus
  far, what do we think are reasonable goals for
  this year?
• What are the objectives for each goal?
• What instructional accommodations are needed?
• What modifications to the general curriculum are
  needed?
• How will progress be reported and how often?

• Given the information we have discussed thus far,
  how will the student participate in state and
  district-wide assessments?
• With peers as given
• With peers and with accommodations or
  modifications
• Alternate assessment

32
Also, consider how model of 1 eligibility
guidelines might apply

• For example Montana's eligibility questions for
  the CRT-Alt.  The student MUST
• Program Have an active IEP
• Learning characteristics Have cognitive
  abilities and adaptive behaviors which require
  substantial adjustments to general curriculum
• Learning objectives and expected outcomes Focus
  on functional application of skills
• Delivery of instruction Requires direct and
  extensive instruction

33
Consider how model of 1 eligibility guidelines
might apply (cont.)

• Montana's eligibility questions for the CRT-Alt.
  Decisions must NOT be based on
• Excessive or extended absence
• Disability category
• Social, cultural, or economic differences
• Amount of time receiving special education
  services
• Expectation of failure on general test

34
So2 eligibility considerations might address

• Learning characteristics
• What are the cognitive abilities and adaptive
  behaviors of the target population?
• What adjustments are needed for the student to
  participate in the general curriculum (e.g.,
  accommodations/modifications)
• Learning objectives and expected outcomes
• How does the student demonstrate application of
  learned knowledge, skills, and abilities?

35
2 eligibility considerations (cont.)

• Delivery of instruction
• What are the deconstructions of constructs
  necessary for the student to master the grade
  level content?
• What adjustments must be made to simplify the
  materials used in instruction?
• Academic achievement
• How is the progress of this student different
  from the pattern of progress typical for all
  students at the targeted grade level?

36
References

• Bechard, S. and Godin, K. (2007). Identifying and
  Describing Students in the Gaps in Large-Scale
  Assessment Systems. Paper submitted for
  publication.
• Colorado Department of Education. (2005,
  December). Assessing students in the gap in
  Colorado Report from the HB 05-1246 Study
  Committee. Denver Author.
• Montana Office of Public Instruction and Measured
  Progress. (2007, May). Determining the
  Feasibility of an Alternate Assessment Based on
  Modified Achievement Standards A Planning
  Project and Pilot Test Final Report for
  Montanas General Supervision Enhancement Grant
  CFDA 84.373X Priority B. Helena MT.
• Parker, C. E., Saxon, S. (2007). They Come to
  the Test, and There is Nothing to Fold Teacher
  Views of Students in the Gaps and Large-Scale
  Assessments. Paper submitted for publication.
• Title IImproving the Academic Achievement of the
  Disadvantaged Individuals With Disabilities
  Education Act (IDEA). Final rule. 72 Fed. Reg.
  1774817781, pts. 200 and 300 (2007, April 9).