Alternate Assessment Achievement Standard Setting Task Force II

1
Alternate Assessment Achievement Standard Setting
Task Force II

• October 17, 2006
• Hilton Downtown
• Indianapolis

2
NCLB Regulations andAlternate Assessment

• Information to Assist Local Education Agencies in
  Understanding Regulations and Current Practice

Distributed April 2006 by the Indiana Department
of Education, Division of Exceptional Learners
3
Our intentions have been to develop an alternate
assessment system that meets the requirements of
regulatory policy while maintaining sufficient
psychometric integrity and practical utility for
families, classroom teachers, and school systems.
Practical Utility
Policy
Psychometric Sufficiency
4
Guiding Principles - Indiana

• All children can be successfully challenged to
  learn.
• It is expected that all children will make
  educational gains
• through out their schooling.
• Students show their abilities in many different
  ways.
• We value professional judgment of student
  performance in
• many contexts over time as being valid and
  useful.
• We respect the importance of the teacher as the
  agent for
• learning.

5

• July 2002 (Title I of NCLB)
• Sec. 200.6(a)(2) a State's academic assessment
  system must provide one or more alternate
  assessments for those students with disabilities
  (as defined under section 602(3) of the
  Individuals with Disabilities Education Act),
  who, in the determination of the student's IEP
  team, cannot participate in all or part of the
  State assessments, even with appropriate
  accommodations.

6

• July 2002 (continued)
• Sec. 200.8(a) individual student reports must
  describe achievement measured against the State's
  academic achievement standards.

7
The Alternate Assessment needs CUT SCORES? NCLB
says YES.

• Content Standards What students should know and
  be able to do.
• Achievement Standards What is good enough and
  what is not good enough as defined through
  performance descriptors and cut scores.
• Performance Levels Labels each level of
  achievement
• Performance Descriptors Describes each level of
  performance
• Cut Score Scores that separate the different
  levels of performance

8
Achievement Levels
Graphic represents what is required by NCLB
There must be a score of basic which would
count as not proficient. (ISTEP uses no
pass)
ISTEP
Pass
No Pass
ISTAR
Pass
9

• Federal Peer Review Requirements
• Submitted November 2005 by Indiana

2.1 Has the State approved/adopted alternate
academic achievement standards for students with
the most significant cognitive disabilities? Evide
nce a description of the process for defining
alternate achievement standards and documentation
in State Board of Education minutes of formal
approval (occurred 09/05/05) 2.3 Do these
alternate academic achievement standards include
at least three levels of achievement, including
two levels of high achievement (proficient and
advanced) that determine how well students are
mastering a States academic content standards
and a third level of achievement (basic) to
provide information about the progress of
lower-achieving students toward mastering the
proficient and advanced levels of
achievement? Evidence Documentation that refers
to the levels, descriptions, and cut scores that
make up the States alternate academic
achievement standards
10
Approval for State Assessment Systems

• National map of status of states
• Education Week article
• eSchool News Online article

11

• August 2005 Statewide Committee Develops
  Alternate Achievement Standards for ISTAR

• Over 40 representatives from school districts,
  universities, and parent groups joined in the
  process.
• Participants were given information as to the
  federal regulations and guidelines and the
  process of standard setting.
• Participants reviewed the data distribution from
  2004 results.
• Participants were given opportunity to discuss
  the consequences of their decisions. Although
  everyone appreciated the idea of setting high
  expectations for all students, there was a
  general understanding that this was only the
  beginning of a long journey to find a valid way
  to include students with special needs in the
  accountability system.
• Participants worked in small groups and then
  came together to compare progress. This was
  repeated until the group had offered a workable
  plan that spanned grades 3-10 in three clusters.

12
Rationale Of Cluster Group Cut Scores From Peer
Review Guidance Document

• When examined across grades, the alternate
  achievement standards are not likely to show the
  same clearly defined advances in cognitive
  complexity as the achievement standards set for
  the regular assessment. States are expected to
  rely on the judgment of experienced special
  educators, administrators, higher education
  representatives, and parents of students with
  disabilities as they define alternate achievement
  standards in a manner that provides an
  appropriate challenge for students with the most
  significant cognitive disabilities as they move
  through their schooling.

13
Blue line is average expected progress, one grade
each year. Pink connected dots are average ISTAR
scores for those eligible in 2004. Heavy pink
lines represent about one standard deviation
range of those scores.
10
8
6
4
2
K
F2
F1
B2
B1
GRADES
14
2004 RESULTS
Passing ISTEP
Eligibility Determination
1
counted as proficient In 2004
(Curve theoretically represents achievement of
special education students)
15
(No Transcript)
16
2005 RESULTS
Passing ISTEP
Eligibility Determination
1
counted as proficient In 2005
counted as basic In 2005
(Curve theoretically represents achievement of
special education students)
17
ISTAR Results 2005
Note Difference between participation total and
sum of achievement levels accounts for students
found ineligible.
18
Reconciling NCLB Regulations andIndianas
Guiding Principles

• Do we believe that we must find a valid way to
  include students with special needs in the
  accountability system?
• YES
• Do we believe that we have found all the answers
  to the questions of how to balance policy,
  psychometric validity and practical utility?
• NO

19
What should school personnel do with the results
indicating a basic academic achievement level?

• Learn the performance descriptors that define the
  lowest level of achievement.
• Verify that raters took quality time to consider
  even the slightest evidence of academic
  performance.
• Verify that instructors are giving all children
  opportunities to express themselves and attempt
  the most basic concepts that have been
  articulated in the lowest level of achievement
  for the alternate assessment in mathematics and
  language arts.
• Plan academic opportunities for children who
  might be able to attempt the challenges.

20
(No Transcript)
21
No effect
22
ETHNICITY
Other 3
Hisp 4
Black 18
White 75
23
GENDER
Female 38
Male 62
24
-1 SD 1
-2 SD 12
-3 Standard Deviations 87
COGNITIVE DEVIATION
25
EXCEPTIONALITIES
Other 6
Autism 17
Multiple 15
Mild 12
Severe 11
Moderate 39
26
After Applying these cut scores
27
21
28
18
29
TASK I Reset Achievement Levelsfor Fall 2006

• Supporting Materials
• Cut Score Packet
• Survey of Consequences sheet
• How ISTAR scores counted sheet
• Yellow booklet of base statements and assessment
  elements

30
TASK I Reset Achievement Levelsfor Fall 2006

• Red Grades 3 5
• Yellow Grades 6 8
• Blue Grades 9 10
• Group Process
• Understand that we need to have cut scores.
• Review and discuss the materials.
• Make a recommendation of if the cut scores should
  be adjusted, and if so, how.
• Record the main points of the discussion and be
  prepared to report back to the larger group.

31
LUNCH
32

• Principles of Inclusive Assessment
• Identified by NCEO (Thurlow, Quenemoen,
  Thompson, Lehr, 2001)

• All students with disabilities are included in
  the assessment system.
• All students with disabilities are included when
  student scores are publicly reported, in the same
  frequency and format.
• 4. The assessment performance of students with
  disabilities has the same impact on the final
  accountability index as the performance of other
  students, regardless of how the students
  participate in the assessment system.
• 6. Every policy and practice reflects the belief
  that all students must be included in the state
  and district assessment and accountability
  systems.

33

• What does proficient mean?

In an ACHIEVEMENT STANDARD model, proficiency
means as good or better than the cut score.
What would proficient mean in a growth model?
34
STATIC
DYNAMIC
INDIVIDUAL
AGGREGATE
35

• Other Ways of Thinking

• States are permitted under NCLB to establish
  multiple performance standards for alternate
  assessments although most employ a common single
  set of standards. In terms of NCLB, as long as
  less that one percent of the total student body
  counts as proficient, there is flexibility in how
  that proficiency is determined.
• Growth models models of accountability that
  measure progress by tracking achievement scores
  of the same students from one year to the next to
  determine progress.
• Value-added models take into account growth and
  background characteristics to determine the
  effect of a particular program.
• North Carolina Based on an initial score, a
  four year trajectory is created for each student
  who did not perform at the proficient level.
• Tennessee Each subgroup has a projected
  percentage of students who must meet the approved
  annual measurable objective

36
Categories of Reporting For Early Childhood to
OSEP

• Maintained peer level
• Achieved peer level
• Improved (near peer)
• Improved (but not near peer)
• Did not Improve

37
(No Transcript)
38
Value Table Neutralfrom the Center for
Assessment
39
Progress Value TableUtah Department of Education
40
Initial Value Table Committee Recommendation Alas
ka Department of Education
41
Language Arts
GRADES
42
(No Transcript)
43
Mathematics
GRADES
44
(No Transcript)
45
Would growth help?
46
TASK II Propose Alternative Ideas for Measuring
Proficiency

• Group A Work within current method with
  adjustments
• Group B Work with a model of individualized
  growth
• Group C Work with a model of aggregate growth
  
• Group Process
• Review pros and cons
• Discuss models that may hold promise
• Outline best ideas and what would need to be done
  in pursuit of these ideas
• Be prepared to share thoughts with large group