Standard Setting: Procedures, Validation, and Documentation

1
Standard Setting Procedures, Validation, and
Documentation

• Marianne Perie
• Center for Assessment

2
Guidance on alternate achievement standards

• Regulation 200.1
• For students with the most significant
  cognitive disabilities, who take an alternate
  assessment, a State may, through a documented and
  validated standards-setting process, define
  alternate achievement standards
• Peer Review Guidance Critical Element 2.1
• States may define alternate academic
  achievement standards, provided those standards
  (1) are aligned with the States academic content
  standards (2) promote access to the general
  curriculum and (3) reflect professional judgment
  of the highest achievement standards possible.

3
Key Considerations

• Write clear performance level descriptions
  aligned with the state content standards The
  task of writing PLDs is similar to that for
  large-scale assessments. Should be written by
  people who understand the population and what can
  be achieved.
• Match the judgmental task of the standard-setting
  method to the test type Methods can focus on
  items, performance tasks, whole bodies of work,
  or students. Choose the methodology, or an
  adaptation, that best fits the assessment.
• Validate that the cut score best reflects the
  intention of the performance level descriptor It
  is important to conduct follow-up studies to
  ensure the right students are being placed in
  each performance level.
• Document the rationale and procedures used in
  setting the achievement standards. Documentation
  is an important and often overlooked step in the
  process.

4
Goals for this Session

• Provide detail on two important considerations
  for setting alternate achievement standards
• Developing PLDs
• Validating the cut scores
• Touch on methodology and choosing the best method
• Describe what should be included in the standard
  setting section of a technical manual

5
Achievement standard
4 Components

• Label
• Descriptor
• Student work
• Cut score(s)

6
PLDs

• Need to put into words how good is good enough
  for achievement of students with significant
  cognitive disabilities in the grade-level content
• Critical element 2.5 highlights the need to
  ensure alignment between its academic content
  standards and the alternate academic achievement
  standards
• Well-developed PLDs will serve both purposes

7
What is a well-developed PLD?

• To run a strong standard-setting workshop, you
  need a general level PLD
• Developed by experts
• More than just a policy definition
• Aligned with content standards
• Represent the highest standard possible for this
  population
• After the cut scores have been set, more
  specifics should be added to the PLD and
  exemplars should be provided
• Link directly to the items, rubrics, or student
  work categorized in each performance level by the
  cut scores
• Provide contextual information for teachers and
  other stakeholders

8
How to Develop Good PLDs

• Convene a panel of experts
• Special education teachers
• Others familiar with the content, curriculum,
  special education students, or assessment
• Discuss the students
• Review relevant research on special education
  learning and achievement
• Review sample student work
• Discuss the desired end product for the
  descriptor

9
PLD Considerations

• Are there basic criteria that you want to
  continue across levels
• Basic skills
• Enriched skills
• Generalizability of skills
• Degree of support required/effectiveness of
  support
• Interactions with others
• Remember to keep the focus on the content
  standards
• Consistency is not a content standard but the
  degree of consistency expected may vary across
  content standards and across levels

10
PLD Considerations Flexibility

• Flexibility in the learning and assessment goals
  creates particular challenges when trying to
  establish a common cut score with content-based
  meaning
• Curriculum for students with significant
  cognitive disabilities may include certain
  functional skills or non-academic skills, federal
  guidance states that achievement standards for
  alternate assessments must be based on academic
  content
• How do we classify students at the lowest
  functional levels?
• Sufficient score points for additional levels?
• A second set of PLDs?
• Must keep focus on providing all students with a
  reasonable opportunity to achieve proficiency

11
PLD Considerations Distinguishing Among Levels

• Quantitative versus Qualitative differences among
  the levels
• Can proficient students do more things than basic
  students or do different things? e.g., more
  content or greater depth of content
• Consider a rubric that allows for more points
  when a student performs the task correctly more
  often versus a rubric that allows for more points
  when a student can generalize from the task
• Also have the adverb approach, such as
  inconsistently versus consistently, or minimally,
  moderately, extensively

12
PLDs to cut scores

• PLDs can be written at any time in the
  development process
• We often argue that PLDs should be written before
  designing the assessment
• Cut scores are typically set after the assessment
  has been administered for the first time

13
Choosing a methodology

• General options
• Examine student work
• Evaluate the rubric
• Analyze the items
• Ordered by preferred approach
• Will depend on the format of the test

14
Standard Setting Workshop

• Choose standard setting method
• Match method to assessment characteristics (see
  handout)
• Convene panel of stakeholders
• Need to be experts in special education and/or
  assessment
• Review content standards and assessment
• Discuss PLDs and borderline student
• Train on and practice methodology
• Run 2-3 rounds with individual judgment,
  feedback, and group discussion
• Provide normative data
• Consider breaking out data by disability type
• Summarize results

15
After the Workshop

• Smooth the data
• After the workshop ends, state DOE might want to
  adjust the cut scores to align the impact data
  across grades or subjects
• Some argue that smooted cut scores should stay
  within error bands (consider SEJs and SEMs)
• Adopt the cut scores
• Standard setting panel makes recommendation but
  policy board adopts cut scores
• Policy board should be given information about
  what happened during the meeting, any adjustments
  made afterwards, measures of variance, and impact
  data

16
Validation

• Cut scores do not have validity but the
  interpretation and uses of the cut scores do
• Just as there are no absolute criteria against
  which specific cut scores can be evaluated, there
  are no perfect criteria for evaluating standard
  setting studies (Kane, 1994, 2001)
• But even though there are no absolute criteria
  you still must provide evidence that the cut
  scores are reasonable and appropriate

17
Validation (continued)

• Key questions
• Was the standard setting procedure internally
  valid?
• Do the cut scores divide students reasonably in
  terms of achievement?
• Do the effects of the achievement standards match
  what was intended?
• Were there any unintended consequences for using
  the achievement levels?

18
Validating the Procedures

• Was a documented method used?
• If method was adopted from another, were the
  modifications justified?
• Were all the typical steps applied?
• Was the panel composition appropriate?
• How much variation existed among the panelists?
• Was there evidence of convergence across rounds?
• What does the panelist feedback tell us about the
  clarity of the procedures, adequacy of training,
  completeness of the materials, and their
  confidence in the results?

19
Validating the Results

• Comparing results to external sources of
  information is a way of checking if the cut
  scores are at approximately the right level.
• Grades
• Teacher evaluations
• Comparing results to results using another
  methodology may provide useful information
• Different methodologies produce different results
  (Zieky, 2001)
• If differences are too large, the validity of the
  process may be questioned (Kane, 2001)

20
Validating the Results Using Another Methodology

• Consider a contrasting groups methodology in a
  subsequent year
• Ask special education teachers to review the PLDs
  and place each of their students in one of the
  categories
• Two calculations
• Compare cut scores based on teacher assignment to
  cut scores from original standard setting
  methodology
• Calculate the degree of mismatch between teacher
  categorizations and current cut score
  categorizations
• Gather more information about students who appear
  to be misclassified
• Are adjustments to the cut score needed?

21
Other Validity Evidence

• Another consideration is the reasonableness of
  the cut score
• Do the impact data make sense?
• Consider the impact data broken out by type of
  disability do they still make sense?
• Try to get multiple opinions on the
  reasonableness of the results using the adopted
  cut scores

22
Consequential Validity

• What effect is the use of the performance levels
  having on student opportunity to learn?
• Teacher professional development
• Pedagogical changes
• Student exposure to richer curriculum
• Are there any unintended consequences from using
  the performance levels?
• Certain content standards being de-emphasized
• Certain students being neglected
• Teacher retention decreasing

23
Validation Over Time

• Follow students over time
• Is improvement captured by the performance
  levels?
• Do the classifications over time make sense?
  (e.g., results that show a student to be
  proficient in grade 3, basic in grade 4, below
  basic in grade 5 and proficient in grade 6 would
  not be considered reasonable)
• Survey teachers/other stakeholders over time
• Are performance levels capturing true
  achievement?
• Do they see possibility for improvement for the
  students?
• What suggestions do they have for modifying PLDs
  and/or cut scores

24
Documentation

• From our joint Standards (AERA, APA, NCME)
• Document the PLDs, selection of panelists,
  training provided, ratings, and variance measures
  (Standard 1.7)
• Document the rationale and procedures for the
  methodology used (Standard 4.19)

25
Document the PLDs

• Document the development Who, when, how?
• Documentation should demonstrate that the
  alternate assessment PLDs
• Represent highest standard possible for this
  population
• Capture essential skills
• Align with state content standards
• Progress logically across levels (e.g., is
  Proficient appropriately higher than Basic)
• Progress logically across grade levels (e.g., is
  Grade 5 proficient sufficiently more advanced
  than grade 3 proficient)
• Represent knowledge and skills that can be
  evaluated by the assessment (e.g., dont discuss
  independence in the PLD if your assessment
  doesnt measure independence)

26
Document the Rationale

• Why did you choose the standard-setting
  methodology you did?
• Consider characteristic of assessments
• Separate items
• Full body of evidence
• Existence and complexity of rubrics
• Discuss types of judgments made
• Item level or holistic
• Focus on rubric or student sample work
• Document any logistical considerations
• Number of potential panelists
• Time available
• Availability of student data or sample work

27
Document the Procedures

• Documentation should allow someone else to
  replicate your study
• Provide exact instructions given to panelists
• Describe focus of group discussions
• Record any major issues or concerns of panelists,
  as well as driving factors
• Use evaluations effectively and record results
• Indicate if/when/what type of impact data were
  provided

28
Document the Selection of Panelists

• Documentation of panelists should include
• Number (minimum should match minimums used in
  large scale assessment for same methodology)
• Association with special education students
  (teacher, curriculum specialists, parents)
• Representation across the state (geographic
  regions, demographic diversity)

29
Document the Training

• Document the following steps
• Familiarizing panelists with the content
  standards
• Introducing the assessment
• Discussing the PLDs and borderline student
• Training on the standard setting methodology
• Practicing the standard setting methodology
• Evaluating panelists understanding of the
  materials and task

30
Document the Ratings and Variance

• Each round should consist of independent ratings
• Document for each round
• Each individual rating
• Central tendency of the ratings (mean or median)
• Variance in the ratings (range and standard error
  of judgments)
• SEM associated with each mean/median cut score
• Document any policy decisions made after the
  standard setting workshop was completed
• Smoothing
• Adoption

31
Document Validity of Cut Score Interpretation

• Document any follow up studies
• External evidence
• Teacher judgment
• Adjustments made to instruction based on
  interpretation of results
• Subsequent student performance

32
Conclusion

• Setting alternate achievement standards should
  follow the logic of the entire alternate
  assessment program
• Each step should be made in consideration of the
  other steps and the goals of the program
• Write clear appropriate PLDs
• Choose an appropriate methodology and implement
  it using best professional practices
• Validate the cut scores and classifications over
  time
• Document every step of the process