Vertical Alignment, The New Frontier: The Delaware Pilot Study

1
Vertical Alignment, The New Frontier The
Delaware Pilot Study

• Laurie Wise, HumRRO
• June 28, 2006
• 36th Annual CCSSO Conference
• On Large Scale Assessment

2
The Delaware Vertical Alignment Study

• CCSSO (the TILSA Collaborative) spent a year
  improving tools for alignment studies.
• Included automating test alignment processes and
  also papers on aligning alternate assessments and
  vertical alignment.
• See Aligning Assessment to Guide the Learning of
  All Students published by CCSSO for details
• At the conclusion of this project, Delaware
  sponsored a vertical alignment study.
• As a pilot test of proposed methods
• Also to meet operational needs for aligning
  expectations and performance level descriptions
  across grades.

3
What Is Vertical Alignment?

• Vertical alignment asks
• How are content standards/objectives related from
  one grade to the next?
• Knowledge or skills extended to wider range of
  content
• Deeper understanding (cognitive processes) for
  the same content
• New content and/or skills
• What more do students at the next grade need to
  know and be able to do?

4
TILSA Work onVertical Alignment

• Initial focus on supporting vertical scales
• Is content alignment sufficient to justify a
  vertical scale?
• How to label points along the vertical scale?
• Changed to focus on quality of vertical
  articulation
• Concerns about misuse of vertical scales
• Inferences about mastery of content not tested
• Scales will vary by content of items used in
  linking
• Other important needs for clarifying content
  standards and their relationship across grades
• Helping teachers talk across grades
• Clarifying test specifications within each grade
• Supporting the development of curriculum materials

5
Measuring Student Growth

• Current pressure to build vertical scales!
• Report growth for individual students.
• Assess each schools value added
• Gains versus status
• Are students learning (not have they learned)
• But what does the vertical scale measure?
• Combined curriculum content across grades?
• How can scale points be described in terms of
  content standards for different grades?

6
Nature of Content Alignment

• Applying Webbs Alignment constructs
• Categorical Concurrence
• What content is new? What content is continued?
• Range of Content
• Broadening or generalizing knowledge/skills
• Depth of Knowledge (DOK)
• Webb DOK ratings are somewhat grade-specific.
• Balance of Representation
• How does content emphasis vary across grades?
• Source of Challenge
• What needs to be clarified about the standards?

7
Quality of Content Alignment

• Content standards are not clearly articulated
  across grades if
• Related standards are not clearly differentiated.
• What new knowledge or skill is required?
• One or both standards may not be described in
  sufficient detail.
• Differences in terminology are not explained.
• Different words for the same skill?
• Terminology drifts.
• The meaning of terms appears to be expanded.
• Specific objectives are omitted at some grades.

8
Gathering Content Alignment Data

• Who should judge?
• Same experts who developed the content
  frameworks.
• Teachers who deliver the curriculum covering this
  content
• What are judges asked to do?
• Make judgments about individual standards.
• Grade-to-grade comparisons (summed up later)
• Within specific content areas or subscales
• To limit search for similar standards
• Identify related prior-grade standard(s)
• Describe relationship
• Qualitative description of what is new or added.
• Code relationship type (Extended, Deeper, New,
  Same)
• Identify quality issues
• Importance of added knowledge or skill
• Source(s) of challenge (Clarity of added
  knowledge or skill)

9
Reporting Vertical Alignment

• Detailed reports
• Content Maps
• List of specific challenges (articulation quality
  concerns)
• Summary indicators
• Rater agreement measures
• Concurrence - new content
• Range - of skills broadened
• Depth - of skills deepened
• Balance - of standards with few/many objectives
• Challenge Average rating flagged with comment

10
Simplified Content Map
11
Initial Uses of Vertical Alignment

• Improving Clarity of the Content Standards
• Improving consistency in language across grades
• Identifying gaps in content coverage across
  grades
• Pointing out areas where specific objectives are
  not clearly distinguished for adjacent grades
• Developing Integrated Curriculum
• Laying groundwork for content at the next grade
• Reinforcing objectives from prior grades
• Facilitating Teacher Communication
• As students are passed from one grade to the next
• About mastery of prerequisite lower grade
  standards

12
New Use

• The problem
• Delaware needed to set performance standards for
  assessments of new grade-level expectations in
  each grade and subject required by NCLB.
• With 5 levels X 8 or 9 grades X 2 subjects
  (initially), thats a lot of levels
• Concern that performance goals will be
  significantly easier at some grades than at
  others.
• Approach
• Use vertical alignment ratings to identify areas
  where increases in expectations are not clear
• Rate the importance of increases at each grade to
  identify areas of focus for developing the
  performance level descriptions used in standard
  setting. (3-point scale)

13
Panel Design

• The concept paper suggested that panelists should
  be experts in the standards/objectives being
  aligned.
• Delawares grade-level expectations were brand
  new, no one outside the developers was an expert.
• The workshop acquired an additional new purpose
  Introducing the expectations to teachers who will
  help students meet them.
• Table Design So many expectations, so little
  time!
• Panelists were divided into 3 language arts
  tables and 4 mathematics tables.
• Each table included 2-3 elementary, middle, and
  high school teachers each.
• Tables were assigned to start with different
  standards (Day 1) and then completed one or two
  additional standards (Day 2).

14
Panel Design

• Sub-Group Design
• Panelists initially worked in subgroups by grade
  level
• Whole-table discussion of transition grades (5
  and 8)

15
Ratings by Grade-Level
16
Importance Ratings Example
17
Complexity Dimensions

• What more was needed
• For Reading, many expectations were identical
  from one grade to the next except for the grade
  level of the text used.
• For mathematics, process dimensions (e.g.,
  mathematical reasoning) had the same expectations
  for all grades, but increases in complexity were
  assumed.
• What was done
• Outside content experts led discussions for each
  subject.
• Assessment items were used to illustrate
  increases in complexity on the assessment.

18
Key Lessons Learned

• Panel Design
• Using teachers not familiar with new expectations
  led to
• Good feedback on the clarity of the new
  objectives
• New insights for teachers in looking beyond their
  own grade
• Low level of consistency in the ratings
  themselves.
• The revised table design
• Increased active participation at all grade
  levels
• Resulted in alignment ratings for a large number
  of expectations, in short time, and included
  independent ratings by two or more tables for
  each expectation.

19
Key Lessons Learned

• Instruments
• Importance Ratings
• Promoted discussion of increases in expectations
  across grades
• Flagged cases where the increases were not judged
  important
• Were not sufficiently reliable to make
  fine-grained distinctions
• Nature of Linkage
• Objectives coded Same or New were flagged for
  review
• Distinction between broaden and deepen not
  useful in supporting development of performance
  level descriptions
• Clarity of Linkage
• Useful in flagging objectives that were not clear
  or not well understood

20
Key Lessons Learned

• Training
• More training is needed for type of linkage
• Idea that changes in verb signaled deepening
  worked somewhat
• Several instances where panelist skipped back to
  earlier grades
• More structure to practice rating exercises
• Need to practice within grade levels as well as
  across whole tables
• By round two, most tables were functioning
  effectively
• Introduce expectations ahead of time
• The need to introduce new grade-level
  expectations led to a delay between initial
  training and actual ratings.

21
Next Steps

• How to think about/align skills that increase
  across a complexity not easily defined
• Reading with increasingly complex (grade level
  referenced) texts
• Math reasoning skills with increasingly complex
  content
• Automating the vertical alignment process
• Collecting alignment judgments
• Building the content maps
• Producing summary indicators

22
Checking Your Vertical Alignment

• Do your standards need an alignment checkup?
• Identify needs for revision.
• Add explanatory text.
• Define common or evolving content for a vertical
  scale.
• Suggest labels for points on the vertical scale.
• Satisfy NCLB requirements for coherent grade
  level expectations!
• If so, volunteer for further pilot testing!

23
References on Vertical Alignment

• Wise, L. Alt, M. (2005). Assessing vertical
  alignment, in Aligning Assessment to Guide the
  Learning of All Students Six Reports,
  Washington DC Council of Chief State Schools
  Officers.
• Wise, L., Zang, L., Winter, P., Taylor, L., and
  Becker, D. (2005). Vertical alignment of
  grade-level expectations for student achievement
  Report of a pilot study, in Aligning Assessment
  to Guide the Learning of All Students Six
  Reports, Washington DC Council of Chief State
  Schools Officers.