Operationalizing Intent

1
Operationalizing Intent

• A State Perspective from the
• Kentucky Department of Education

• Council of Chief State School Officers
  Large-Scale Assessment Conference
• San Francisco, CA
• June 28, 2006

2
Setting a Kentucky Historical Context

• 1990 Kentucky Education Reform Act signed
• Assessment and Accountability System (KIRIS)
  established
• 1998 Legislation around assessment and
  accountability program passed
• Transition to Commonwealth Accountability Testing
  System began
• Due to expiring assessment contracts, August 2005
  CATS RFP released
• March 2006 assessment contracts approved

3
Pressures for Modifying the System

• Seven Steps Forward in Assessment
  A Tool for Conversations with Kentuckians
• A clearer and more focused Core Content for
  Assessment (subset of Program of Studies)
• A way to promote locally-applied student
  accountability
• The right balance of items (open response to
  multiple choice)
• Hold the line on time (testing and reporting)
• A measure to help forecast college success
• Other steps monitor each students progress
  improve writing assessments and scoring create
  new ways to assess arts and humanities and
  practical living/vocational studies

4
Pressures for Modifying the System

• Federal (No Child Left Behind) Requirements
• Annual testing of Reading and Mathematics grades
  3-8
• Annual alternate assessment linked to grade-level
  core content reporting separate scores in reading
  and mathematics
• State Legislative Requirements
• Statute (1998) discusses a sound longitudinal
  measure
• Senate Bill 130 (2006) requires readiness test
  for high school and college, WorkKeys and ACT
  remediation and accelerated learning plans and
  inclusion of scores in state accountability
  calculations
• House Bill 197 establishes a pilot for
  end-of-course assessments in high school
  mathematics

5
Transition Whats New for Kentucky?

• Content standards (Program of Studies and Core
  Content for Assessment)
• Prioritization, refinement of Core Content for
  Assessment
• Addition of a cognitive dimension to
• encourage rigor and relevance in Kentucky
  classrooms
• support learning and understanding expected in
  Proficient and Distinguished student performance
• Kentucky Core Content Test design (common core
  with matrix)

6
Transition Whats New for Kentucky?

• Analytic scoring for writing (better student
  information)
• Assessment contracts (3 contracts, 1 MOA, 6
  vendors)
• Additional assessment components
• Annual reading and mathematics (grades 38)
• EPAS (Explore, Plan, and ACT)
• End-of-Course pilot in mathematics at high school

7
Balancing Depth of Knowledge of Core Content

• Refinement/prioritization of the Core Content
• Where to spend your nickels (given the
  opportunities for assessment in the test design)?
  
• Are all content standards equally important?
• What content is so critical that it is important
  for all students to master?
• As students move from grade level to grade level
  what content is key to success or to continue to
  be proficient in the following year?
• What is the impact of grade span versus annual
  testing on prioritizing?
• Annual provides more opportunity to assess Core
  Content
• Grade span may have content that aligns to
  multiple grades

8
Balancing Depth of Knowledge of Core Content

• Addition of the DOK ceiling for testing
• What is the highest level of DOK that students
  are expected to demonstrate on the state
  assessment given the limitations of test time and
  item format?
• DOK enters the state assessment 2006-07
• The test will reflect a range of DOKs from year
  to year, and the DOKs reflected within a
  subdomain for a content area will vary from year
  to year.
• For most content areas, the ceiling for the
  assessment will be a DOK of 2 or 3.
• For high school reading, there are two DOK level
  4s listed as a desired goal.
• The writing assessment, particularly the
  portfolio, will include entries that reflect a
  level 4.

9
Developing Test Blueprints

• Drafting a Blueprint
• During the Core Content refinement process,
  teachers with staff developed a draft test
  blueprint
• Blueprint defines percentage of items by
  reporting category or the relative emphasis of
  the content in the assessment. Where do you
  spend your nickels?
• Blueprint and Core Content remains draft until
  used with item development work (Final--August
  2006)
• Evaluating existing item pool with teachers
• Conduct mock alignment activity
• Establish DOK designations
• Match items to refined Core Content
• Information from evaluation/alignment used to
  tweak Core Content/Blueprint

10
Developing Test Blueprints Next Steps

• Creating test specifications
• Distribution of DOK across the items within an
  assessment year (setting targets for DOK)
• Representation of the content within the common
  set of items
• Emphasis of content within standards
• Defining item specifications
• Identification of assessable content for each
  standard
• Implementing the test design
• Do more with multiple choice
• Transition plan to achieve DOK targets over time
• Recognition of the limits of tests to assess DOK 4

11
Implications and Instructional Effects

• Promote the role of DOK as a classroom
  instructional guide.
• Emphasize that the Program of Studies reflects
  the expectation that the depth of instruction and
  student learning should be at a DOK level 4.
• Prepare students to perform on the assessment
  whether at a DOK Level 2 or 3 by using DOK 4 as
  the goal for instruction.
• Expose teachers to training and experience with
  DOK.
• Use content and DOK to model best practices for
  instruction and assessment.
• Focus schools on required curricular goals
  (Program of Studies) beyond what is targeted for
  assessment (Core Content for Assessment).
• Make the curriculum decisions that are best for
  students.

12
Contact Information

• Rhonda L. Sims, Director
• Kentucky Department of Education
• Office of Assessment and Accountability
• Division of Assessment Support
• 18th Floor, Capital Plaza Tower,
• 500 Mero Street,
• Frankfort, KY 40601
• Telephone [phone]
• Fax [phone]
• Email Rhonda.Sims_at_education.ky.gov