Assessment Directors Meeting August 10, 2006

1
Assessment Directors MeetingAugust 10, 2006

• Judy W. Park
• Assessment Accountability Director
• Utah State Office of Education
• July 2006

2
Welcome

• Introductions
• District Sharing Cards

3

• District
• Assessment Handbook

4
Put Students in the Drivers Seat

• Clear Core Curriculum
• Clear Test Purpose
• Quality Assessments
• Error Free
• Available on time
• Trained Administrators

5
Put Students in the Drivers Seat

• Testing Environment
• Standard Administration
• All tools available
• Immediate Results
• Quality Results Interpretation
• Test Results are clearly connected to preparing
  for the next test

6
Put Students in the Drivers Seat

• Clear Core Curriculum
• Clear Test Purpose
• Quality Assessments
• Trained Administrators
• Testing Environment
• Immediate Results
• Quality Results Interpretation
• Test Results are clearly connected to preparing
  for the next test

7
Fall Road Trip

• District Responsibility
• Invite A A
• Host meeting computer lab setting
• Audience Determined by District
• School Administrators?
• School Test Coordinators?
• School Counselors?
• Teachers?
• Agenda
• Put Students in the Drivers Seat
• Assessment Accountability Literacy
• Standard Test Administration Test Ethics
• Understanding Summative Test Reports
• Using U-PASS other data to inform instruction
  improve student achievement

8
Test Development

• August Blueprint
• September - Item Development
• October Item Review (Content)
• November Bias Sensitivity Review
• January Construct Pilot Test Form
• Take Test
• February - Data Review
• March Construct Operational Test Form Take Test
  
• May Standard Setting, Scoring, Reporting
• Review Test Results

9
Test Development

• August Blueprint
• Just as a blueprint of a house delineates the
  houses framework, the test blueprint outlines
  the tests framework.

10
Decision Guidelines for Blueprint Construction

• The standards and objectives selected for the
  blueprint must represent the entire curriculum.
• Only those objectives which can clearly be
  assessed in a multiple-choice format are selected
  for the blueprint.
• The number of test items assigned to each
  objective reflects the depth and breadth of the
  indicators within the objective.
• The following questions should be asked
• What proportion of the items on a test should
  represent each standard of the core curriculum?
• What proportion of the items should represent
  each objective?

11
Decision Guidelines for Blueprint Construction

• The total number of questions on a test must be
  large enough to provide a fair sample of student
  performance across the standards and objectives.
• We need to determine the appropriateness of
  making inferences about a students proficiency
  with respect to the standard and objective based
  upon x number of items.
• Items are aligned to the standards and objectives
  and not the indicators.

12
Test Aligns to Standards and Objectives

• Item writers first refer to the core curriculum,
  specifically to the portions of the core
  assessed on a CRT.
• Standards are very broad, educational goals.
• Objectives are broad statements of what a student
  should be able to do.
• Indicators are the descriptions of how students
  demonstrate that they can perform the objective.
• There are multiple indicators for each objective,
  which provide a broad spectrum of content skills
  assessed on each years test.
• Indicators provide a reference for composing test
  items.

13
Caveats

• Every blueprint shows
• The Standard
• The Objective
• The number of questions on the CRT for each
  objective
• Only those objectives which can clearly be
  assessed in a multiple-choice format are selected
  for the blueprint.
• The number of test items assigned to each
  objective reflects the depth and breadth of the
  indicators within the objective.

14
Assessment Updates

• IOWA
• UBSCT
• Math CRT
• UTIPS
• NAEP

15
Accountability Update

• Applied Math 1 2
• Not included in accountability calculations
• August 15?
• Electronic Reports on ftp site
• Research disk
• August 20 30
• CRT Reports Profiles electronic copy
• Contact Sharon Marsh if you want hard copies
• September 20
• Cumulative Student Report electronic copy
• Contact Sharon Marsh if you want hard copies

16
Accountability Update

• Accountability Meetings
• for
• U-PASS, AYP, AMAO
• Sevier District Office August 18
• Granite District Office August 22
• Weber District Office August 23
• Nebo Learning Center August 25

17
U-PASS 2006 Reports

• May Board Meeting
• Request to explore changes
• June Board Meeting
• Changes approved

18
U-PASS 2006 Reports

• Participation 95
• Status 80
• Progress 190
• Confidence Interval 95
• Projection - Identify 12.7 of schools

19
U-PASS High School Task Force

• June 5, 2006
• Minutes from this meeting are in red

20
2006 Timeline

• April 2006
• Meeting to review designs and make decisions
• May July
• Draft business rules
• August Assessment Directors Meeting
• Share business rules
• August October
• USOE run all of the schools through the business
  rules
• Districts may replicate analyses in their
  districts
• USOE run multiple scenarios for status/progress
• November 14Task Force Meeting!
• Meeting to review and clean up the business
  rules
• Review data and the multiple scenarios
• Make initial decisions for status progress

21
2007 Milestones

• January - February 2007
• USOE Rerun all reports and fix all business rules
• Districts replicate the rules for their districts
  
• March 2007Task Force Meeting!
• Meeting to review data and business rules
• Final planning meeting to approve business rules
  and cutscores
• April 2007
• Develop initial school designations
• Meeting for districts to review all school
  designations
• Make any final adjustments to business rules and
  calculations
• July 15
• Run all AYP/U-PASS/AMAO Reports
• August 15
• Release all Accountability to districts for 30
  day review
• September 15
• Release Accountability reports to the public

22
U-PASS Purpose

• To identify schools in need of assistance to meet
  state standards
• This is a minimum competency goal. It will be
  important to keep this in mind as we consider
  what values are reflected in the model.

23
General Principles

• The accountability system should be based on
  multiple indicators to provide a more complete
  picture of schools
• But should not overwhelm schools data capacity
• The measures should be combined using a
  compensatory framework to improve the reliability
  and validity of the decisions
• Every group should be required to meet an
  acceptable level of status or progress, but not
  both

24
Subgroups

• The same approach for holding subgroups
  accountable (i.e. the super-subgroup) in the
  3-8 system will be used in the high school system
  as well

25
A Stratified Approach

• Long-term plan that focuses on valuing the most
  important things that high schools should be
  doing
• Short-term plan that addresses things that can be
  measured fairly across schools and not create a
  huge data burden

26
The Short-term reality

• While the long-term system is appealing, we are
  probably several years away from implementing
  such an approach
• We need to find a short-term solution that meets
  the requirements of the law, supports the
  long-term view, is fair, and can be implemented
  in 2007

27
Short-term proposal

• This approach focuses on a specific checkpoint
  to identify schools most in need of assistance to
  meet state standards
• Participation, growth, and status should be
  required components of the system
• Subgroups must be held to the same standards as
  the full school

28
Participation

• Will be calculated using
• UBSCT--10th graders attempting all 3 subtests/all
  10th graders
• CRTsall CRT test takers/all students enrolled in
  CRT courses
• DWAall students tested/all 9th graders
• Agreed!!!!

29
HS Status Components

• 10th grade UBSCT scores inreading, writing, math
• CRTscounting all those scores earned by
• ELA--9, 10, 11
• Math CRTsAlgebra, Geometry
• It was suggested including all math CRT scores
  whenever they take it
• ScienceEarth Science, Biology, Chemistry,
  Physics
• All 9th grade Science CRT would count
• All 10th grade Science CRT would count
• Only the 11th grade CRT for students who did not
  take the science CRT in both 9th and 10th. (For
  the kids who have not received the 2 credits by
  11th grade, their 11th grade science CRT will
  count)
• Denominator 11th graders and below with 2 CRT
  credits or fewer

30
HS Status Components

• Math Coursesfull year credit for 9th and 10th
  grade students enrolled in a non CRT course
• Students earning credit/total number of students
  enrolled in non CRT classes in 9th and 10th grade
• What is a non CRT course? (Concurrent
  enrollment, Remedial math, CTE courses?)
• DWA9th grade
• UAAin 9th and 10th grades
• Graduation/completion rate
• Attendance (18 or more absences/year)????
• The 3-8 system has attendance as 15 or more
  absences per year.

31
UBSCT Status

• The denominator will be all 10th graders in the
  school
• The numerator will be the count of students
  passing the first administration

32
Graduation rate

• Same calculation method as AYP
• Will move to a longitudinal system when the data
  system allows for it
• What are the targets for graduation?
• Same as AYP?
• Will this simply be incorporated into the status
  calculation? If so, what metric will we use?
• For example, given the current status calculation
  plan a difference in grad rates between 80 and
  95 might NOT have a noticeable effect on overall
  status.
• Should graduation rate be a separate, conjunctive
  indicator?
• Graduation rate was not discussed we ran out
  of time.

33
Status Computation Questions

• How do we want to weight the various status
  components?
• All content areas weighted the same?
• Do math courses count the same as CRT scores?
• Graduation? Separate or incorporated?
• UBSCT weight
• This was not discussed we ran out of time

34
Growth components

• ELA--longitudinal growth from grades 8-11
• For Science and Math
• Use 8th grade scores as a pretest for the 10th
  grade CRT
• For UBSCT
• Use the same subject CRT score in 8th grade as a
  pretest score for the UBSCT subject area test and
  measure student longitudinal growth
• These would all be measured using a value table
  approach similar to the 3-8 assessments
• We could build a UBSCT improvement
  statisticevaluate the difference between initial
  pass rate and ultimate pass rate
• This was not discussed we ran out of time

35

• U-PASS High School Next Steps
• Committee Meetings will continue

36
AYP

• New Amendment
• An LEA is identified for improvement only when it
  misses AYP in the same subject and in all grade
  spans for two consecutive years, or the other
  academic indicator in all grade spans for two
  consecutive years.
• New Report

37

• Standard Testing Administration
• Testing Ethics
• What are the concerns?
• Lack of Training?
• Inappropriate behavior?
• What are the solutions?

38

• District
• Sharing

39

• New
• Accommodations
• Policy

40

• Utah
• Academic Language Proficiency
• Assessment
• UALPA

41
Proficiency Levels

• A P Pre-Emergent
• B E Emergent
• C I Intermediate
• D A Advanced
• E F Fluent

42
Students to be assessed

• All P (pre-emergent) students (A)
• All E (emergent) students (B)
• All I (intermediate) students (C)
• No A (advanced) students (D)
• No F (fluent) students (E)
• No Native Speakers tested

43
Students to be assessed

• Students double tested to bridge scores from IPT
  to UALPA
• The same student must take the IPT and the UALPA
  in the same time frame
• Minimal Requirement
• Double test 8 of P (A) students
• Double test 3 of E (B) students
• Double test 5 of I (C) students

44
Placement Test

• IPT used for Placement 06-07 year
• UALPA used for Placement in 07-08 year?

45
Test Materials

• Student Booklet
• 1 booklet for each grade span
• K - flip chart
• Speaking flip chart
• One flip chart for 1 6
• One flip chart for 7 12
• 1-2 consumable booklet
• 3-6, 7-8, 9-12 student booklet
• Booklet includes listening, reading, writing

46
Test Materials

• Answer document for each student
• K
• Teacher completed
• 1-2 consumable booklet
• 3-6, 7-8, 9-12
• Speaking teacher completed
• Listening, reading, writing - Student completed

47
Student Booklet
48
Answer Document
49
Test Materials

• Test Administration Manual (TAM)
• Includes basic training information
• Includes all administration information
• Examiner script
• Includes scoring guides for
• Speaking teacher completed
• K teacher completed
• Coordinators Manual
• Test distribution information
• Test return information
• Scoring information

50
TEST DESIGN
51
Test Design Proposal2007-08
52
Student Report Card
53
(No Transcript)
54
Scoring for Student Written Responses

• 2006-07
• All student written responses for reading and
  writing will be scored by contractor
• 2007-08
• All student written responses for reading and
  writing will be scored at the local level

55

• Yet to be finalized
• Writing scoring guide
• By December 2006
• Process Training for Districts to score writing
  in future years
• By May 2007

56
Training

• Administration Manual
• Provides all training materials (2 pages)
• State Training Meeting
• 2 Meetings - 2 choices
• October 5 (Utah Valley?)
• October 17 (Salt Lake Valley?)
• Train the trainer model
• Training Power-point provided
• Training for scoring speaking

57
Timeline

• July
• Item Selection Edits
• Test Direction Edits
• Form Construction begins
• August September
• Form construction completed
• All Test Materials completed
• Test printing completed
• New RFP

58
Timeline

• October
• Train the Trainer Workshops
• Testing materials distributed to districts as
  printing completed.
• Pre-print file to Sharon by October 15
• All testing materials distributed by
• October 30

59
Timeline Options

• March
• Score all tests administered by end of Feb.
• Minimum of 50 students assessed
• All double testing completed
• Measured Progress
• April
• Standard Setting Scoring
• Test Results
• May
• Score remaining tests
• Test Results

60
AMAO Accountability

• New Report
• All data taken directly from USOE
• Contractor and Computer Services do all scoring

61
(No Transcript)