Assessment Director Meeting March 12, 2009

1
Assessment Director MeetingMarch 12, 2009

• Judy W. Park
• Associate Superintendent
• Data, Assessment Accountability
• Utah State Office of Education

2
Welcome

• Introductions

3
(No Transcript)
4
(No Transcript)
5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
Thank You

• . . . for hanging in there.

9
Federal Stimulus Package

• American Recovery and Reinvestment Act (ARRA)

10
Federal Stimulus PackageAmerican Recovery and
Reinvestment Act (ARRA)

• Governor will use some for budget short fall
  (back fill) 279 M
• HB 2 There is appropriated from Federal Funds -
  American Recovery and Reinvestment Act (H.R. 1,
  111th Congress) to the State Board of Education
  for fiscal year 2009-10 only, 207,176,000 for
  Social Security and retirement as provided in
  Section 53A-17a-125.

11
Federal Stimulus PackageAmerican Recovery and
Reinvestment Act (ARRA)

• Spring Money
• 50 of Total available funds
• IDEA Title I
• Coming to State on March 30
• Must get to LEAs by April 30
• Must spend by September 2011
• Use for existing statutes with some flexibility
• More information is coming in next 2 weeks

12
Federal Stimulus PackageAmerican Recovery and
Reinvestment Act (ARRA)

• Four Principles
• Spend quickly to save and create jobs
• Improve student achievement
• Transparency to the school level
• Separate accountability for how money is spent
• 4. Invest one time ARRA funds thoughtfully to
  minimize funding cliff

13
Federal Stimulus PackageAmerican Recovery and
Reinvestment Act (ARRA)

• More money in July and in the Fall
• Competitive grants
• Technology
• Statewide Data
• Teacher quality

14
Federal Stimulus PackageAmerican Recovery and
Reinvestment Act (ARRA)

• Fall race to the top
• Focus on increased student achievement
• Fall - Invest in what works innovation
• States already showing high achievement and/or
  achievement gains
• Money to replicate what you are already doing

15
2009 Legislative Session
16
1S SB 81Senator Margaret Dayton

• Concurrent Enrollment Program Amendments
• Bill Status Passed
• 53A-15-101. Higher education courses in the
  public schools Cooperation between public and
  higher education
• to ensure that students are prepared for college
  level work, an appropriate assessment is given
• Accuplacer
• What is the role of Assessment Directors?

17
HB 334Representative Merlynn Newbold

• Writing Assessment and Instruction
• Bill Status Passed House, Senate 1st reading
• a direct an online writing assessment for
  grades 6 5 and 9 8,

18
HB 334Representative Merlynn Newbold

• Writing Assessment and Instruction
• RFP
• Separation of formative and summative
• Formative
• Grades 4 12
• Implementation varies for each LEA
• Summative
• Secure prompts
• Responsibility of assessment directors
• Administration Training?

19
HB 334Representative Merlynn Newbold

• Writing Assessment and Instruction
• (6) (a) A school district or charter school, as
  applicable, is encouraged to administer an online
  writing assessment to students in grade 11.
• (b) The State Board of Education may award a
  grant to a school district or charter school to
  pay for an online writing assessment and
  instruction program that may be used to assess
  the writing of students in grade 11.

20
1S SB 159 Senator Howard Stephenson

• Math Education Initiative
• Bill Status
• Passed Senate
• House 2nd reading
• Grants to pay for Singapore math

21
1S SB 159 Senator Howard Stephenson

• Math Education Initiative
• Award grants to
• (i) school districts and charter schools to help
  pay for a school district's or charter school's
  costs of adopting Singapore math and
• (ii) institutions of higher education or
  nonprofit educational organizations for math
  teacher training programs

22
1S SB 159 Senator Howard Stephenson

• Math Education Initiative
• A school district or charter school that
  participates in the Math Education Initiative
• shall adopt Singapore math for grades
  kindergarten through six,
• may adopt Singapore math for grades seven and
  eight
• may phase in the adoption of Singapore math
• in grades K-8 over two or more years.

23
1S SB 159 Senator Howard Stephenson

• Math Education Initiative
• shall at the secondary school level, offer
• (i) honors and regular sections of algebra 1,
  geometry, and algebra 2
• (ii) an honors pre-calculus course that
  corresponds to college algebra and
  trigonometry110
• (iii) calculus and statistics courses that meet
  the standards for the Advanced Placement exams

24
1S SB 159 Senator Howard Stephenson

• Math Education Initiative
• Participation is Voluntary
• Competitive Grant process
• Grants up to 150 per student
• Grant monies to pay for Singapore math
  instructional materials and/or professional
  development
• shall administer math achievement tests at the
  beginning and the end of the school year

25
Budgets
26
100 Computer Based TestingCriterion-Referenced
Tests

• 2007 8
• 2008 50
• 2009 65
• 21 Districts and 42 Charter Schools have
  committed to test 85 or more of their CRTs via
  CBT. 
• 2010 ?

27
100 Computer Based TestingCriterion-Referenced
Tests

• 2009 - Scale and Equate from CBT only
• Innovative Item Types
• Hardware Options
• Technology funding - AARA

28
Computer Based TestingUBSCT

• Optional only for those schools who choose to
  be 100 CBT for UBSCT
• As early as October 2009?
• Pre-Equated
• Quick/immediate return of reading and math scores
• Quicker return of writing scores?

29
Digital SAMS UTREx Update
30
Changes to Clearinghouse Procedures for SY 2008-09

• USOE Data Assessment and Accountability
• (801.538.7953)
• john.brandt_at_schools.utah.gov

31
Objectives

• Increase Clearinghouse data quality
• Ensure on time (July 15) final data submissions
• Increase confidence in accountability reports
• Eliminate indecisions about resubmission of data

32
Major Changes for 2008-09

• Elimination of November Resubmissions.
• Data Quality Auditors will be available to help
  improve submitted data.
• New Clearinghouse Validation Reports will be
  available.
• More Formal Assurance from the LEAs that good
  data has been submitted.

33
LEA-USOE Major Data Flows
LEAs
July - Sept
Local SIS
May- July
Summary Validation Reports
Edits/Errors
May- July
AYP, CRT, U-PASS Reports Cognos, SERF
USOE Clearinghouse
CACTUS AP ACT SAT S3 CRT YIC Concurrent
Enrollment DWA Supplemental Reading UAA IOWA Dis
cipline
July -Sept
July 15
USOE Warehouse
Public
Sept 15
July 15
Ongoing
USOE Web Sites
USOE Partners
34
No More November Resubmissions

• Less than three LEAs were taking advantage of
  this option in any given year.
• The November option sometimes promoted
  insufficient attention to review of the data in
  June and July.
• LEAs need to review and correct data in June and
  July when the data are new and before the new
  school year.
• Resubmissions were very time consuming and risky
  for all concerned.

35
Data Quality Auditors

• Beginning in the spring of 2009, the USOE will be
  assigning two data auditors/analysts to work with
  LEAs in an effort to improve the quality of data
  flowing between the LEAs and the USOE.
• These analysts will help review Clearinghouse
  reports and the LEAs state data submission
  procedures. Their main focus will be helping to
  ensure the submitted data accurately represents
  the LEAs schools, students, personnel and
  programs.
• As the roles and operating procedures of these
  individuals become more well defined additional
  details will be available.
• The USOE welcomes suggestions in regards to this
  initiative.

36
New Clearinghouse Validation Reports

• There will be new Clearinghouse reports this
  spring to accompany the summary reports that are
  currently produced each time a LEA updates the
  USOE Clearinghouse.
• These new validation reports will compare last
  school years counts/totals (e.g. membership and
  demographic) with this school years
  counts/totals and indicate any that appear to
  deviate significantly.
• There are still some design details to be
  finalized, and the USOE is open for suggestions.
  

37
Current Clearinghouse Report
38
Current Clearinghouse Report
39
New Clearinghouse Validation Reports

• It may not be feasible to compare every data
  element/count in the current years report to
  last years report.
• There have been suggestions that some validation
  reports, based on new data, be added. One would
  be counts of the number of students by course
  type and the number of classes in that course,
  resulting in a preliminary class size statistic.
• While not based on the more sophisticated
  business rules used in final U-PASS reports (e.g.
  compensation for multi-grade and multi-section
  classes), this report could assist LEAs in
  reviewing and correcting data that impacts this
  important statistic. The next slide shows a
  prototype of such a report.

40
New Clearinghouse Validation Report sNew Class
Size Estimation for Validation Report
41
New Clearinghouse Validation Reports

• With the new Clearinghouse Validation reports,
  modified procedures for the Clearinghouse
  submissions may need to be defined. One
  possibility is that every LEA must submit at
  least one Clearinghouse file that can pass the
  preliminary edit/error reports and thus be
  eligible for a full summary and verification
  report cycle prior to July 8.
• This would give every LEA at least one week prior
  to the July 15 deadline to review the summary and
  validation reports and resubmit its Clearinghouse
  file if necessary. Modifying this procedure is
  important because LEAs will no longer be able to
  resubmit a clean-up Clearinghouse file in
  November

42
More Formal Data Assurance

• Consideration is being given to the idea of
  having each LEA sign an assurance letter
  indicating they have reviewed the Clearinghouse
  reports and believe their data to be correct.
• If the LEA cannot do this, it must submit a
  substitute letter indicating they recognize
  possible errors in their data but are allowing
  the USOE process the data and produce official
  state and federal reports from it.
• One of these two letters must be submitted no
  later than July 15.

43
Next Steps - Conclusions
44
Computer Based Testing Update

• Julie Quinn

45
Computer Based Testing Hardware Options

• Julie Quinn

46
Innovative Item Types

• Kevin King

47
UALPA Update

• Kevin King

48
Submission Dates

• March 2nd/6th
• Just passed
• Thank you
• Reports returned after April 20th
• May 8th Close of Testing Window
• The FINAL
• For ALL schools, for ALL students
• Reports returned after June 22

49
Recap of Procedures

• Ordering of Materials
• Thank you for being patient with some production
  shortfalls
• We are now fully stocked with all materials
• IF LEAs need more materials, please use UTOS
• - IF you are a multi-pilot district, please
  continue communicating as you have with Sarah
  Moore

50
Production Errors

• Affected Documents
• B1aB2b TAM
• B1aB2b Flipchart
• E1aE2c Flipchart
• These are isolated incidents of missing pages
• Please let us know ASAP
• Please send the materials back to the USOE

51
Recap of Procedures

• Answer Documents Sharons News
• Make sure all student files are in prior to
  sending in answer documents
• Package by level, form, grade
• - NOT by schools
• Need to bubble in teacher AND section on header
  sheets
• Need header sheets between grades
• Separate level 1 and level 2 forms

52
Recap of Procedures

• Answer Documents
• BUBBLE in FORM CODE!!!!!
• THANK YOU to LEAs who have done a good job at all
  of this.

53
Item Writing Evaluation
54
RFP Status
55
Thank You

• . . . for hanging in there.

56
Iowa Tests of Basic Skills

• 3rd Grade Spring Reading Test

57
Iowa Tests of Basic Skills

• 3rd Grade Spring Reading Test
• Testing Window
• Apr. 13 - May 14, 2009

58
Iowa Tests of Basic Skills

• 2009 Analyses
• http//www.schools.utah.gov/assessment/
• Documents/Results_Iowa_State_20090309.pdf

59
(No Transcript)
60
(No Transcript)
61
(No Transcript)
62
(No Transcript)
63
(No Transcript)
64
(No Transcript)
65
Pre-Equating
66
Pre-Equating

• 2009
• Completion of robust item banks
• 2010
• Pre Post Equate
• Scale Scores only after post-equating
• 2011
• Pre-Equate
• Immediate Scale Scores

67
Pre-Equating - 2010

• Embedded pilot items
• May have a slightly longer test (a few more items
  in the test section)
• Reduced number of sections because there will no
  longer be a pilot section
• Student level spiraling
• Easy for CBT
• All paper test booklets returned to USOE
• Each booklet must be accounted for

68
Scaling and Equating Considerations for Utah

• David Smith
• Assessment Development Coordinator

69
Background

• Utah score scalewith 160 established as the
  proficient scorewas established in 2004
• Subsequent years tests were equated back to this
  2004 base year scale
• This works as long as the construct we are
  measuring (e.g., the Core) and the test blueprint
  remains stable

70
But things change

• As you all know, the secondary ELA Core and
  mathematics Core have both changed
• ELA was made more specific at each grade level,
  while math was made somewhat more rigorous
• As with most things in life, everything is a
  matter of degree with score equating

71
What are our options?

• Depending on the degree of change, we have a
  variety of options. The greater the change, the
  fewer the options.

72
ELA Options

• If changes are judged to be minor, we can simply
  equate back to the previous score scale. In
  fact, weve been bridging from 2006 through
  2008 because of the slow evolution of the ELA
  CRTs to follow the change in Core.
• We can recognize that the Core/test has changed
  enough to warrant establishing a new baseline,
  but can still use the 160 scale with some sort of
  designation that the scale is new.
• If we think the changes are major, we should
  reset standards and create a completely new score
  scale.

73
Pros and Cons

• For every complicated issue, there is usually a
  single simple solution.except it is almost
  always wrong
• On the following slides, we discuss some of the
  pros and cons of each of the options

74
Option 1 Scale Equate

• Pros
• Can maintain score trends
• Easy for people to understand and interpret

• Cons
• Could be seen as papering over changes
• Could be psychometrically questionable if the
  construct did in fact change more than we thought

75
Option 2 Establish a new 160 scale

• Pros
• Communicates that we have a new scale, but not
  that new
• Allows the proficiency scale to have similar
  meanings as now, and we can move back to 160
  relatively quickly (within a year or two)

• Cons
• Could be hard for people to recognize that 160n,
  for example, is really different than 160

76
Option 3 Create a completely new scale

• Pros
• Could be the most psychometrically sound approach
• If the change was large enough, this would be
  truth in advertising

• Cons
• Would be hard to communicate since people just
  got used to 160
• Would have tests with different score scales

77
How do we decide?

• Empirical and judgmental analyses of the ELA core
  changes suggested that the changes were not
  major, which led the technical advisory committee
  to recommend standards validation rather than
  standard setting.
• Validation presumes that things will not be that
  different and attempts to check that assumption

78
Secondary ELA Validation Results

• Put on the 160 scale for comparison purposes, the
  recommended cutscores from the validation ranged
  from 158.5 to 163.5 depending on the grade level.
• Taking into account both measurement error and,
  more importantly, the variation (e.g., error)
  in panelist judgment, these recommended cutscores
  are all well within chance differences from 160.
• This suggests that both internal and external
  (the panelists) judgments support the fact that
  the new Core is not very different from the
  previous Core.

79
Conclusion and Input

• These results and prior judgments suggest that we
  can eliminate option 3. How do you feel about
  options 1 and 2?
• Option 1 Scale Equate
• Option 2 Establish a new 160 scale
• Option 3 Create a completely new scale

80
A Look Ahead to Math

• It appears that the math Core changes are more
  significant that ELA
• Currently, we are conducting the empirical and
  judgmental analyses to help us make these
  decisions
• We could possibly arrive at a different answer
  for math than ELA
• How would you feel about that?

81
UBSCT

• February 2009 Test Information
• Nolan Fawcett

82
Tentative Report Schedule

• SSID file cleanup
• 3/16-3/18
• Preliminary State results to USOE
• 4/8-4/13
• Slice Files and Reports released to website
• 4/16
• Paper reports shipped
• 4/23

83
Reports on Web site
84
Questions Answers