Alaska Presentation

1
The Scoring JourneyWhat Happens After You
Return Your Test Materials to DRC
2
DRC Facilities
Operations Facility Brooklyn Park, MN
Scoring Center Cincinnati, OH
Scoring Center Woodbury, MN
Scoring Center Minnetonka, MN
3
Security
The following security measures are in place at
DRC

• Employees are issued key-card identification
  badges to gain access to buildings
• Visitors are asked to sign in and are also issued
  badges that must be worn
• The secure access system keeps a log of all
  persons entering our facilities, including all
  after-hours and weekend activity
• Unauthorized personnel are not allowed in the
  receiving, check-in, document processing, or
  materials assembly areas unless accompanied by a
  permanent staff member
• All incoming and outgoing shipments are logged in
  and logged out

4
Security

• Confidentiality of individual data is maintained
  at all times
• Client confidentiality and privacy are
  maintained. All client documents and products are
  stored in locked files when not in use
• All electronic files are maintained at
  password-protected work stations and are
  accessible only to key personnel on the project
  team
• Security requirements are discussed with vendors,
  who must certify adherence to these standards

5
Security

• Corporate headquarters and all associated sites
  are fully secure facilities
• All materials are developed, produced, processed,
  and stored in an environment secure from access
  by the general public or unauthorized staff
• DRC enforces strict security measures to prohibit
  unauthorized personnel from gaining access to
  client materials
• We have refined our security systems,
  disaster-recovery processes, and confidentiality
  procedures to be the best in the industry

6
Document Services

• DRC has complete, in-house capabilities to
  produce high quality scannable forms for either
  traditional OMR or imaged projects
• The Document Services Division is staffed by
  skilled professionals who understand the critical
  need for data integrity
• Greater scheduling flexibility by keeping all
  scannable printing on site
• Exacting quality control procedures
• Multiple shift operation

7
Production Workflow
1
2
3
4
4
5
Confidential and Proprietary
8
Operations
State-of-the-art operations systems and
experienced operations staff

• Focus
• High quality
• High volume
• On-time delivery
• Brooklyn Park Facilities
• Designed and built to support high volume
  document processing

9
Operations
Materials Management System

• Custom designed to support materials processing
• Barcode-driven system achieves the highest
  quality possible for secure document processing
• Materials are managed to the student level

10
Image Scanning and Scoring Technology

• Seven image scanners with a combined throughput
  of over four million sheets per week
• Processing capabilities, custom tailored to the
  scoring requirements of educational testing
  programs
• Captures a picture of a students constructed
  responses and electronically sends the image to
  our online scoring facilities
• Constructed responses scored online using PCs
• Produces reader reliability and quality control
  monitor reports that support all of our
  performance assessment projects
• Images of test booklets can be retrieved easily
  to accommodate client inquiries

11
Image Scanning and Scoring
Image Processing Experience

• Image scanning and scoring since 2000
• Custom-designed system

• Capacity
• Will scan more than 50 million sheets this year
• Will score over 20 million items via image this
  year

12
Material Tracking
Barcode Scanning

• DRC and Manna Freight use fixed and portable
  barcode readers as tools for shipping and
  managing materials
• Allows us to accurately track all materials
• Enables reporting of missing materials at the
  school or district level
• Ensures accountability of secured materials for
  an assessment project
• Reduces the time and labor needed to monitor and
  maintain security of materials coming into or
  going out of DRC

13
Performance Assessment
Performance assessment capabilities

• Twenty million essays/constructed response items
  scored in the last school year
• 1,500 readers employed at peak of scoring season
• Over 60 return rate of experienced, highly
  qualified, degreed scorers
• Consistently meets or exceeds clients deadlines
• Strong content expertise

14
Performance Assessment
Experienced performance assessment

• Combined experience includes designing and
  monitoring scoring of many large-scale
  assessments
• Assessments have included numerous content areas,
  scoring models, and procedures

15
Performance Assessment
Qualifications for scorers

• All DRC scorers have a minimum of a four-year
  degree
• Degree must be in a field closely related to the
  content area being scored
• Many of DRCs scorers have advanced degrees
• Preference is given to candidates with previous
  experience scoring large-scale assessments
• Approximately 550 professional test scorers from
  DRC assessed the constructed-responses for the
  Alaska tests last year.

16
Performance Assessment
Scorer training

• Intensive 2-3 day training session before scoring
  begins
• Rubrics and anchor responses serve as their guide
  through the training, qualifying, and scoring
  process
• Scorers must demonstrate their ability throughout
  the process
• If a scorer does not qualify, he/she is released
  from the project
• Daily quality control reports detail score point
  distributions and individual reader production

17
Performance Assessment
Scorer training continued

• Daily quality control reports detail score point
  distributions and individual reader production
• DRCs Team Leaders and Scoring Directors
  supervise the readers
• Supervisors read behind the scorers

18
Performance Assessment
Highly regarded and respected staff

• Our staff participates in
• Rubric development and clarification
• Anchor paper selection
• Reader recruitment and selection

• Scoring site set-up
• Reader training and monitoring
• Program evaluation

19
Performance Assessment
Reliability and accuracy

• Thorough training of readers and scoring leaders,
  customized for each project
• Regular monitoring of inter- and intra-reader
  reliability regularly
• Computerized systems developed by DRC to analyze
  and report on reader performance

20
Performance Assessment

• Each constructed response item has
• An item-specific rubric
• A scoring guide with annotated anchor papers
  representing each score point
• Training sets of responses
• Qualifying sets of responses
• Writing has all of the above, except it has a
  general rubric for the essay questions
• Alaska performance assessment
• Average training and qualifying time for readers
  was two days
• All of Reading, Math, and Alternate Assessment
  was scored in Minnetonka
• All of Writing was scored in Woodbury
• The same content staff supervised the work for
  the fall and spring assessments

21
Performance Assessment

• The number of readers and days needed to
  complete each Alaska Assessment are
• 2004 Fall High School Retest 100 readers for 8
  days
• 2004 Fall Field Test Grades 3-9 135 readers for
  10 days
• 2005 Spring HSGQE 330 readers for 9 days
• 2005 Spring SBA 340 readers for 9 days
• 2005 Spring Alternate Assessment 20 readers for
  9 days