NAEP SCIENCE INTERACTIVE

1
(No Transcript)
2

• NAEP SCIENCE INTERACTIVE
• COMPUTER TASKS - 2008

3
Measurement Challenges

• Developing tasks that match the framework
• Building tasks that measure both science content
  and process
• Developing substantive tasks that can be
  completed within 30 minutes
• Developing interfaces that are appropriate and
  engaging and that do not
• introduce construct irrelevance
• take too long to learn
• Developing assessment, NOT teaching tasks

4
ICT Development and Review Process

• Work at ETS, NCES, NESSI, Westat, Pearson,
  Fulcrum IT, and with external advisors
• Tryouts with students
• Various iterations of tasks

5
2008 Pilot

• Two Goals (broader than most pilot tests)
• Pilot test tasks
• Evaluate student engagement in and understanding
  of tasks
• Try out scoring processes
• Evaluate item performance
• Learn about issues related to delivery of
  simulation-based assessments on nationally
  representative samples

6
2008 Pilot Delivery Goals

• To use TestNav, an existing system from Pearson
• To use school equipment
• To use schools internet connection as much as
  possible to deliver content and retrieve results

7
Initial Challenges Delivering ICTs

• School hardware and connectivity
• Wide variety of hardware, OS, network
  connectivity, security policies
• Variety of applications running in background
• Pre-Assessment Visits, including installation of
  software, longer and more complex than PP
• Lack of incentive for school personnel to address
  these challenges

8
Problems With Initial Administrations

• Students unable to begin test
• Test sessions frequently aborted
• Installation and technical support burden
  untenable

9
Response to Delivery Challenges

• Adjusted desktop settings
• Switched to completely disconnected mode
  utilizing USB flash drives
• Use NAEP hardware instead of school computers as
  much as possible
• Support field administrators with IT staff

10
Delivery Status Update

• Significantly higher success rate
• First 3 weeks approximately 50
• Final 2 weeks approximately 90
• Combination of NAEP hardware and disconnected
  mode create much more predictable environment
• Issues of scalability

11
Looking Ahead to 2009, 2010, 2011

• Short-term Use NAEP hardware as much as
  possible, change desktop settings, work in
  disconnected mode
• Long-term Remove any untenable reliance on
  school hardware, software, or connectivity
• Memorandum describing options under development

12
The Tasks Themselves

• Field data not yet available
• DAC raised questions about the constructs
  underlying the extended tasks
• A few issues about interface comfort have surfaced

13
Next Steps for Task Development

• Reconvene external advisory panel and DAC
  subcommittee, and involve experts in simulation
  development from places like MIT
• Revamp tasks as needed over next 6 months
• Lock-down content no later than September 1

14
What to Expect for 2009

• Until we have full data, we will not be able to
  determine the most appropriate analysis approach
• We would recommend a special report on these
  tasks and what we have learned after the main
  release of results

15
View from the Field

• From January 28 to March 7, NAEP administered
  computer delivered assessments in 273 schools
• Sample
• 280 schools
• Grades 4, 8 and 12
• 7 schools declined
• Method
• Both online and flash drives

16
View from the Field

• Operationally, what does it mean to administer a
  computer delivered assessment?
• Staffing
• Training
• Resources at school

17
View from the Field

• Staffing for computer delivered assessments
• Skills needed for paper and pencil assessments
• Understanding of and experience working in a
  school environment
• Experience working with students
• Skills needed for computer delivered assessments
• Understanding of and experience working in a
  school environment
• Experience working with students
• Basic computer skills (flash drives)
• Computer knowledge to understand basic network
  infrastructures (online)
• Ability to transport and set up 15 laptops (NAEP
  provided equipment)

18
View from the Field

• Training for Online Assessments
• PowerPoint Demonstration
• Live demonstration (online)
• School computer lab simulation
• Review of and resolution common failures
• In addition
• Help Desk
• Troubleshooter field staff
• Pearson technical professionals

19
View from the Field

• Resources at School
• At the planning visit (online only)
• School/district computer specialist
• Time in computer lab to test computers
• On assessment day
• Use computer lab
• Computer specialist on standby (online)

20
View from the Field

• Reaction to computer administered testing?
• School Staff
• Students
• NAEP State Coordinators

21
View from the Field

• School Staff Debriefing
• School coordinators
• School computer specialist
• Overall reactions
• Pleased with student engagement in computer
  assessment
• Concerns about the amount of time that computer
  labs and classrooms were occupied generated the
  small number of negative reactions

22
View from the Field

• School Coordinators
• Brochure prepared them well
• Computer laboratory occupied too long
• Assessment process was time consuming

23
View from the Field

• School computer specialist
• Process is time consuming
• District coordination required in many schools
• Needed to grant access so students could write
  to the subfolder was complicated
• Would like more advance notice in future
• Need a back up plan
• Liked idea of using flash drives

24
View from the Field

• Administrator observations and student
  comments/questions
• Students at all grades were engaged
• Patient and understanding of technical
  difficulties
• Students did not use scratch paper
• Many students at all grades did not finish task
  in time allowed

25
View from the Field

• Grade 4 students had questions about
• Navigation
• How to play videos
• How to advance to the next page
• Double clicked and could not go back to the
  previous section
• Unsure about what was direction or part of task
• Other questions from 4th graders
• What does horizon, data mean
• How do you make the degree sign , 97
• Did not always understand directions

26
View from the Field

• Grade 8 and 12
• Could not advance until key questions were
  answered
• Found directions difficult or unclear
• Even older students did not complete tasks in
  time allowed

27
View from the Field

• NAEP State Coordinator comments
• Suggestions to improve planning visit
• If NSC has all technical requirements, can help
  network administrators to prepare
• More detailed instructions needed for school IT
  staff

28
View from the Field

• NAEP State Coordinator comments about the
  sessions
• Even with work done at planning visit, students
  could not access site or if access gained,
  students kicked off (online)
• Should schedule in mid morning, so computers can
  be tested again before assessment starts
• Too much time required of district/school IT
  staff
• Send a techy to each school
• Help desk support inadequate