SES Data Collection Methods and Multi-State Results

1
SES Data Collection Methods and Multi-State
Results

• Allison Potter
• Steven M. Ross
• Center for Research in Educational Policy
• The University of Memphis

2
Research Focus
Overall implementation efforts and outcomes by
state, district, and local school educators Are
providers

• Raising student achievement in reading/language
  arts and mathematics?
• Communicating effectively with LEAs, principals,
  teachers, and parents?
• Aligning tutoring curriculum to standards?
• Developing instructional plans for students?
• Serving the needs of ELL/Special Education
  students?

3
District/Local Data Collection
Surveys

• Online survey system for LEAs, principals/site
  coordinators, teachers, and providers
• Login information e-mailed to LEAs
• Login information mailed to schools with parent
  surveys

• Paper-based surveys for parents
• Mailed (FedEx) to schools for distribution

4
District/Local Data Collection
Student achievement analysis

• Request SES student names, identifiers (SSN), and
  demographic information from LEAs
• LEAs send confidential student data to SEA
• State department data division adds test scores
  to SES student files (pretest and current year)
• Comparison group test scores are pulled and added
  to data file
• Student identifiers are removed (or encrypted)
  and sent to CREP for analysis

5
Data Collection Tools

• Selected survey questions

• What was the start date of provider services?

• In which subjects did your students receive
  services from this provider?

• Are you employed by the provider for which you
  are completing this survey?

How often does the provider

• Communicate with you during the school year?

• Meet the obligations for conducting tutoring
  sessions?

6
Aggregate Responses

• 2005-2006 School Year Evaluations

• 4 states (KY, LA, TN, VA)

Respondent groups

• 148 LEAs
• 209 Principals/Site Coordinators
• 753 Teachers
• 1,388 Parents
• 54 Providers

7
Data Collection ToolsAggregate sample responses
8
Data Collection Tools

• Aggregate sample responses

District and Local School Implementation

• All LEAs notified parents in their district of
  their rights under NCLB and the SES eligibility
  status of their child.
• The majority of parents were pleased with the way
  their LEAs helped them obtain SES for their child
  and felt they had enough time to make an informed
  choice on providers.
• Some parents asked for help from school leaders
  in choosing an SES provider and found it
  difficult to get information on program/provider
  details from their local school.

9
Data Collection Tools

• Aggregate sample responses

Parent perceptions

• Some parents were pleased with the amount of
  communication from the provider and with the
  progress reports sent home with their child.
• A few parents expressed frustration at attempts
  to sign their child up for services. They told
  of providers never calling them back and quitting
  mid-way through tutoring, with no prior
  notification.

10
Data Collection Tools

• Aggregate sample responses

Parent perceptions (continued)

• Some parents felt the at-home providers took
  advantage of their lack of knowledge on parental
  rights under NCLB and did not fulfill their
  obligations.
• A few providers acted in an unprofessional manner
  at the home of the student.
• Some parents were asked by providers to pay for
  services or the tutoring would be stopped.

11
Data Collection Tools

• Aggregate sample responses

Teacher perceptions

• The majority of teachers did not have enough
  information about providers serving students in
  their school to respond to questionnaire items.
• A few teachers saw school-based online providers
  letting children view inappropriate websites
  during their tutoring services.

12
Data Collection Tools

• Aggregate sample responses

Teacher perceptions (continued)

• Teachers noted grammatical and spelling errors in
  many reports they saw from providers.
• A few teachers witnessed providers grouping
  students in large (20 students) groups with only
  one tutor per site, with students fighting and
  running throughout the school.

13
Data Collection Tools

• Aggregate sample responses

Teacher/Principal perceptions

• Many teachers and principals were unaware of
  which providers were working with their students,
  citing no communication with them during the
  year.
• Many school leaders described aggressive attempts
  by providers to gain access to student names and
  to school sites.

14
Data Collection Tools

• Selected survey questions

Overall assessment

• I believe the services offered by this provider
  positively impacted student achievement

• Overall, I am satisfied with the services of this
  provider

15
Data Collection Tools

• Aggregate sample responses

16
Data Collection Tools

• Aggregate sample responses

Overall assessment

• Parents were the most satisfied group of all
  those surveyed, with the majority expressing
  gratitude at the efforts tutors made to help
  their child succeed.
• Many parents, teachers, and principals expressed
  frustration that provider services did not start
  sooner in the year, with many students not being
  served until March or April.
• A few LEAs noted providers were not returning the
  required paperwork on student attendance but were
  demanding in their requests for payment.

17
Data Collection ToolsSingle state parent
responses n 3,523
18
Data Collection Tools

• Provider survey selected questions/responses

• Describe the format of your services
• Program duration - about 6 weeks
• Setting - school site, online, home
• Format - small groups, individual

• Describe qualifications of tutors - most
  included using certified teachers and background
  checks

• List information regarding students served, goals
  achieved, and tutoring sessions attended -
• Most reported students achieving their goals if
  they attended the complete program.

19
Rubric of Overall Evaluation of Provider
Effectiveness
20
Decision Tree for SES Providers
Probation I