Patterns in Child Outcomes Summary Data: - PowerPoint PPT Presentation

1 / 58
About This Presentation
Title:

Patterns in Child Outcomes Summary Data:

Description:

ENHANCE. Project launched by the Early Childhood Outcomes Center (ECO) and SRI International. Funded by the U.S. Dept. of Education, Institute for Educational ... – PowerPoint PPT presentation

Number of Views:161
Avg rating:3.0/5.0
Slides: 59
Provided by: ecta57
Learn more at: http://ectacenter.org
Category:

less

Transcript and Presenter's Notes

Title: Patterns in Child Outcomes Summary Data:


1
Patterns in Child Outcomes Summary Data

Analytic Approaches and Early Findings from the
ENHANCE Project
Cornelia Taylor, Lauren Barton, Donna
Spiker September 19-21, 2011
Measuring and Improving Child and Family Outcomes
Conference New Orleans, LA
2
Todays session
  • Provide a brief update about ENHANCE
  • Identify the purpose and approach of the state
    data study
  • Describe some preliminary findings from initial
    states involved in the state data study
  • Explain how other states could examine their own
    data in the same way as that presented
  • Discuss any emerging implications for validity of
    the COS and for interpreting individual state
    data

3
Origins of ENHANCE
Need for Outcomes Data Challenging to Collect
4
Origins of ENHANCE
COS Process
Implemented gt 40 States, Little Systematic
Validation for Use in Accountability
Need for Outcomes Data Challenging to Collect
5
Origins of ENHANCE
COS Process
?
Implemented gt 40 States, Little Systematic
Validation for Use in Accountability
Investigate Learn
Need for Outcomes Data Challenging to Collect
6
Early Evidence
  • Belief in potential for COS process to be valid
    based on
  • Existing literature team-based decision-making
    can be reliable and valid
  • Existing literature teams are effective in
    identifying individual childrens functioning so
    that they can plan and deliver appropriate
    services
  • Early data from states pilot sites, small ns
    showing similarity in distributions, sensible
    patterns for subgroups
  • Anecdotal data from trainers participants reach
    decisions fairly easily and consistently

7
ENHANCE
  • Project launched by the Early Childhood Outcomes
    Center (ECO) and SRI International
  • Funded by the U.S. Dept. of Education, Institute
    for Educational Sciences July 1, 2009
  • Series of studies designed to find out
  • the conditions under which the Child Outcomes
    Summary (COS) Process produces meaningful and
    useful data for accountability and program
    improvement
  • the positive and/or negative impact of the COS
    process on programs and staff
  • what revisions to the form and/or the
    process are needed

8
Four ENHANCE Studies
  • Comparison with Child Assessments
  • Team Decision-Making
  • Provider Survey
  • State Data Study

9
Studies 1-334 Project Data Collection Sites
  • 17 Part C (Birth to 3)
  • Illinois
  • Maine
  • Minnesota
  • New Mexico
  • Texas
  • North Carolina
  • 17 Part B Preschool (3-5)
  • Illinois
  • Maine
  • Minnesota
  • New Mexico
  • Texas
  • South Carolina

10
Comparison with Child Assessments Study
  • Goals
  • Compare COS ratings to BDI-2, Vineland-II scores
  • Program Entry
  • Program Exit
  • Compare conclusions from COS and assessments
  • Sample
  • 108 children - birth to 3
  • 108 children - 3 5 years
  • Study Status
  • Recruiting families
  • About ½ of the sample enrolled
  • See expected variability in sample (ages,
    disability types) and initial COS
    ratings/assessment scores

11
Team Decision-Making Study
  • Goals
  • Learn more about the implementation of the COS
    process, including how the team reaches a
    decision about a rating and what is discussed.
  • Do COS ratings assigned match the developmental
    level of the behaviors presented in the meeting?
  • What is team understanding of outcomes and rating
    criteria?
  • Sample
  • 180 children each from Part C Part B 619
    ½ entry ½
    exit meetings
  • Study Status
  • Starting data collection now in about ½ the sites
  • 19 videos received
  • Expect to start coding videos Summer 2012

12
Provider Survey
  • Goals
  • What processes are being used to determine COS
    ratings?
  • What is the impact of the COS process on
    practice?
  • What have providers learned about the COS?
  • What else would be helpful?
  • Sample
  • All providers in the program who
  • participate in the COS process
  • are invited to participate
  • Study Status
  • Developing survey content
  • Survey expected Spring 2012

13
State Data Study
  • Goals
  • Analyze characteristics of COS data and
    relationships to other variables
  • Look for consistency in patterns across states
  • Examples of Questions
  • Are patterns in COS data across states consistent
    with those predicted for high quality data?
  • How are COS ratings related to hypothesized
    variables (e.g., disability type) and not to
    other variables (e.g., gender)?
  • How are team variables related to COS ratings?
  • Sample
  • All valid COS data within the state for a
    reporting year
  • 15-18 states conducting all analyses
  • Additional states sharing select analyses

14
State Data Study Status
  • Refined procedures for gathering data tables by
    gathering data from a preliminary group of 6
    states
  • Mostly states used procedures and generated data
    tables
  • A few provided formatted data files for SRI to
    analyze
  • Beginning to analyze data from that preliminary
    group
  • Soon will request data from other states in state
    data study and permission to use relevant data
    additional states have already analyzed and shared

15
State Data StudyPreliminary Data from 5 States
  • 3 Part C (Birth to 3)
  • 3 Part B Preschool (3-5)

16
How would these data analyses be conducted?
  • States would send data to SRI annually
  • de-identified data files OR
  • aggregate output or reports from a set of
    requested analyses
  • Examples of analyses include
  • the distributions of entry and exit COSF scores
  • relationships between outcomes
  • relationships between outcomes across time
  • relationships of outcome scores to other factors
    such as disability and gender

17
What data would I need to submit?
  • Data collected at entry and exit from Part C and
    Part B 619 programs
  • COSF ratings
  • Additional child descriptors (e.g. race, gender,
    primary disability)
  • Variables that describe the setting or
    composition of the services

18
How will I submit data?
  • De-identified data files
  • Templates developed in MS Excel
  • Submitted through a secure server
  • Analyzed data
  • Table shells developed in MS Word and MS Excel
  • Submitted through secure server or emailed

19
Who do I contact for more information?
  • Cornelia Taylor
  • cornelia.taylor_at_sri.com
  • (650) 859-3092

20
Questions? Comments? Reactions?
21
Entry rating distributions
22
Entry Rating Expectations
  • What should entry ratings look like?
  • Should they differ across outcomes?
  • Where do most of the ratings fall?
  • How much should the extremes of the scale be used
    ( 1 or 7)?

23
Entry Data Analysis
  • The following data are from 3 Part C programs and
    2 Part B programs
  • All data are from 08 09
  • The data are entry cohorts
  • i.e. all children who entered during the FFY

24
Part C 08-09 entry ratings across states Outcome
A
1 2 3
4 5 6
7
25
Part C 08-09 entry ratings across states Outcome
B
1 2 3
4 5 6
7
26
Part C 08-09 entry ratings across states Outcome
C
1 2 3
4 5 6
7
27
Outcome A Average Entry Ratings
28
Outcome B Average Entry Ratings
29
Outcome C Average Entry Ratings
30
Things to notice
  • The difference in distributions between Part C
    and Part B are largest for Outcome C
  • Children in Part B enter with higher ratings

31
Part C 08-09 average ratings across outcomes
32
Things to Notice
  • Variations in patterns across outcomes

33
Conclusions Across Part C and Part B
  • More that ½ of all children enter with a COS
    rating of
  • 3,4 or 5 across outcomes.
  • An average of 12 of children enter at with the
    very lowest (1) or the very highest (7) across
    outcomes.
  • The typical entry distribution has most children
    towards the middle of the distribution.

34

Pattern Check if the distribution of entry
scores in your state seems to be heavily weighted
towards one end or the other of the distribution.
No Action Interpretation You may be serving a
population that is higher or lower functioning
that other states.
Action Interpretation Your providers may be
systematically misunderstanding the definition
of COS rating points.
35
Additional Entry Analysis
  • Correlations between entry ratings
  • Cross tabs of entry ratings by
  • Program
  • Primary disability
  • Race/ethnicity

36
Exit distributions
37
Part C 08-09 exit ratings across states Outcome A
1 2 3
4 5 6
7
38
Part C 08-09 exit ratings across states Outcome B
1 2 3
4 5 6
7
39
Part C 08-09 exit ratings across states Outcome C
1 2 3
4 5 6
7
40
Part B 08-09 exit ratings across states Outcome A
1 2 3
4 5 6
7
41
Part B 08-09 exit ratings across states Outcome B
1 2 3
4 5 6
7
42
Part B 08-09 exit ratings across states Outcome C
1 2 3
4 5 6
7
43
Outcome A Average Exit Ratings
44
Outcome B Average Exit Ratings
45
Outcome C Average Exit Ratings
46
Part C 08-09 average exit scores across outcomes
(state n 3)
47
Part B 08-09 average exit scores across outcomes
(state n3)
48
Things to Notice
  • Variation in ratings across outcomes
  • The exit distribution is shifted toward a higher
    rating than is the entry distribution
  • For Part B, the average percent of children with
    a rating of 7 is much higher for Outcome C than
    for the other two outcomes

49
Pattern Check the distribution of exit scores in
your state is not skewed towards the higher end
of the rating scale.
  • No Action Interpretation You may be serving a
    lower functioning group than other states
  • If this interpretation is true, it should also be
    apparent in your entry distribution

Action Interpretation The children in your
programs may not be making expected gains.
50
Entry-Exit Paired Distribution
  • Choosing a metric for looking at paired
    distributions
  • Progress categories
  • Side-by-side entry exit comparisons
  • Both of the above can be completed using the COS
    calculator 2.0

51
Exit rating minus Entry rating
  • How many points the childs rating changed
    between entry and exit?
  • What would you expect to see?

Exit Rating Entry Rating Exit rating Entry rating
7 3 4
4 4 0
5 2 3
3 4 -1
52
Part C exit score entry score 08-09
-4 -2 0
2 4
53
Part B exit score entry score 08-09
54
Things to Notice
  • Most childrens ratings increase 1, 2, or 3
    points, or they stay the same
  • Very few children have ratings that decrease
  • However, more children have ratings that decrease
    in Part C than in Part B

55
Pattern Check if a large percentage of children
in your state make large increases in their
ratings
No Action Interpretation Your programs are very
effective and children make large gains (verify!).
Action Interpretation Providers are not using
the scale correctly and may be inflating exit
ratings and/or deflating entry ratings.
56
Additional entry-exit analysis
  • Correlations between entry and exit
  • Progress categories by other variables (e.g.,
    disability type, primary language)

57
Summary of pattern checks
  • The distribution of entry scores in your state
    seems to be heavily weighted towards one end or
    the other of the distribution
  • The distribution of exit scores in your state is
    not skewed towards the higher end of the rating
    scale.
  • A large percentage of children in your state make
    large increases in their ratings

58
Find out more
  • ENHANCE Website
  • http//ENHANCE.sri.com
  • ECO Center Website
  • http//www.the-ECO-center.org
  • Contact ENHANCE staff
  • E-mail ENHANCE_at_sri.com
Write a Comment
User Comments (0)
About PowerShow.com