Using the ECO Child Outcomes Summary Tool

1
Using the ECO Child Outcomes Summary Tool

• Kathy Hebbeler
• ECO Center at SRI International

Prepared for the State TA Call, October 20, 2005
2
Important Information

• We will not be going over the material on slides
  4 to 37 on the call. It is for your information
  and will be discussed at the OSEP Early Childhood
  Meeting post-season on outcomes in December.
• State options are indicated with a ?.

3

• Go to Slide 38 ?

4
ECO Framework for Developing an Outcomes System

• 1. Identify purposes
• 2. Identify outcomes
• 3. Specify evidence statements
• 4. Identify measurement approaches (including
  how to transform the data)
• Implement procedures for data transmission
• And other steps..

5
Decision Purpose of the outcomes system

• Produce data for OSEP
• Provide evidence of effectiveness of program
  (state legislature, ICC, etc.)
• Provide information for program improvement
• Etc.

6
Decision Identify child outcomes areas

• Given OSEP child outcome areas
• Positive social emotional skills (including
  positive social relationships)
• Acquisition and use of knowledge and skills
  (including early language/communication)
• Use of appropriate behaviors to meet their needs
• State decision Other outcome areas?
  Sub-outcomes?

7
Decision Identify family outcomes areas

• Given OSEP family indicators
• Percent of families participating in Part C who
  report that EI services have helped the family
• Know their rights
• Effectively communicate their childrens needs
• Help their children develop and learn
• State decision Other outcome areas?

8
Child outcome areas What states are doing

• Adopting the 3 as is
• Adding sub-indicators under the outcomes
• ?

9
Family outcome areas What states are doing

• Add to/expand the OSEP indicators
• Add to/expand the ECO family outcomes
• E.g., quality of life
• ?

10
Evidence Statements OSEP only or More?
11
Decision What kind of evidence statements does
state want to be able to produce?

• Evidence Statement - a statement that
  incorporates a statistic and provides evidence as
  to whether not an outcome has been achieved

12
Decision What kind of evidence statements to
produce?

• Given OSEP indicators/measures (evidence
  statements)
• a. of infants and toddlers/preschoolers who
  reach or maintain functioning at a level
  comparable to same-age peers
• b. of infants and toddlers/preschoolers who
  improve functioning but are not in a
• c. of infants and toddlers/preschool children
  who did not improve functioning

13
OSEP Indicators and Measurement Categories
Group a maintained or reached typical
5 year old level
2 year old level
Entry
Exit
14
Indicators and Measurement Categories
5 year old level
Group b made progress but
2 year old level
Entry
Exit
15
OSEP Indicators and Measurement Categories
5 year old level
Group c did not make progress
2 year old level
Entry
Exit
16
OSEP Indicators and Measurement Categories
Group a maintained or reached typical
5 year old level
Group b made progress but..
Group c did not make progress
2 year old level
Entry
Exit
17
Decision Does the state want to be able to
produce any other evidence statements?

• Note this does not necessarily mean collecting
  more data. It could mean recording or analyzing
  the same data in a different way.
• If the state wants outcome data for anything
  other than to meet the federal reporting
  requirements, the answer to this question is
  almost certainly Yes.

18
Decision Does the state want to be able to
produce any other evidence statements?

• The investment in building the outcomes system is
  in collecting good assessment data. Rolling up
  or analyzing the data differently adds almost
  nothing to the cost.

19
Good news, bad news

• OSEP evidence statements are relatively
  straightforward
• - OSEP evidence statements are of limited
  usefulness if state wants data for demonstrating
  effectiveness or for program improvement

20
Who is in OSEP Group a?
Group a maintained or reached typical
5 year old level
children who closed the gap
2 year old level
Entry
Exit
21
Who is in OSEP Group b?
5 year old level
Group b made progress but
children who made progress toward closing the
gap
2 year old level
Entry
Exit
22
OSEP Indicators More Powerful Alternative
a. of infants and toddlers who reach or maintain functioning at a level comparable to same-age peers a1. of children who made sufficient progress to maintain functioning at a level comparable to same age peers
a. of infants and toddlers who reach or maintain functioning at a level comparable to same-age peers a2. of children who made sufficient progress to achieve functioning at a level comparable to same age peers
b. of infants and toddlers who improve functioning but are not in a b1. of children who mover nearer to functioning comparable to same age peers but did not achieve it
b. of infants and toddlers who improve functioning but are not in a b2. of children who made progress but not sufficient progress to move nearer to functioning comparable to same-age peers
c. of infants and toddlers children who did not improve functioning c. of infants and toddlers children who did not improve functioning
a2 b1
23
Changing developmental trajectories

• The strongest evidence for the effectiveness of
  EI/ECSE is the percentage of children whose
  developmental trajectories changed after they
  received services
• Gap closers a2 b1

24
Illustration Getting more powerful evidence
(OSEP only)

• cat

OSEP Category 2007 2008 2009
a 45 46 44
b 53 52 55
c 2 2 1
25
Illustration Getting more powerful evidence
(same data, more information)
Category 2007 2008 2009
a1 18 17 15
a2 27 29 29
b1 45 46 50
b2 8 6 5
c 2 2 1
72
74
79
26
Measurement

• Good assessment procedures are the foundation for
  the child outcome system

27
What is assessment?

• Assessment is a generic term that refers to the
  process of gathering information for
  decision-making.
• (McLean, 2004)

28
What is assessment?

• Early childhood assessment is flexible,
  collaborative decision-making process in which
  teams of parents and professionals repeatedly
  revise their judgments and reach consensus about
  the changing developmental, educational, medical,
  and mental health services needs of young
  children and their families.
• Bagnato and Neisworth, 1991
• Quoted in DEC Recommended Practices, 2005

29
DEC Recommended Practices for Assessment

• Involves multiple sources (families, professional
  team members, service providers, caregivers,
  etc.)
• Involves multiple measures (observations,
  criterion-curriculum-based instruments,
  interviews, curriculum-compatible norm-referenced
  scales, informed clinical opinion, work samples)

30
State decisions related to assessment

• What assessment information to include in the
  outcome system?
• Single measure statewide
• List of approved measures
• Whatever measures programs are using

31
State decisions related to assessment

• What assessment information to include in the
  outcome system?
• Collect new assessment information
• Use existing assessment information
• States now gathering information to learn what
  measures are being used in the state

32
State decisions related to building on existing
assessment information

• Options
• Build upon data from the eligibility
  determination process for outcomes
• Build upon data from ongoing progress monitoring
  for outcomes
• Combination

33
Building on Eligibility Measures

• States use standardized, norm-referenced tests as
  part of eligibility
• Easy to anchor to typical development
• Not necessarily reflective of young childrens
  skills and behaviors in regular contexts
• Challenges in aligning domains to functional
  outcomes
• Challenges with cultural relevance for diverse
  populations
• States add other information sources (informed
  clinical opinion) for eligibility determination

34
Building on Ongoing Progress Monitoring

• States use curriculum based/ criterion referenced
  measures for ongoing progress monitoring
• Based on observation in everyday contexts by
  those who know children well
• Many c-b measures are a closer match than
  norm-referenced measures for the 3 functional
  outcomes
• Some have age expectations but for others a
  challenge in establishing age anchors
• Training is needed to insure measures is
  administered reliably, especially if have large
  numbers of assessors with varying levels of
  experience and expertise

35
State decisions related to assessment

• What kind of measures to include?
• Norm-referenced
• Curriculum-based
• Combination

36
State decision related to assessment

• Parent input is critical
• Consistent with best practice
• Impossible to determine how child is doing across
  a variety of settings without information from
  caregivers
• How to incorporate parent input?
• As part of the item scoring within a measure
• As a separate source of information
• Both

37
State decisions related to assessment

• Frequency of data for outcome system?
• Given for OSEP (Near) entry and (near) exit
• Anything in between?
• every 6 months
• annually
• at a fixed point (e.g., every March)

38
Transforming the assessment data
39
Need to transform the data

• Under any assessment option, state needs a way to
  transform the assessment information into the
  indicators/evidence statements.
• No assessment provides information directly on
  the 3 outcomes areas
• No assessment provides information directly on
  the 3 categories (a,b,c) in the OSEP indicators

40
Need to transform the data

• if assessment process involves multiple sources
  of information
• if more than one assessment is being used in the
  state

41
DesiredOutput
Input
?
OSEP Indicators Social a, b, c Knowledge
a, b, c Meet Needs a, b, c
Time 1 Scores Cognitive Communication Social Adapt
ive Motor
Time 2 Scores Cognitive Communication Social Adapt
ive Motor
(for 10,000 children!)
42
Decisions related to transforming the assessment
data

• What is the process by which the data gets
  transformed?
• What is the rubric used to roll up the data?
• Where does the transformation occur?
• State
• Local program

43
What is the rubric?

• Depends on the kind of evidence statements
  desired
• If OSEP only, 2-point scale is sufficient
• If more is desired, need a scale with more points
• ECO has drafted a rubric with a 7-point scale

44
If state only wants 3 OSEP indicators

• Transform data for each child into 2 categories
  at entry
• Q1. Based on assessment and other information
  Are childs skills and behavior related to
  taking action to meet needs at the level expected
  for his or her age?
• 1. yes
• 2. no

45
If state only wants 3 OSEP indicators

• And 2 items, 2 categories each, at exit
• Q2a. Based on assessment and other
  information Are childs skills and behavior
  related to taking action to meeting his or her
  needs at the level expected for his or her age?
• 1. yes
• 2. no
• Q2b. Based on assessment and other information
  Has the child acquired new skills and behaviors
  related to taking action to meet needs since
  entering the program?
• 1. yes
• 2. no

46
If state only wants 3 OSEP indicators

• Using the answers to those 3 questions, the state
  can classify every child as a, b, or c.
• However, transforming the data differently allows
  the state to do a lot more with the data.

47
If states want stronger evidence statements ECO
Child Outcomes Summary Form (under development)

• Rubric that summarizes child status on each
  outcome on a 7 point scale
• NOT an assessment rather a summary of
  information on child functioning
• Has gone through several revisions and is still
  being revised
• In an ideal world, would be extensively
  researched before being released

48
A way to think about how children are doing with
regard to each outcome
49
Child Outcomes Summary Form
50
Definitions for Outcome Ratings
51
(No Transcript)
52
Purpose of the Summary Form

• Form serves as a way to reduce complex
  information to a common scale so the data can be
  aggregated
• The form is not intended to provide information
  to guide interventions
• It does not capture the rich diversity of
  childrens strengths and needs. It does just the
  opposite it is a technique for data reduction.

53
Using assessment data in rating

• ECO cross-walking common assessments to the
  outcomes to provide a consistent source of
  information on which items/score areas apply to
  which outcome.
• ECO is NOT endorsing any assessment instrument

54
Crosswalk High Scope (COR) Infant-Toddler
Outcome 1 Positive Social Relationships II. Social Relations E. Forming an attachment to a primary caregiver F. Relating to unfamiliar adults G. Relating to another child H. Expressing emotion I. Responding to feelings of others J. Playing with others V. Communication and Language T. Participation in give-and-take communication Outcome 2 Knowledge and Skills III. Creative Representation K. Pretending L. Exploring building and art materials M. Responding to and identifying pictures and photographs V. Communication and Language R. Listening and responding S. Communicating interest nonverbally U. Speaking V. Exploring W. Showing interest in stories, rhymes, songs VI. Exploration and Early Logic X. Exploring objects Y. Exploring categories Z. Developing number understanding AA. Exploring space BB. Exploring time Outcome 3 Action to Meet Needs I. Sense of Self A. Expressing initiative B. Distinguishing self from others C. Solving problems D. Developing self-help skills IV. Movement N. Moving parts of body O. Moving whole body P. Moving with objects Q. Moving to music V. Communication and Language S. Communicating interest nonverbally
55
Documentation

• ECO is recommending that states also institute a
  process for documenting the basis for the summary
  rating
• Documentation could be reviewed to insure ratings
  are being derived properly (e.g., identify the
  need for additional training)

56
(No Transcript)
57
How data could be transformed

• Options
• By a team (e.g., at IFSP meetings)
• By an individual (e.g., therapist)
• By computer based on item level data or scores
• Options 1 and 2 will require training
• All 3 require guidance, directions, rules as to
  how the transformation gets performed
• What is typical development for each outcome?

58
Identifying who made progress

• Progress moving up a point on the scale in a
  subsequent rating, e.g., 3 to 4
• Progress staying at the same rating but having a
  yes for the progress question (right column)

59
Use of the Summary Form

• The Summary Form is free.
• States can modify it, however, ECO will be
  researching and supporting (developing guidance
  materials, training, etc.) one version. If the
  state makes significant modifications, neither
  the ECO research nor materials will apply.

60
ECO Next Steps for Child Outcomes Work

• Developing additional crosswalks mapping
  assessments to the three outcomes
• Working with publishers, test developers and
  state staff to develop guidance for how to
  transform assessment results (scoring
  algorithm)

61
ECO Next Steps for Child Outcomes Work

• Researching the Summary Form (reliability, extent
  of training required, etc.)
• Possibly/probably revising the tool based on
  pilot work and additional research.

62
ECO Next Steps for Child Outcomes Work

• Additional information including new versions of
  the Summary Form will be posted on the web site
  as they become available.
• www.the-eco-center.org