Early Childhood Outcomes Revisited

1
Early Childhood Outcomes Revisited Improving
Data Quality
Early Childhood Outcomes Revisited Improving
Data Quality
Margy Hornback, Early Childhood Leadership
Project Carolyn Nelson, Kansas Department of
Health and Environment Dave Lindeman, Misty
Goosen, Phoebe Rinkel, Peggy Miksch Kansas
Inservice Training System (KITS)
2
PartnerAcknowledgements

• Early Childhood Outcomes Center
• Training Materials
• Child Outcomes Summary Form (COSF)
• Juniper Gardens Childrens Center Project
• Research Partners
• Charlie Greenwood, Ph.D
• Dale Walker, Ph.D

3
Kansas Pilot Sites

• Central Kansas Special Education Cooperative
• Harvey County Special Education Cooperative
• ANW Special Education Cooperative
• Northwest Kansas Educational Service Center
• Manhattan-Ogden USD 383 Special Services
• Olathe USD 233 Special Services
• Southwest Kansas Area Cooperative
• Geary County Infant-Toddler Services
• Arrowhead West, Inc.
• Salina Regional Health Center Infant-Child
  Development
• Johnson County Infant-Toddler Services
• Southeast Kansas Birth to Three Programs

4
Agenda
5
Accountability Review
Accountability Review
6
Critical Events in Accountability

• The Government Performance and Results Act (GPRA)
  required all Federal agencies to develop a set of
  indicators to judge effectiveness of programs
• Office of Management and Budget (OMB) developed
  the Program Assessment Rating Tool (PART) to
  provide a consistent approach to evaluate federal
  programs during budget formulation

7
PART Findings

• Part C While the program has met its goal
  relating to the number of children served, it has
  not collected information on how well the program
  is doing to improve the educational and
  developmental outcomes of infants and toddlers
  served.
• Part B The Department has no information on
  preschool children with disabilities served by
  this program.

8
PART Aftermath

• Debate about whether child outcomes should be
  measured at all
• Much discussion about the problems in trying to
  measure child outcomes
• Moved past discussions
• Must meet requirements of measuring child outcomes

9
SPP/APR

• IDEA 2004 required the development of a State
  Performance Plan (SPP)
• SPP describes by indicator how each state will
  improve its implementation of Part B and Part C
  programs
• SPP sets six year targets for performance and
  compliance indicators
• Annual Performance Report (APR) describes
  progress in meeting SPP indicator targets

10
Department of Education Actions

• Department of Education approved states SPPs in
  2006
• States submitted first APRs in February 2007
• Secretary of Education makes annual determination
  as to whether each state is meeting the
  requirements of the statute
• States received letters with first determinations
  in 2007

11
Level of Determinations

• Meets the requirements and purposes of the IDEA
• Needs assistance in implementing the requirements
  of the IDEA
• Needs intervention in implementing the
  requirements of the IDEA
• Needs substantial intervention in implementing
  the requirements of the IDEA

12
Data and Criteria for Determinations

• State demonstrated compliance on indicators or
  corrected noncompliance in a timely manner
• State made progress in ensuring compliance over
  prior performance in that area
• State provided valid and reliable data for all
  indicators
• State resolved problems it had with other IDEA
  compliance issues (i.e. monitoring, audit)

13
Enforcement Actions

• Specific technical assistance and enforcement
  actions are aligned with each of the
  determinations
• Actions are consistent with level of concern
  signaled by the determination
• Enforcement actions will be applied to states on
  the basis of the APR submitted in February 2008

14
Early Childhood OutcomesA Performance Indicator
15
Document Child Progress on Functional Outcomes

• Positive social-emotional skills
• Acquisition and use of knowledge and skills
• Use of appropriate behaviors to meet their needs

16
Measurement of Progress

• a. Children who did not improve functioning
• b. Children who improved functioning but not
  sufficient to move nearer to functioning
  comparable to same-aged peers
• c. Children who improved functioning to a level
  nearer to same-aged peers but did not reach it
• d. Children who improved functioning to reach a
  level comparable to same-aged peers
• e. Children who maintained functioning at a
  level comparable to same-aged peers

17
COSF

• Kansas was looking for an approach that..
• Could be used for ALL children in Part C and Part
  B programs
• Used existing assessment data
• Could combine data when different assessment
  tools were used
• Stressed assessment as an ongoing process
• Allowed providers to collect data
• Used input from families for skills/behaviors
  across settings
• Focused on functional skills/behaviors rather
  than domains
• Combined information from multiple sources and/or
  multiple measures
• Kansas adopted the Child Outcomes Summary Form
  developed by the ECO Center

18
OWS

• Kansas developed a web based application for
  collecting child outcome data
• The Outcomes Web System (OWS) refers to all
  fields in the web application (i.e. history,
  name, DOB, COSF)
• Outcomes Web System users Guide Including
  Instructions and Data Dictionary provides step by
  step instructions for data entry

19
Biggest Challenges for States

• Developing State data system to collect data
• Getting it on the radar as a high priority
• Training providers to collect and enter data
• Allocating resources at the state and local level
  (personnel, money, and time)
• Building local capacity for ongoing training and
  quality assurance

20
Putting Data to Use and Importance of Data Quality
21
(No Transcript)
22
Begin Sharing the Data

• How are we going to talk about the data at state
  and local levels with
• Agency/School Boards?
• Local Interagency Coordinating Councils?
• State Legislators?
• Families?
• Other Key Stakeholders?

23
Think Ahead

• Write out specific messages we want to give
• Develop a 1-2 page fact sheet that summarizes the
  message
• Use public dissemination opportunities to get out
  the message

24
Provide the Data in Simple Terms

• Nine of 10 children showed improvement on
  outcomes from entry to exit in the program (b c
  d e)
• Nearly three-fourths of the children made more
  progress than expected or maintained functioning
  like same age peers (c d e)
• Two in 10 children continued to have age level
  skills on the outcomes from entry to exit (e)

25
Provide the Data in Simple Terms

• Six of 10 children are catching up to their same
  age peers from entry to exit (d)
• Eight of 10 children showed improvements in
  acquiring and using knowledge and skills from
  entry to exit (b c d e)
• Six of 10 children improved to the level of same
  age peers and caught up in their development (d)

26
Types of Summary Statements

• Program effectiveness message (c d)
• Prevention effectiveness message (e)
• Prevention plus program effectiveness message (c
  d e)
• Ready-at-exit message (d e)
• Progress message ( b c d e)

27
Interpret Numbers

• Data show that children are making progress from
  entry to exit in the program
• Many children are catching up or getting closer
  to same age peers
• Point out how programs are contributing to school
  readiness
• Link message to broader EC issues (i.e. cost
  effectiveness of high quality EC programs)

28
Remind Public First Wave of Outcome Data

• This is first year for reporting progress data
• These are preliminary data
• These data are on relatively few children
• These data are collected since April 2006
• The quality of the data will improve over time

29
End by returning to the big picture

• The goal of these programs is for children to be
  active and successful participants now and in the
  future

30
Importance of Data Quality

• Only valid data can be used for program
  improvement or anything else
• Are all children included?
• Are instructions followed when determining
  ratings?
• Is an approved curriculum assessment used as the
  cornerstone for all ratings?
• Is data timely?
• Is rating defensible?
• Continue to improve data

31
Improving Data Quality is an Ongoing Process
32
Exploration of KITS Website andOWS Data Entry
Demonstration
33
Top Ten Challenges Findings of Data Quality
Activities
34
10 Meeting Timeline Challenges
35
Important Timelines for Submitting ECO Data

• Part C exit data must be entered into the OWS no
  later than 90 days after the childs third
  birthday
• July 30, 2008 last date for submitting child
  entry and/or exit data for children entering
  and/or permanently exiting the program between
  July 1, 2007 through June 30, 2008

36
Timelines (Cont.)

• Data over a year old cannot be entered into the
  OWS
• No entry data can be entered for a child who is 6
  or older at the time of data entry
• No exit data can be entered for a child who is 7
  or older at the time of data entry
• Data cannot be entered for a child until after
  the services begin (initial date of services) or
  after the services end (last date of services)
  per the IFSP/IEP

37
9 Challenge of Including All Children
38
COSF Rating Required for ALL Children

• FAQs at http//www.kskits.org answer questions
• Q. Will our organization still be required to
  submit a Child Outcomes Summary Form if the child
  will only be receiving special education services
  in a community setting?
• A. The requirements of collecting early childhood
  outcome data apply to all children who receive
  Part C or Part B 619 services beginning on April
  1, 2006. This requirement is not dependent on a
  child's placement, amount of service, disability
  category, or citizenship status.

39
COSF Rating Required for ALL Children

• Q. Is it necessary to use one of the eight
  identified curriculum based assessment tools when
  a childs only area of need is articulation?
• A. All children entering a program (Part C or
  Part B) after April 1, 2006 must have child
  outcome entry summary rating data entered into
  the OWS if they can be in the program for at
  least 6 months. However, the requirement for the
  curriculum based assessment is waived for
  children with only one area of concern (i.e.
  articulation) if the team can confidently rate
  the child a 6 or 7 in all 3 outcome areas on the
  basis of record review, interview, observation
  and screening.

40
Develop a Process

• Who will be responsible to determine that all
  children are entered into the OWS by the
  deadline?
• How will they do it?

41
8 Challenge of Involving Team Members in Rating
Process
42
Involve Parents in Planning Whenever
PossibleFamilies Are Important
43
Importance of Team Members

• Between them team members must
• Know the childs functioning across settings and
  situations
• Understand age-expected child development
• Understand the content of the 3 child outcomes
• Know how to use the rating scale

44
Meet as a Team

• Reach consensus rating
• Parent or guardian involved in team process in
  person or by supplying information
• Meet when team is already together or via media
• Immediately after IFSP/IEP meeting or transition
  meeting
• At team meetings
• Via media (i.e. ITV, conference call, instant
  messenger, e-mail)

45
Parent Information is Needed

• Family members see the child in situations that
  professionals do not
• Need to determine what child does in a variety of
  settings (i.e. home, store, child care setting)
• Develop a method for getting information needed
  from family

46
How to Reach Team Consensus

• Structure conversation to decrease the likelihood
  of impasse
• Focus most of the discussion on the childs
  skills dont go to the rating too quickly
• Discuss rationale for different ratings focus on
  concrete descriptions and how these support a
  rating
• Include more discussion on what skills and
  behaviors you would see in a typically developing
  child this age
• Develop a procedure for dealing with an impasse
• Supervisor decides
• Majority rules
• Use decision making tools

47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
7 Challenge Documenting Evidence Sources
51
Use Evidence from Approved CBA for Each Outcome
at Entry and Exit

• Assessment and Programming Evaluation System
  (AEPS)
• Carolina Curriculum for Infants and Toddlers or
  Preschoolers with Special Needs
• Child Observation Record (High Scope)
• Creative Curriculum Developmental Continuum
  Assessment
• Hawaii Early Learning Profile (HELP)
• Individual Growth and Development Indicators
  (IGDIs)
• Transdisciplinary Play-Based Assessment (TPBA)
• Work Sampling System

52
Additional Evidence

• Use Table 2 for documenting source information
• Document the who or what of record review,
  interview, or observation in the summary of
  relevant results (i.e. SLP observed Jamie in his
  home, review of Dr. Smiths records)
• Use last date of assessment rather than multiple
  dates

53
Additional Evidence (Cont.)

• Include strengths in summary of relevant results
• Only include information actually used in
  determining rating
• Document what evidence led to the rating
• Age expected functioning
• Performance across all or almost all settings and
  situations
• Immediate foundational skills
• Skills and behaviors that will lead to immediate
  functional skills

54
6 Challenge Documenting Functional Behaviors in
Summary of Relevant Results
55
Functional Behaviors Are Not

• A single behavior
• The sum of a series of discrete behaviors or
  splinter skills such as..
• Knows 10 words
• Pincer grasp (picks up a raisin)
• Smiles at mom
• Goes up and down stairs with one foot on each
  stair
• Stacks 3 blocks

56
Thinking Functionally

• Isolated skill
• Knows how to imitate a gesture when prompted by
  others
• Uses finger in pointing motion
• Uses 2-word utterances

• Functional skill
• Watches what a peer says or does and incorporates
  it into his/her own play
• Points to indicate needs or wants
• Engages in back and forth verbal exchanges with
  caregivers using 2-word utterances

57
Thinking Functionally

• Emphasize how the child is able to carry out
  meaningful behaviors in a meaningful context
• If you know that a child can point, do you know
  that the child can communicate wants and needs?
• If you know that a child cant point, do you know
  that the child cant communicate wants and needs?
• How does knowing about pointing help you
  understand how the child takes action to meet
  needs?

58
5 Challenge Completing Exit COSF Ratings
59
Two Steps to Exit COSF Ratings

• Ratings from 1-7 for each of the outcomes
• This is where child is functioning at the time of
  exit
• Progress questions 1a, 2b, 2c
• This is a comparison of entry and exit data to
  determine if there was progress (i.e. any new
  skills or behaviors)
• There are a number of impossible COSF responses
  to progress questions

60
(No Transcript)
61
4 Challenge Matching Paper COSF to Information
Needed in OWS
62
Customize COSF Form

• Determine as a team what additional information
  is needed
• Add information to the form (i.e. circumstance,
  use last summary, parental permission)
• Save data entry time

63
3 Challenge of Timely Documentation of Moves in
and out of IT Networks and Districts
64
Moves Between Organizations

• Enter each move into the OWS no matter how long
  the child was in organization
• Enter the appropriate circumstancechild
  exiting an organization not a program
• Enter last date of EI or SPED services provided
  by the exiting organization
• Enter data immediately after last date of service

65
Moves

• Receiving organization will enter the appropriate
  circumstanceChild entering an organization not
  a program
• Enter first date of EI or SPED services provided
  by the entering organization
• If you are a Part B Multi Org User select
  district ID number for the district the child is
  entering (moved into)

66
Develop Process for Movers

• Who will notify child entry person that child is
  moving or entering organization?
• Who will notify new organization of need to
  complete entry COSF?
• Child moves before 30 days and COSF is not
  completed
• Child moves before first date of service and COSF
  is not entered/completed

67
2 Challenge of Developing Joint Part C Exit and
Part B Entry Ratings
68
Working Together

• When developing Memorandums of Understanding
  (MOUs) that outline responsibilities in
  transition include COSF ratings
• Agreements formalize understandings and
  discussions between agencies
• Involve State technical assistance resources to
  help develop MOUs

69
1 Challenge of Real Life Situations
70
Life Happens

• A child enters our organization and we develop
  and implement an IEP/IFSP but
• the child has very few service dates before the
  child disappears and is permanently exited from
  the program and then the child enters another
  organization in the state
• the child permanently exits from the program
  before 6 months
• A child moves in from out of state with a current
  IEP/IFSP
• A parent tells us that they are moving out of
  state in just a few months so we dont complete
  an entry COSF rating and then the family doesnt
  move

71
Life Happens

• All ECSE staff leave the organization at the end
  of the school year and no one knows how to make
  COSF ratings
• A child transitions to kindergarten from an ECSE
  program and an exit COSF rating is completed but
  after kindergarten begins the child is placed
  back in ECSE part time
• A child starts kindergarten and then it is
  decided that the child is eligible for special
  education services and is placed full time in
  ECSE services
• There is no Part C exit data for a child in the
  OWS history section but the child was
  transitioned from a Part C network

72

• Choice of Activities
• Practice Completing COSF Rating with Team
• Q A Part C
• Q A Part B
• Critique COSF Ratings