We Changed the COSF

1
We Changed the COSF

• The Flip
• The Skills
• The Decision-Tree Questions

2
The Flip

• We flipped source info and overall skill rating.
• I.e., the source info (assessment, parent,
  teacher observations, etc.) is at the top of the
  page skill rating at the bottom.

3
Reason for The Flip

• We want to make sure that providers base their
  ratings on the data theyve collected.
• We want providers to list their data, i.e., the
  relevant source information, and then arrive at a
  rating.
• We wanted to build the decision-making process
  that providers should follow onto the form
  itself.

4
The Skills

• We added a section that asks the provider to list
  the highest-level of Age-Appropriate (AA),
  Immediate Foundational (IF), or Foundational (F)
  skills the child demonstrates.

5
Reason for The Skills

• We want to make clear that the overall rating
  should relate to whether the child is
  demonstrating AA, IF, or F skills.
• The overall rating should not be solely tied
  (necessarily) to percentile scores and
  age-equivalent scores (for example) that might be
  listed under Summary of Test Results.
• Once again, we wanted to build the
  decision-making process that providers should
  follow onto the form itself.

6
The Decision-Tree Questions

• We added two questions that reflect the ECO
  Center Decision Tree to the COSF.
• 3. Think of the positive social-emotional
  age-appropriate skills the child demonstrated as
  indicated in the tables above. Did the child
  demonstrate these age-appropriate skills across
  all or almost all everyday situations (multiple
  settings)? (Circle one response)
• a Yes ? 7 or 6 b Somewhat ? 5 c Rarely ? 4 d
  Child demonstrated no age-appropriate skills (go
  to Q4)

7
The Decision-Tree Questions

• 4. Think of the positive social-emotional
  immediate foundational skills the child
  demonstrated as indicated in the tables above.
  Did the child demonstrate these immediate
  foundational skills across all or almost all
  everyday situations (multiple settings)? (Circle
  one response If you answered a, b, or c for
  question 3, no need to answer)
• a Yes ? 3 b Somewhat ? 2 c No ? 1 d Child
  demonstrated no immediate foundational skills ? 1
  

8
Reason for the Decision-Tree Questions

• We want to make clear that the decision tree must
  be used to arrive at the rating, i.e., providers
  must consider the degree of AA, IF, F skills, and
  the extent to which they are demonstrating these
  skills across multiple settings.
• By building into the form, were not relying on
  providers to remember to use the decision tree or
  to locate a copy of the tree.

9
Additional reasons for COSF Changes

• We wanted to collect the skill info and the
  decision tree question info so that we could use
  this info to validate the ratings.

10
Additional Reasons continued. . .

• If a provider says (on the decision tree
  question) that a child is demonstrating
  age-appropriate skills some of the time, but
  gives the child a rating of 2 then we know
  theres a problem somewhere!
• If a provider says (on the decision tree
  question) that a child is demonstrating
  app-appropriate skills some of the time, and
  gives the child a rating of 5 (so far so good),
  BUT does not list any age-appropriate skills in
  the skills boxes we know theres a problem
  somewhere!

11
To Sum Up Re the Form Changes

• We wanted to incorporate the decision-making
  process into the form.
• We wanted to collect info on the form that would
  help us validate the ratings.

12
The COSF Data Submission Process

• Providers complete on paper and mail hard-copy to
  the DDD.
• September 2007 a Word document will be provided
  that will allow providers to type in information.
• Future creating an online form as part of the
  DDD Early Childhood database system.

13
Are We Doing a Good Job?

• Are providers following the correct process in
  gathering data on child outcomes?
• Are providers correctly completing the COSF?
• Are providers making valid ratings?

14
Gathering Data

• - Are providers following the correct process in
  gathering data on child outcomes?
• Providers are encouraged to collect relevant data
  from parents. And they are.
• Providers are told to use one of several core
  assessments. And they are.
• Providers are told to use multiple sources of
  data. And they are.

15
Gathering Data

• Percent Collecting Relevant Info from Parents

16
Gathering Data

• Percent Using a State-Approved Core Assessment

17
Gathering Data

• Percent Using Multiple Sources of Information

18
Completing the Form

• -- Are providers correctly completing the COSF?
• Providers are to give summary of actual test
  results. And they are.
• Providers are to list AA, IF, F skills. And they
  are.
• Providers are supposed to answer every question.
  And they are. For the most part!
• Most of the issues we have with incomplete forms
  is readily fixed by offering training on the COSF
  and by providing sample completed COSFs.

19
Percent who did not answer key questions

• Decision tree questions about 3.5 for each
  skill area for each of Part C and Part B
• Overall ratings About 3 of Social Skills and
  Acquiring Knowledge and Skills about 5.5 for
  Taking Appropriate Action
• Did child make progress questions (the question
  asked on exit form and used to determine
  improvement category) about 40 blank for each
  skill area!

20
Tackling Not Answering Questions

• Leaving questions blank was addressed in
  training
• Some providers thought the decision tree
  questions and improvement questions were
  optional.
• Some providers thought that overall ratings were
  necessary for only the outcome area the in which
  the child had a problem.

21
Making Valid Ratings

• -- Are providers making valid ratings?
• The decision-tree questions are supposed to guide
  the decision-making process.
• The listed skills are supposed to guide the
  decision-making process.
• The summary of actual test scores is supposed to
  guide the decision-making process.

22
Making Valid Ratings

• The percent whose decision-tree questions
  matched the overall rating about 67 on each
  skill area for each of Part B and Part C
• More training is needed!

23
Making Valid Ratings

• Content Analysis We have produced a document of
  the listed skills and their corresponding rating
  by age group so that a content analysis can be
  done to determine if providers are making an
  appropriate overall rating given the listed
  skills a child is demonstrating.
• We will also produce a document from this content
  analysis that will illustrate behaviors
  representative of different scale points at
  different age groups.

24
Making Valid Ratings

• Score Analysis We will be correlating the
  actual test scores on the core assessments to the
  overall ratings to determine if this tells us
  anything valuable.
• Even though providers are supposed to use
  multiple sources of information and not rely
  solely on the test score, we would expect at
  least some relationship between test scores and
  overall ratings.
• We want to determine what the relationship is and
  if this has implications for technical
  assistance.

25
Bottom Line

• We want to make sure that providers are making
  valid ratings AND
• We want to make sure that the ratings are
  consistent across regions.