Weighing Outcome Research

1
Weighing Outcome Research

• Third Summer Leadership Institute
• Amherst, MA
• Center for School Counseling Outcome Research

2
Evidence for Important Causal Relationships

• Not all research is equal
• Research quality is related to the confidence
  with which causal inferences can be made
• Research quality is related to
• Internal validity
• External validity

3
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 1. Measurement
• Principle Important academic, career and/or
  personal/social outcomes are measured using
  reliable and valid instruments
• Strong Evidence
• Outcomes measures have established high
  reliability and validity characteristics
• Outcome measures are established to be
  appropriate for the population under study
• Promising Evidence
• Outcome measures have been used in previous
  studies,
• Reliability characteristics are evaluated in the
  study and show adequate reliability
• Logical argument supports the appropriateness of
  the measures for the population under study

4
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 2. Comparison Groups
• Principle Comparison groups with adequate
  controls are included so that resulting group
  differences can be attributed to the intervention
• Strong Evidence
• Active Comparison Groups (alternative treatment)
  with adequate controls (attention, placebo) are
  included in an outcome study
• Initial Group Equivalence is assured through
  random assignment
• Group Equivalence in mortality/attrition is
  established
• Promising Evidence
• Groups equated through matching or statistical
  procedures (e.g. ANCOVA) or strong pre-post-test
  designs are used with adequate controls

5
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 3. Statistical Analyses of Outcome Variables
• Principle Statistical analysis documents low
  probability of Type 1 error and potency of
  intervention
• Strong Evidence
• Statistically significant finding using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Moderate Effect size for critical
  outcome variable
• Promising Evidence
• Statistically significant finding using
  appropriate test
• Control for Experimentwise error rate
• Adequate N
• At least a Small Effect size for critical outcome
  variable

6
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 4. Implementation Fidelity
• Principle Intervention can be delivered with
  fidelity across contexts and is not contaminated
  with implementer
• Strong Evidence
• Intervention is extensively documented (manual or
  protocol) so that it can be reliably replicated
• Intervention is delivered by multiple people with
  adequate training and checks for adherence to
  protocol
• Promising Evidence
• Intervention is standardized and can be delivered
  across contexts
• Intervention is delivered by multiple people with
  adequate training

7
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 5. Replication
• Principle The same intervention independently
  implemented with an equivalent population results
  in equivalent outcomes
• Strong Evidence
• Independent evaluators find equivalent outcomes
  with a similar population
• Promising Evidence
• Same evaluator finds equivalent outcomes with
  same population

8
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 6. Ecological Validity
• Principle The intervention can be implemented
  effectively in a public school with consistent
  effects across all student subgroups or with
  known differences between student subgroups.
  Limitations of the generalizability of results
  are clearly explicated
• Strong Evidence
• Study conducted in a diverse public school
• Outcomes are assesses across different subgroups
  of students or clearly specified as valid for a
  specific subgroup
• Promising Evidence
• Study conducted in a private, laboratory, or
  charter school or in a public school with limited
  diversity

9
National Panel for Evidence-Based School
Counseling Practice Criteria for Strong
Evidence

• 7. Persistence of Effect
• Principle The intervention results in a lasting
  effect on an important outcome measure.
• Strong Evidence Treatment-Comparison Group
  Differences are demonstrated to persist for a
  practically significant time period.
• Promising Evidence Treatment-Comparison Group
  Differences are demonstrated to persist beyond
  the immediate implementation.

10
Activity

• In groups, review the research report and
  evaluate it in terms of National Panel Criteria

11
Methods for Weighing Evidence

• Research Briefs
• Analytic Reviews
• Meta-Analytic Reviews
• Expert Panels, Centers and Institutes
• Research Literature Searches School-Based Teams

12
Research Briefs

• Center for School Counseling Outcome Research

13
Analytic Reviews

• Whiston and Sexton, 1994
• McGannon, Carey and Dimmitt, 2005

14
Meta Analytic Reviews

• Gene Glass
• Select studies
• Compute effect size
• Group studies
• Interventions
• Outcomes
• Quality

15
Meta Analytic Reviews

• Effect Size (Welkowitz, Ewen Cohen)
• .20Small
• .50Medium
• .80Large

16
Meta Analytic Reviews Group Exercise

• Review Evans Burck (1992)
• What can we conclude about the effects of career
  education on academic achievement?
• Are the effect sizes surprising?
• How would you explain this article to a school
  committee?

17
Meta-Analysis of School Counseling Interventions

• Whiston et al (2005)

18
Information on Studies

• 117 studies
• 69.2 published
• 30.8 thesis/dissertations
• Participants
• 16,296 total
• Age
• Elementary 59 studies (50.4)
• Middle/Junior High 21 studies (17.9)
• High School 29 studies (24.8)
• Mixture 7 studies (6)
• Unreported 1 study (.9)

19
Interventions

• 153 school counseling interventions
• Types of interventions
• Guidance Curriculum 44 studies/57 interventions
• Individual Planning 9 studies/10 interventions
• Responsive Services 58 studies/74 interventions
• Program-Wide Services 8 studies/9 interventions
• Other 2 studies/3 interventions
• Length of interventions
• Number of sessions M12.44 , SD 15.53, Range
  1-120
• Number of hours M8.41 , SD 6.87, Range .42-40
• Duration M71.76 , SD 87.52, Range 1-720
• Providers of interventions
• Experienced counselor 37.9 (58 interventions)
• Counselor in training 9.8 (15 interventions)
• Teacher 8.5 (13 interventions)
• Student peer 3.3 (5 interventions)
• Student independent 5.9 (9 interventions)
• Other 5.9 (9 interventions)
• Combination 13.7 (21 interventions)

20
Outcomes

• 785 outcomes
• Cognitive 12.2 (96 outcomes)
• Behavioral 34.8 (273 outcomes)
• Affective 39.0 (306 outcomes)
• Effective Role Functioning 9.4 (74 outcomes)
• Satisfaction .1 (1 outcome)
• Global Evaluation of School Counseling Program
  .4 (3 outcomes)

21
Effect Sizes

• Unweighted .46
• Weighted .29 (based on 153 interventions)
• CI .26 to .32
• Q 616.124, p lt .001

22
Grade Level and Type of Intervention

• Grade level effect sizes
• Elementary ES .25
• Middle School ES .39
• High School ES .34
• Type of intervention
• Guidance Curriculum ES .33 (44 studies/57
  interventions)
• Individual Planning ES .27 (9 studies/10
  interventions)
• Responsive Services ES .35 (58 studies/74
  interventions)
• Program Wide Evaluation .19 (8 studies/9
  interventions)

23
Program Component by Grade Level

• Guidance Curriculum
• Elementary ES .29
• Middle School ES .41
• High School ES .39
• Individual Planning
• Elementary ES -.137 (1 intervention)
• Middle School ES 1.01 (3 interventions)
• High School ES .22 (6 interventions
• Responsive Services
• Elementary ES .40 (45 interventions)
• Middle School ES .22 (10 interventions)
• High School ES .34 (20 interventions)

24
Cognitive Outcomes

• Overall ES .19 (94 outcomes)
• GPA ES .19 (49 outcomes)
• Academic achievement tests ES .16 (16 outcomes)
• Career knowledge ES .61 (12 outcomes from 3
  studies)
• Other ES .73 (19 outcomes from 11 studies)

25
Behavioral Outcomes

• Overall ES .39 (274 outcomes)
• Behavior rating scales ES .24 (123 outcomes)
• No study examined change in enrollment (e.g.,
  taking more difficult classes) as an outcome
  measure
• Attendance ES .30 (19 outcomes)
• Interpersonal or social skills ES .34 (55
  outcomes)
• Aggressive behavior/fights ES .26 (12 outcomes)
• Problem-solving skills ES .96 (26 outcomes)
• Peer counseling skills ES 1.12 (15 outcomes)
• Number of disciplinary referrals ES .82 (9
  outcomes)
• Other ES .25 (14 outcomes)

26
Affective Outcomes

• Overall ES .22 (306 outcomes)
• Self-esteem ES .18 (172 outcomes22 of total
  outcomes)
• Personal/social adjustment ES .24 (27 outcomes)
• Anxiety ES .39 (35 outcomes)
• Depression ES .35 (6 outcomes)
• Other ES .24 (66 outcomes)

27
Effective Role Functioning

• Overall ES .12 (74 outcomes)
• Career maturity/decidedness ES .20 (30
  outcomes)
• Academic functioning ES -.17 (9 outcomes)
• Other ES .12 (35 outcomes)

28
Conclusion

• School counseling interventions moderately
  effective
• Individual planning at middle school level has a
  large effect size (small number of studies)
• Guidance curriculum more effective at middle and
  high school level whereas, responsive services
  have a larger effect size at elementary level
• Large effects
• Career knowledge
• Problem solving
• Disciplinary referrals
• Contributes to academic achievement
• Moderately effective in reducing anxiety and
  depression, increasing interpersonal social
  skills, and positively influencing attendance

29
Expert Panels Centers and Institutes

• National Panel for Evidence-Based School
  Counseling Practice

30
Second Step EBP Protocol Summary

• Protocol Domain
• 1. Measurement
• 2. Comparison Groups
• 3. Statistical Analysis of Outcome Variables

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence

31
Second Step EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• Second Step
• Strong Evidence
• Strong Evidence
• Strong Evidence
• Strong Evidence

32
SSS EBP Protocol Summary

• Protocol Domain
• Measurement
• Comparison Groups
• Statistical Analysis of Outcome Variables

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence

33
SSS EBP Protocol Summary

• Protocol Domain
• 4. Implementation Fidelity
• 5. Replication
• 6. Ecological Validity
• 7. Persistence of Effect

• SSS
• Strong Evidence
• Promising Evidence
• Strong Evidence
• Weak Evidence

34
Other Resources

• What Works Clearing House
• CDC/DASH Programs That Work
• SDFS Exemplary and Promising Programs
• SAMHSA National Registry of Evidence-Based
  Programs and Practices
• Collaborative for Academic Social and Emotional
  Learning

35
School-Based Teams

• Best done in the context of data-based decision
  making
• Team includes key perspectives and resources
• Team has identified problem data, (academic,
  career, psychosocial) and vision data (what
  problem data should look like in the future
• Now the focus is on what evidence-based
  interventions are most likely to result in a
  change in the problem behavior

36
School-based Teams

• What research-based interventions are out there?
• Does the evidence support a link between the
  intervention and the desired outcomes?
• Does the evidence support intervention
  effectiveness with the desired target students?
• Can the intervention be implemented with fidelity
  in your setting?
• Is the intervention feasible in your setting?
• Are evaluation materials included in the
  intervention (if not, how will it be evaluated)?

37
School-Based Teams

• Online tutorial

38
School-Based Teams

• Review Sample Matrix of Interventions
• Review Feasibility Checklist
• Assignment

39
Discussion
40
National Center for School Counseling Outcome
Research

• Thank You

www.cscor.org