What Works Regarding Social Skills Interventions Using Single Subject Design

1
What Works Regarding Social Skills Interventions
Using Single Subject Design

• Jeffrey Chenier, M.A., Aaron J. Fischer,
  Katherine Hunter, Emily Patty, Lisa Libster,
  M.A., Kristen OLeary, Haley York, Natalie
  Robichaux
• and
• Frank Gresham, Ph.D.

2
Introduction

• Scientifically Based Research
• Section 9101(37) NCLB
• Research that involves the application of
  rigorous, systematic, and objective procedures to
  obtain reliable and valid knowledge relevant to
  education activities and programs
• (No Child Left Behind Act, 2001)

3
Introduction

• What Works Clearinghouse (2006) Group Design
• Meet Evidence Standards
• well designed and implemented randomized
  controlled trials
• Meet Evidence Standards with reservations
• quasi-experiments with equating and no severe
  design or implementation problems or randomized
  clinical trials with severe design or
  implementation problems

4
Introduction

• What Works Clearinghouse (2010) Single Case (SC)
• Meet Evidence Standards
• IV must be systematically manipulated, with the
  researcher determining when and how the IV
  conditions change
• Each outcome variable must be measured
  systematically over time by more than one
  assessor, and the study needs to have IOA
  calculated 20 of the time in each condition, and
  IOA percentage must meet minimum thresholds
• 0.80 IOA or 0.60 Cohens Kappa
• Study must include at least three attempts to
  demonstrate an intervention effect at three
  different points in time or with three different
  phase repetitions
• Phase must have a minimum of three data points
• Effect size estimation follows if a study has
  either Strong Evidence or Moderate Evidence

5
Introduction

• Meta-Analysis
• Strube Hartmann (1982)
• Objective method for summarizing a body of
  empirical findings
• Emphasizes the direction and magnitude of effects
  across studies for a particular intervention

6
What Works Clearinghouse (2010)

• No agreed upon method or gold standard to
  calculate effect sizes from single-case design
  research
• Problems
• How to quantify the effect?
• How accurate is the effect?
• How comparable are the effects across other SC
  designs?
• How comparable are the effects compared to group
  design effect sizes?

7
Current Effect Size Estimators (WWC, 2010)

• Nonparametric Methods
• Percentage of Nonoverlapping Data (PND),
  Percentage of All Nonoverlapping Data (PAND),
  Percent Exceeding the Median (PEM)
• Distributional properties of these measures are
  unknown, so standard errors and statistical tests
  are not formally justified.
• Additionally, trend is not addressed
• Because of the lack of statistical justification,
  only use if an approximate size of the effect is
  desired.
• Wolery et al. (2010) compared four overlapping
  methods to visual inspection of effect and each
  method had its own host of issues, so much that
  they called for their abandonment
• Visual analysis only agreed 121/160 on whether
  the treatment was effective or not

8
Current Effect Size Estimators (WWC, 2010)

• Parametric Methods
• Regression Estimates
• Advantages
• Familiarity
• Ability to model trends
• Ability to attain an Effect Size from a single
  case
• Disadvantages
• Inability to deal with complex structures present
  in single case design

9
Current Effect Size Estimators (WWC, 2010)

• Parametric Methods
• Multilevel Modeling
• Advantages
• Ability to account for complexity of design
• Disadvantages
• Unfamiliarity
• Technically challenging and time consuming
• Different metric from group design Effect Sizes,
  therefore the estimate is not comparable

10
Current Effect Size Estimators (WWC, 2010)

• Quantitative Methods
• Differing methods to calculate a Standardized
  Mean Difference statistic (current study)
• Advantages
• Encourages inclusion of SC designs in evaluating
  effects of interventions
• Potentially gives another method in which to rank
  order interventions
• Disadvantages
• Not completely comparable to group design
  research
• Pooled within-group variance not comparable to
  pooled within phase variance
• Small n leads to imprecise estimates
• Trend is not assessed

11
Summary of Effect Size Estimators for SC Design
(WWC, 2010)

• Simply put, science is not there yet
• Nonparametric estimators should be reported with
  a parametric estimator (regression)
• Multilevel methods are not ready
• Quantitative methods are not as statistically
  sound as they should be, but the base from which
  to build is present

12
Social Skills

• Learned behaviors that enable positive
  interactions and allow for escape/avoidance of
  negative interactions
• Academic Enablers (DiPerna Elliott, 2002)
• Better predictor of academic achievement in 8th
  grade than 3rd grade academic achievement
  (Caprara et. al, 2000)
• Myriad of problems co-occurring with social
  skills deficits
• Both externalizing and internalizing

13
Introduction

• Does social skills training work?
• Gresham, Cook, Crews, and Kern, 2004

Meta-Analysis n studies ES g ES r BESD Control BESD Treatment
Ang Hughes (2001) 38 .62 .30 35 65
Beelman et al. (1994) 49 .47 .23 38 62
Losel Beelmn (2003) 84 .38 .19 40 60
Schneider (1992) 79 .89 .40 30 70
Schneider Byrne (1985) 51 .65 .31 34 66
Quinn et al. (1999) 35 .20 .10 45 55
Means Means M .60 M .29 M 35 M 65
14
Introduction

• Does social skills training work?
• Godbold et. al, 2010
• 34 group design studies
• Random Assignment with Equivalent Starting Groups
  
• g0.67, plt0.05 BESD treatment 82
• Significantly higher than quasi-experimental
  designs

15
Introduction

• Godbold et. al, 2010
• Contrast Analyses

16
Introduction

• Research question
• Evidence is there for Primary Programs is there
  evidence for Secondary Programs?

17
Other Meta Analyses Since 2000
Meta Analysis Disability Type n studies Intervention Type ES statistic ES Degree of Effect
Bellini Akullian, 2007 Autism Spectrum 15 Video Modeling PND 81 Effective
Bellini Akullian, 2008 Autism Spectrum 7 Video Self Monitoring PND 77 Questionable/Effective
Bellini et al., 2007 Autism 15 Child Specific PND 71 Questionable
Bellini et al., 2007 Autism 7 Collateral Skills PND 75 Questionable
Bellini et al., 2007 Autism 20 Comprehensive PND 72 Questionable
Bellini et al., 2007 Autism 10 Peer Mediated PND 62 Questionable
Bellini et al., 2007 Autism 55 Total PND 70 Questionable
Kokina Kern, 2010 Autism Spectrum 18 Social Stories PND 60 Questionable
Wang Spillane, 2009 Autism 2 CBT d .47-1.24, .24-.59 Medium - Large, Small - Medium
Wang Spillane, 2009 Autism 1 CBT PND 100 Very Promising
Wang Spillane, 2009 Autism 9 Others PND 80.77 Effective
Wang Spillane, 2009 Autism 9 Peer Mediated PND 60.69 Questionable
Wang Spillane, 2009 Autism 6 Social Stories PND 67.21 Questionable
Wang Spillane, 2009 Autism 11 Video Modeling PND 84.25 Effective
Durlak Weissberg, 2010 Typical 68 After School Programs (Targeting Positive Social Behaviors) g 0.19 Small (Significantly different from zero)
Schneider et al, 2008 Autism 19 Social Behavior phi 0.72 Large
18
Summary

• 7 total studies
• 6 with Autism Spectrum, 1 with Typically
  Developing
• Multiple interventions available
• Effectiveness
• 3 very effective
• 3 moderately effective
• 1 not as effective (but still statistically
  significant)

19
Method

• Literature Search, 2000-2009
• Keyword Search in PsycINFO
• 5940 Articles Total

Social Skills Competence
Social Skills Intervention
Social Skills Training
20
Method

• Coding 1 Primary Inclusionary Criteria
• No Books, Reviews, Meta-Analyses, Group Designs,
  Dissertations
• Coding 1 Secondary Inclusionary Criteria
• Is study a social skills intervention or does it
  target a social skill? (YES)
• Does study focus on ages 3-21, or through high
  school? (YES)
• Does study target drugs, alcohol, or sexual
  offenders? (NO)
• 296 studies remained, 100 IOA in coding 1
  (approximately 22 of articles)

21
Method

• Coding 2 Primary Inclusionary Criteria
• Is the full article in English?
• Does article include single subject graphs?
• No AB design
• Coding 2 Secondary Inclusionary Criteria
• Does study fit our Social Skills definition?
• Gresham, Van, and Cook, 2006
• Facilitates initiating and maintain positive
  social relationships
• Contributes to peer acceptance and friendship
  development
• Results in satisfactory school adjustment
• Allows individuals to cope with and adapt to the
  demands of the social environment
• Is the study not part of a larger treatment
  package?
• Coded 190 unique studies
• 64 studies on to Coding 3 (IOA 92 for 38 of
  studies)

22
Method

• Coding 3 Primary Inclusion Criteria
• If one participant, more than 1 replication
  across setting or behavior
• Presence of variability in baseline and treatment
  conditions across at least 2 participants,
  settings, or behaviors
• Graphs in which UnGraph was able to score
• 40 studies eligible for analysis (IOA 100 on 30
  of studies)

23
Method

• Coding 3

• Design Type and Subtype
• Research Question
• Main Unit of Comparison
• Participant Info
• Phase Info
• Dataset Info
• Measurement Strategy
• DV information
• IOA
• Treatment Efficacy

• Study Quality
• Three replications across or within?
• Treatment Integrity?
• IV Operationally Defined?
• DV Operationally Defined?
• IOA

24
Method

• Data Extraction
• UnGraph (Biosoft, 2004)
• Extracts numerical data from graphs and puts it
  into Microsoft Excel
• High reliability and validity in collecting data
  from single subject graphs (Shadish et al., 2009)

25
Method

• Effect Size Calculation (Shadish, 2007)
• G (Mt Mb) / sp
• yields a standardized mean difference statistic
• Currently the best quantitative method available,
  but not absolutely accurate

26
Method
1) Calculate Mean of Baseline and Tx Panels One,
Two, and Three
3) Calculate Standard Deviation of BL and Tx from
the Mean of Means
2) Calculate BL and Tx Mean of Means across the
three panels
4) Calculate Effect Size for Positive Social
Interactions
27
Results
Participants   Disability Type   Ethnicity  
Total 148 ADHD 24 Reported of n 41.10
Age 2-5 28 ODD 9 Caucasian 28
Age 6-8 8 Asperger's Disorder 9 African American 21
Age 9-12 50 Autism 37 Hispanic 6
Age 13-18 18 BD, EBD, ED 19 Asian 1
Unspecified 8 Language Delay 2 Bi-Racial 1
Male 123 Intellectual Disability 9 Vietnamese 1
Female 23 Typically Developing 44 Latino 1
Classrooms 2 PDD 1 Native American 1
    Williams 1    
28
Journals
Journal Name of Articles
Education Treatment of Children 5
Behavioral Disorders 5
Focus on Autism and Other Developmental Disabilities 5
Journal of Positive Behavior Interventions 4
Psychology in the Schools 3
Behavior Modification 2
Therapeutic Recreation Journal 2
Journal of Autism and Developmental Disorders 2
Education and Training 2
Journal of Applied Behavior Analysis 2
Behavior Therapy 1
School Psychology Quarterly 1
Remedial and Special Education 1
Topics in Early Childhood Special Education 1
Journal of Early Intervention 1
Journal of the Association for Persons with Severe Handicaps 1
International Journal of Play Therapy 1
Education Psychology 1
Educational Psychology in Practice 1
29
Results

• Are social skills interventions evaluated by
  single subject methods effective?
• Yes
• g3.06

30
Results

• Independent Variable

IV n ES
Direct Instruction 17 2.79
Computer-Based 3 2.02
Peer Mediated 5 2.67
Self Management 6 3.53
Social Stories 4 3.94
SODA 2 5.54
Reinforcement Based 3 2.17
plt.05
31
Results

• Independent Variable
• Largest effects occurred with interventions that
  target self-awareness / self-monitoring of
  behavior as the primary independent variable

32
Results

• Dependent Variable

DV n ES
AET/On Task Bx 5 1.57
Total Destructive Behaviors (TBD)/ Problem Behaviors / Negativistic Bx 7 1.30
Prosocial Behaviors    
Cooperative Play 2 5.54
Positive Statements, Terminations 4 2.12
Prosocial Behavior 13 2.80
Sportsmanship 3 2.01
Social Play Skills - Recess 3 3.18
SILA 2 1.21
Seeking Help when lost 1 6.92
plt.05
33
Results

• Dependent Variable
• Larger effects are seen when targeting prosocial
  behavior over problem behavior

34
Results

• Recipient

• Director

Recipient n ES
Student 37 3.15
Classroom 1 2.68
Teacher 1 2.7
Peer 1 2.34
Director n ES
Experimenter 14 3.41
Teacher 10 3.02
Student 4 2.48
Computer 3 2.02
Coach/Caregiver 3 2.04
Interventionist 4 3.62
Peer 2 3.81
plt.05
plt.05
35
Results

• Recipient and Director
• Largest effects are seen when the student is the
  target of the intervention and when either an
  independent experimenter/interventionist or peer
  is the director of the intervention

36
Results

• Disability

Disability n ES
Autism 13 4.04
Asperger's 3 4.00
ADHD/ODD 4 2.47
ED/BD/EBD 4 2.31
Language Impaired 1 2.68
Intellectual Disability 2 1.82
Typically Developing 13 2.48
plt.05
37
Results

• Disability
• Interventions are most effective with children
  who have disabilities along the Autism spectrum

38
Results

• Setting

Setting n ES
School / Classroom 23 3.29
School settings not classroom 6 2.70
Recess 5 2.43
College Campus 2 2.60
Other 4 3.25
39
Results

• Setting
• The largest effects were seen when interventions
  were implemented in schools

40
Results

• Number of Components

Components n ES
2 4 4.02
3 9 2.66
4 8 3.38
5 3 2.31
6 10 3.02
7 4 3.34
8 1 1.84
9 1 2.68
41
Results

• Components
• Increasing the number of components in an
  intervention did not increase the magnitude of
  effect
• Size may not matter, quality matters

42
Results

• Treatment Integrity

Integrity n ES
Monitored 3 2.55
No 19 2.71
Yes 18 3.46
43
Results

• Treatment Integrity
• Studies that report percentage of integrity had
  largest effects

44
Results

• Reinforcement

Reinforcement Provided n ES
Yes 23 2.88
No 17 3.29
45
Results

• Reinforcement
• No difference in these studies in regards to
  reinforcement
• Could be a definitional issue. Coded articles
  that specified reinforcement given, not
  necessarily lack of reinforcement
• Reinforcement should be provided if necessary to
  acquire behavior change
• Quality of intervention may be more important for
  some

46
Results

• Study Quality

Quality n ES
3-3.5 5 3.49
4-4.5 22 2.30
5 13 4.16
47
Results

• Study Quality
• Highest effects were found in studies that had
  highest quality ratings, although not
  statistically significant
• Studies/interventions should be implemented with
  highest quality possible

48
Summary of Results
Contrasting Significantly Higher Significantly Lower
IV Self Awareness(1) Computer-Based IVs
DV Prosocial Behavior Problem Behavior
Recipient Student  
Director Peer, Interventionist Computer, Coach/Caregiver
Disability(2) Autism Spectrum Intellectual Disability
Setting School Recess
Components(3) Less More
Integrity Taken(2) Yes No
Reinforcement(3) Neither Neither
Study Quality Higher(4) Lower

1. Mixture of Ivs, all having self awareness
   qualities
2. Matches effect of group design study
3. Opposite of effect of group design study
4. Not significantly higher, but higher, and similar
   to our group design finding

49
Limitations

• Effect size estimator not entirely accurate
• No correction for small sample size
• Stringent selection criteria / data analytic
  method responsible for abandonment of nearly 40
  of single subject studies
• Both parametric and nonparametric methods would
  have had a larger n

50
Discussion and Future Directions

• All interventions were effective
• If assessment leads to a social skills deficit,
  with almost any kind of student, there are
  interventions that work
• Single subject meta-analyses are not yet as
  informative/definitive as they could be, but
  current best practice is still to aggregate
  magnitude and direction of effects across studies
• Calculate results using other available methods
  and comparing effects (NASP in Philly 2012????)

51
Questions and Comments?

• For additional copies of this presentation,
  either check the NASP website or contact Jeffrey
  Chenier at jcheni1_at_tigers.lsu.edu

52
Selected Citations

• Kratochwill, T. R., Hitchcock, J., Horner, R. H.,
  Levin, J. R., Odom, S. L., Rindskopf, D. M
  Shadish, W. R. (2010). Singlecase designs
  technical documentation. Retrieved from What
  Works Clearinghouse website http//ies.ed.gov/nce
  e/wwc/pdf/wwc_scd.pdf.
• Wolery, M., Busick, M., Reichow, B., Barton,
  E.E. (2010). Comparison of overlap methods for
  quantitatively synthesizing single-subject data.
  The Journal of Special Education, 44, 18-28.