Research and Evaluation in School Counseling

1
Research and Evaluation in School Counseling

• Third Summer Leadership Institute
• Amherst, MA
• Center for School Counseling Outcome Research

2
Terms

• Research use of the scientific method to
  discover generalizable truth about the
  relationships between variables
• Evaluation use of the scientific method to
  improve local decision-making

3
Variables

• Independent (manipulated)
• Dependent (outcome)

4
Validity of Experimental Inference

• Internal Validityconfounding variables have been
  eliminated as causes
• External Validityfindings may be generalized to
  the population at large

5
Threats to Internal Validity

• Selection bias
• Dropout
• History
• Reliability of Measures
• Small Sample Size
• Experiment-wise Error Rates

6
Threats to External Validity

• Representatives of the Sample
• Ecological Validity

7
Research and Evaluation
8
EVALUATION

• Intervention Evaluation
• Did the Real Game lead to incased career
  exploration at Carey Middle School?
• Program Evaluation
• Did implementing a National Model program at
  Carey High School result in better services to
  students?
• Policy Evaluation
• Are MA school counseling programs that are
  staffed at a 1/250 ratio able to achieve better
  outcomes that higher ratio programs?

9
RESEARCH

• Measurement Studiesdevelop reliable and valid
  measures of important variables
• Correlational Studiesunderstand the
  relationships among variables in order to develop
  possible causal pathways
• Laboratory Studiesunderstand the impact of
  simple interventions on outcomes in order to be
  able to construct comprehensive interventions
• Outcome Studiesunderstand the causal
  relationships between school counseling
  interventions and outcomes in real life settings

10
Research Issues

• Association Causality
• Association is the observation that certain
  characteristics or events tend to co-occur, e.g.
  smoking and cancer
• Causality is the observation that the first event
  (smoking) brings into being the second event
  (cancer)
• It is smoking, rather than some other
  characteristic that causes cancer
• Without smoking, cancer would not have occurred
• Association (or correlation) is straightforward
  to demonstrate, causality is much more difficult

11
Research Issues

• Prediction and Control
• Knowing a correlation allows prediction
• Knowing a causal relationship allows control

12
Research Issues

• Evidence for Causality
• X and Y must be correlated
• X must precede Y
• The relationship must not be spurious (due in
  fact to Z)

13
Research Issues

• Experimentation and Causal Relationships
• Group A gets X (Experimental Group)
• Group B gets O (Control Group)
• Y is observed in Group A but not in Group B

14
Research Issues

• Assignment of Ss to Groups
• True Experiment, Random Assignment
• Quasi Experiment, Artificial Equivalence
• Matching
• Statistical Equating

15
Research Issues

• Control Groups
• No Treatment
• Placebo
• Attention and Placebo Effect
• Placebo Single Blind
• Expectancy OR Experimenter Bias
• Placebo Double Blind
• Expectancy AND Experimenter Bias

16
Research Issues

• Statistical Significance vs. Size of Effect
• Statistical Significance
• A finding is statistically significant if there
  is a low probability that a difference as large
  as the difference found between the Control and
  Treatment group occurred by chance
• Is related to the size effect
• Is related to the number of subjects
• With a large number of subjects a finding may be
  statistically significant with a very small
  effect size

17
Research Issues

• Statistical Significance vs. Size of Effect
• Size of Effect
• Estimate of the difference between the Means of
  the Control and Treatment Groups in standard
  units
• Estimates how much of a difference the Treatment
  makes in terms of the measured outcome

18
Statistical Significance vs. Size of Effect
19
Research Issues

• Causal Chains and Path Analysis
• Models of causal relationships with many
  variables

20
Causal Model Lau, Roeser Kupermintze
www.cse.ucla.edu/CRESST/Reports/TR570.pdf
Test Engagement
Science Self-Efficacy
Classroom Engagement
Values Science
Standardized Science Test Scores
General Ability
21
Causal Model Lau, Roeser Kupermintze
www.cse.ucla.edu/CRESST/Reports/TR570.pdf
Test Engagement
Science Self-Efficacy
Classroom Engagement
Values Science
Science Course Grades
General Ability
22
Causal Model Lau, Roeser Kupermintze
www.cse.ucla.edu/CRESST/Reports/TR570.pdf
Test Engagement
Science Self-Efficacy
Classroom Engagement
Values Science
Choice of Science Major and Career
General Ability
23
Research and Evaluation Example Working Together

• Measurement Studies develop reliable and valid
  measures for key variables related to academic
  achievement
• Achievement Outcome Test
• Classroom Performance Assessment
• Academic Self-Efficacy
• Test Taking Skills
• School Social Skills
• Self Management Skills
• Academic Ability

24
Research and Evaluation Example Working Together

• Correlational Studies Suggests Likely Causal
  Relationships

Self- Management Skills
Classroom Performance Assessment
School Social Skills
Academic Ability
Academic Self-Efficacy
Achievement Outcome Test
Test-Taking Skills
25
Research and Evaluation Example Working Together

• Laboratory Studies Establish Causal Links
• Self Management Training leads to increases in
  classroom performance
• Social Skills Training Leads to increases in
  classroom performance
• Academic Self-Efficacy training leads to
  increases in both classroom performance and
  achievement test scores
• Test Taking Training leads to increases in
  achievement test scores

26
Research and Evaluation Example Working Together

• Outcome Studies establish that Success for All
  intervention (includes test-taking, self
  efficacy, social skills, and self-management
  components) leads to increases in classroom
  performance and achievement test scores for all
  subgroups of students

27
Research and Evaluation Example Working Together

• Intervention Evaluation indicate that Success for
  All increases academic achievement in Carey High
  School

28
Research and Evaluation Example Working Together

• Program Evaluation study indicates that the Carey
  High School school counseling program which
  implements Success for All and other
  research-based interventions increases the
  cognitive, personal social, and career
  development of students

29
Research and Evaluation Example Working Together

• Policy Evaluation Study indicates that a ration
  of 250/1 is necessary so that all students in
  Massachusetts High Schools have adequate exposure
  to a comprehensive program consisting of Success
  for All and related research-based interventions

30
Discussion
31
National Center for School Counseling Outcome
Research

• Thank You

www.cscor.org