Title: Topics in Special Education Research
1Topics in Special Education Research
- Session 3-Experimental
- Quasi-experimental Research
2This Afternoons Agenda
- Review Course Requirements Upcoming Assignments
- Ethics in Research
- Review for, Take, Correct Quiz
- Discussion Lecture on Experimental Research
- Activity
- Dismissal
3Updates/Questions
- Discussion Guides- Please submit to dropbox
folder and label - 3 main points for all of the readings for that
session. - No need to double space
- Address all of the headings (even the
difficulties, concerns, questions section). - Use complete sentences
- These are designed to guide your discussions.
- Please put your questions for ME on the feedback
guide, so if they arent answered by your
discussion group, type them on the feedback
guide!
4Upcoming Assignments
- Today NIH/CITI training modules
- Annotated Bibliography (individual assignment)
- Each individual reviews 3 research articles
regarding their topic - See Example http//rxsped596.pbworks.com/w/file/f
etch/54804527/Example20Annotated20Bibliography.p
df - Conceptual Framework (group assignment)
- Group submits short summary of literature and
presents a conceptual framework for theories that
drive their proposal. - Article Review Assignment (group assignment)
- Written Research Proposal (group assignment)
- Presentation of Research Proposal (group
assignment)
5Research Question(s) Guidelines
- Briefly and clearly state how each research
question will be addressed. - For example, This research question will be
answered by comparing the end-of-year state wide
test scores of students who received the
intervention and those who did not receive the
intervention. - Briefly present the proposed theoretical and
practical implications of the findings. - e.g., The results of this study may have
implications for the use of the evidence-based
Super-Duper Reading Intervention by elementary
special education teachers
6Research Question Tips
- Framed based to operationalize (clearly define so
that it can be replicated) the objectives of the
proposed research project. - Mention the IV and DV and how they will affect
each other - Framed based on methodology
- Experimental/Quasi-experimental/Single-subject
Is there a causal of functional relationships
between IV and DV - Correlational Is there a correlational
relationship between IV and DV - Descriptive/Qualitative Describe a phenomenon or
issue better, What is the prevalence of
intellectual disabilities in African-American
middle school students?
7Ethics in Research
- What did you learn from doing the CITI course?
- Regarding
- Research misconduct
- Data Management
- Responsible authorship
- Collaboration in Research
- http//www.youtube.com/watch?vW7sfIA1dIGQ
- http//www.youtube.com/watch?vyr5cjyokVUs
- http//www.youtube.com/watch?vb8McGyYAwcU
8The Belmont Report (1979), National Commission3
Principles
- 1. Respect for Persons-
- Required to obtain informed consent respect
privacy of participants - 2. Beneficence-
- Required to use best possible design to maximize
benefits minimize harms, show they can perform
the procedures and handle the risks, prohibit
research that that is without a favorable
risk-benefit relationship - 3. Justice
- Required to select participants equitably
avoid exploitation of vulnerable populations or
populations of convenience.
9PSU Human Subjects Research Review Committee
(HSRC)
- http//www.rsp.pdx.edu/compliance_human.php
- Portland State University (PSU) is responsible
for the rights and welfare of human subjects
involved in research sponsored or conducted by
the university. In order to meet this
responsibility, the University established the
Human Subjects Research Review Committee. -
- Members are charged with reviewing all research
conducted under the auspices of PSU that involves
human subjects to ensure adequate protections are
in place.
10Review for Quiz
11In-text Citations Formatting Quotations
When quoting, introduce the quotation with a
signal phrase. Make sure to include the authors
name, the year of publication, the page number,
but keep the citation briefdo not repeat the
information.
Caruth (1996) states that a traumatic response
frequently entails a delayed, uncontrolled
repetitive appearance of hallucinations and
other intrusive phenomena (p.11).
A traumatic response frequently entails a
delayed, uncontrolled repetitive appearance of
hallucinations and other intrusive phenomena
(Caruth, 1996, p.11).
12In-text Citations A Work with Two Authors
When citing a work with two authors, use and in
between authors name in the signal phrase yet
between their names in parenthesis.
According to feminist researchers Raitt and Tate
(1997), It is no longer true to claim that
women's responses to the war have been
ignored (p. 2). Some feminists
researchers question that women's
responses to the war have been ignored (Raitt
Tate, 1997, p. 2).
13In-text Citations A Work with Three to Five
authors
When citing a work with three to five authors,
identify all authors in the signal phrase or in
parenthesis. (Harklau, Siegal,
Losey, 1999) In subsequent citations, only use
the first author's last name followed by "et al."
in the signal phrase or in parentheses.
(Harklau et al.,
1999)
14In-text Citations A work with 6 or more authors
- When citing a work with more than 6 authors, you
do NOT have to identify all authors in the signal
phrase or parenthesis. - You identify the first author and use et al.
after his/her name with the date of the
publication - (Carr et al., 1999) OR
- signal phrase in text e.g.
- Carr et al. (1999) conducted a meta-analysis of
behavior support practices
15Steps in the Research/Scientific Process
- 1. Identify socially important issue
- 2. Review current literature
- 3. Define conceptual model
- 4. Define specific hypothesis(es) and research
question(s) - 5. Define dependent variable(s)/measure
- 6. Identify independent variable(s)/measures
- 7. Select appropriate research design
- 8. Obtain consents
- 9. Collect data
- 10. Analyze data
- 11. Communicate results
- Written presentation
- Oral presentation
16Conceptual Models (or theoretical models)
- Theory that drives the research.
- Guides our thinking and provides rules,
principles that guides the research and
practice. - Structure of assumptions, principles, and rules
that holds together the ideas of a broad concept.
- Outlines your research
17Logical Flaws of FBA use in public schools
(Scott et al., 2005)
- FBA is used mainly as a reactionary approach.
- opportunity is lost to utilize FBA technology to
develop interventions that address minor
behaviors that usually precede more serious
problems. - FBA is restricted to set of procedures used by
experts - The rich supply of information from people with
whom the student interacts with the most is lost. - FBA is restricted to rigorous procedures that are
unrealistic for public school settings. - Disincentive for using FBA technology.
- Cynicism as to the practicality of FBA .
18ProactiveParsimoniousPractical FBA in schools
- FBA conceptualized by Scott et al. as a proactive
pre-referral routine that uses the most
parsimonious procedures required to create an
effective behavior support plan. - Given the time resource constraints in schools,
we must encourage schools to work smarter to
develop capacity to implement technology to
effectively support more students. - Use Practical FBA procedures to develop capacity
within a school to utilize FBA technology.
19Practical FBA Logic Model
Individualized Supports 5 of Students
Behavior Specialist responsible for 25 FBAs in
school of 500
Personnel with flexible roles conduct proactive
Practical FBA to expand the scope of FBA,
prevent intensive problem behaviors, decrease
reliance on specialist.
Secondary Group Supports 10-15 of Students
School-wide Positive Behavioral Supports 80 of
Students
20Literature Review Guidelines
- A) You should educate readers about the topic and
provide a clear rationale as to why the study is
important and necessary based on the previous
research and writing on the topic. -
- B) Within your literature review you will present
the logic or conceptual framework as to why and
how your current study (topics, methods, designs)
is organized the way it is. - C) Make this section compelling. Concisely
explain the social importance of what you are
studying. - e.g., Start with a powerful statement or
statistic
21Quiz
22Correct Quiz
23Discussion
- Get together with 2-3 other people and use your
discussion guides to guide your discussion. - Spend more time on the chapter reading and the
Quality Indicator Article as we will be doing an
activity on the other two articles later!
24Lecture
- Experimental Quasi Experimental Research
- Research Designs Threats to Validity
- Attempt tonight to apply the principle of
teaching less more thoroughly. - https//www.youtube.com/watch?vqtLnBz6lbRQ
25Experimental Design
- Gold Standard in Research (Moore McCabe,
1993 Feuer, Towne, Shavelson, 2002 Slavin,
2002). - Rooted in postpositivist paradigm.
- Seeks to make causal conclusions.
- Difference between experimental design and
quasi-experimental design is the use of random
selection of participants.
26In both experimental quasi-experimental
designs.
- Emphasis is on operationally defining the
variables (dependent independent) and the
context of the research. - Dependent variable(s)- outcome variables (e.g.,
reading scores) - Independent variable(s)- variable that is
manipulated (the intervention or practice e.g.,
reading curriculum) - Context- defined clearly so replication can occur
(e.g., K-3 school with 200 students, etc.)
27Experimental Group vs Control Group
- Experimental (or treatment group)- receives the
intervention - Control group- business as usual.
- For true experimental research, participants
are randomly assigned to each group. - In order for to be considered random, every
person must have an equal chance of being in
either group
28Experimental and quasi-experimental research
involves
- Direct manipulation of an independent variable
(intervention)
29Validity
- Refers to whether a study is able to
scientifically answer the questions it is
intended to answer. - Extent to which your test (or study) measures
what it intends to measure.
30Internal Validity
- Changes observed in the dependent variable
(outcome) are due to the effect of the
independent variable (intervention).. not to
some other unintended variables (extraneous,
alternative explanations) - 12 threats to internal validity (noted by
Mertens, 2010) - https//www.youtube.com/watch?v_UPUtlHDM0A
3112 Threats to Internal Validity (think when
something other than the IV affect the results of
a study)
- History- events other than IV affected results?
- Maturation- changes in participants?
- Testing- participants became test-wise?
- Instrumentation- difference between pre- and
posttests? - Statistical regression- extreme groups used?
- Differential selection- groups differed in ways
other than exposure to IV? -
32Threats to Internal Validity continued
- 7. Experimental mortality- drop outs of study?
- 8. Selection-maturation- was selection a problem
based on the characteristics of the sample?
(e.g., participants in one group may have been
older) - 9. Experimental treatment diffusion- treatment
control groups shared information? - 10. Compensatory rivalry by the control group
- 11. Compensatory equalization of treatments
extra resources given to control group? - 12. Resentful demoralization of the control group
33External Validity (think generalizability)
- External Validity extent to which findings in
one study can be applied to another situation. - AKA ecological validity, generalizability
- 10 threats posed as questions (noted by Mertens,
2010)
3410 Threats (questions) to External Validity
- Were the variables, context, and treatment
described in sufficient detail? - Were multiple treatments used? Did they interfere
with each other? - Was the Hawthorne effect (special attention as
part of study affecting results) operating? - Was the treatment influenced by being novel or
disruptive? - What was the influence of the individual
experimenter?
35Questions to external validity continued
- 6. Were the participants sensitized by taking a
pretest? - 7. Were the participants sensitized by taking a
posttest? - 8. What was the influence of the type of
measurement used for the dependent variable? - 9. Was there an interaction of history and
treatment effects? - 10. What was the influence of the time period
that elapsed between the treatment and
administration of the dependent variable?
36Treatment Fidelityanother threat
- Treatment fidelity- implementer of the
independent variable follows the exact procedures
specified for administering the treatment(s).
37Strength of treatmentanother threat
- May not be reasonable to expect participants to
improve given the strength of the intervention.
- Intervention may not have been tried long enough
or delivered with adequate intensity.
38Coding system used for research designs
- R Random assignment of subjects to conditions
- XExperimental treatment (e.g., reading
curriculum) - O Observation of the dependent variable (e.g.,
test or observation measure)
39Single Group Designs
- One-shot case study
- X O
- Threats
- History, maturation, mortality (drop out)
- Other concerns using this design?
- No control group No pretest to know if it was
the intervention that affected outcome. - Very weak design
40- One-Group Pretest-Posttest Design
- O X O
- Threats
- History, maturation,
- What would help control for these threats?
- Control group- both groups taking the tests at
same time, but one not receiving the intervention - But sometimes it is difficult to find a control
group
41Time Series Design
- Involves measurement of the dependent variable at
periodic intervals. - O O O O O X O O O O O
- If behavior is stable in baseline (before
intervention), then change can be attributed to
intervention. - Controls for several threats
- Maturation, testing, differential selection (same
persons involved) - https//www.youtube.com/watch?vGUq_tO2BjaU
42Experimental Designs
- Pre-test-Post-test Control Group Design
- R O X O
- R O O
- Controls for what threats?
- history, differential selection, mortality
(pre-test can show differences in drop-outs) - Posttest-only Control Group Design
- R X O
- R O
- Controls for what threats?
- Same as above, except for mortality (no pretest)
43- Single-Factor Multiple-Treatment Designs
- X1 intervention 1
- X2 intervention 2
- .
- R O X1 O
- R O X2 O
- R O O
- Controls for threats because participants
randomly assigned to comparison groups and
pre-post-tests conducted.
44Solomon 4-group Design
- If worried about pretesting affecting validity
- R O X O
- R O O
- R X O
- R O
- https//www.youtube.com/watch?v7RRVW4iO7gA
45Quasi-experimental designs
- Similar designs to experimental designs, except
for - Lacking.
- Random assignment
- Cant make a strong causal statement
- https//www.youtube.com/watch?v_WpBxERGNVw
46How do we know if a research study involves
rigorous, systematic and objective procedures?
- CEC-Division for Research
- Sponsored prominent researchers to author papers
to propose - Parameters for establishing that reported
research has been conducted with high quality
(quality indicators) - Criteria for determining whether a practice has
been studied sufficiently (enough high-quality
research studies conducted on its effectiveness)
and shown to improve student outcomes (effects
are strong enough)
Graham, S. (2005). Criteria for evidence-based
practice in special education special issue.
Exceptional Children, 71.
47Exceptional Children (2005) volume 71(2)
- Group Experimental and Quasi-Experimental
Research (Gersten, Fuchs, Comptom, Coyne,
Greenwood, Innocenti) - Single-Subject Research (Horner, Carr, Halle,
McGee, Odom, Wolery) - Correlational Research (Thompson, Diamond,
McWilliam, Snyder, Snyder) - Qualitative Studies (Brantlinger, Jimenez,
Klingner, Pugach, Richardson)
48Quality Indicators (QIs) for Experimental (and
Quasi-Experimental) Research
- Describing Participants
- Sufficient information about participants and
interventionists, selection procedures as well as
comparability across conditions - Implementation of Intervention and Description of
Comparison Conditions - Clear description of intervention (and comparison
conditions) with implementation fidelity assessed - Outcome Measures
- Use of multiple measures at appropriate times
- Data Analysis
- Analysis techniques appropriate to questions and
unit of analysis with effect size calculated
49Sampling Strategies
- Probability-Based
- Non-probability Based
- Examples???
50Random vs Non-random sampling
Random Non-Random Sampling (Purposive)
All have equal chance independent chance Chosen based on criteria
Selects a representative of population No Equal chance
Should be large and random Some have no chance at all
No bias Some types show biases
51Nonprobability Sampling
- Convenience
- Judgmental
- Quota
- Snowball
- https//www.youtube.com/watch?v-kwdXEXC7yE
- https//www.youtube.com/watch?v1n_6kT42jO8
52Random Sampling Methods
- Simple Random Sampling
- Stratified Random Sampling
- Cluster Random Sampling
- Two-stage Random Sampling
53Simple Random Sampling
- Each individual has equal independence chance
of selection - The larger the sample the more it represents the
population - Any differences is not due to bias
- https//www.youtube.com/watch?vyx5KZi5QArQ
54Stratified Random Sampling
- Certain strata selected
- Sample in same proportion as they exit in the
population - Increases the likelihood of representativeness
- https//www.youtube.com/watch?vsYRUYJYOpG0
55Cluster Random Sampling
- Ideal to include certain groups/cluster
- However at times it is not possible to select
individual due to time, effort - Select individuals based on (not individuals)
- Groups
- Clusters
- Subjects
- Ex All 5th Graders in Selected Schools (cluster)
in Portland. - https//www.youtube.com/watch?vQOxXy-I6ogs
56Two-Stage Random Sampling
- Combination of Cluster Individual Random
Sampling - First select clusters randomly
- Then select students randomly from clusters
- https//www.youtube.com/watch?vQFoisfSZs8I
- https//www.youtube.com/watch?vx0XU0ngWpek
57Statistics, statistics
Descriptive Statistics
Inferential Statistics
Who is in your data?
What your sample says about the population
sample
population
Mean, Median, Mode, Standard Deviation, Variance
Tests of significance (t-, F-Tests)
58Tests of Significance
- Statistical analyses to determine whether a
difference is statistically significant
(probability for result to occur by chance). - Yes or No answer
- Alpha level (p)
- An established probability level which serves as
the criterion to determine whether to accept or
reject the null hypothesis - Common levels in education
- .01
- .05
- .10
Objectives 4.1 6.1
59Statistics, statistics
Descriptive Statistics
Inferential Statistics
Who is in your data?
What your sample says about the population
sample
population
Mean, Median, Mode, Standard Deviation, Variance
Tests of significance (t-, F-Tests)
60Inferential Statistics
- T tests- used when have two groups to compare.
- Independent samples t- if groups are independent
- Different people in each group
- Dependent samples t- if two sets of scores are
available for the same people (e.g., pre and
post-tests of same group) - Matched groups
- ANOVA (analysis of variance)- when you have more
than 2 groups to compare OR more than one
independent variable (reports an F-statistic,
which is basically a t-value squared) - ANCOVA (analysis of covariance)- ANOVA that
allows for control of the influence of an IV
(e.g., characteristics of people) that may vary
between your groups before treatment is
introduced. - Post-hoc method for matching groups on variables
such as age, prior education, SES, or a measure
of performance
61Effect Size
- Way of quantifying the difference between two
groups. - Not just was there an effect, but the magnitude
of the effect. - Many ways to calculate
- ES Mean of experimental group Mean of
control group/Standard Deviation - R-squared, Cohens-D
- Standard deviation is how well the mean
summarizes the data
62(No Transcript)
63(No Transcript)
64Visible Learningby John Hattie
- Over 800 Meta-analyses
- https//www.youtube.com/watch?vsng4p3Vsu7Y
65Article Review Use questions as your guide
- Implications for your practice- why you selected
the article - Introduction- what was the aim of the article,
what were the specific research questions - Methods- Describe
- Participants
- Setting of study
- Target behaviors- dependent variables
- How they measured the dependent variables
- Materials used
- Intervention procedures used- specify
- Were the procedures replicable?
66Article Review more questions
- Methods used for treatment integrity
- This asks how well does the author measure the
implementation of the intervention - What was the design?
- Experimental, quasi-experimental, single-subject
(specify what type of designmultiple baseline,
etc.) - Results- give quantifiable information of how the
intervention worked - Discussion- issues discussed, limitations?
- Conclusion- did they answer the research
question? - Reflection- commentary, questions
67In-Class Activity
- Get together with a partner.
- Practice completing an article review for one of
the articles you read this week. - Note the design of the study.
68Proposal Assignment Group Work
- A detailed explanation of the assignment is
posted on the wiki - What should you be doing in your groups?
- At this point you should have a topic and start
coming up with your framework for your research
project (based on literature). - Start to draft your conceptual framework,
research questions identify your dependent and
independent variables - You should walk away from your group time with a
list of tasks to complete.
69- Socially Important Issue
- 2. Conceptual Model/Hypothesis
- 3. Research Question(s)
- 4. Dependent Variable
- 5. Dependent Variable Measure
- 6. Independent Variable
- 7. Independent Variable Measure
- 8. Research Design
70Reflection feedback survey