EvidenceBased Practices: Scientific Procedures to Guide Instruction

1
Evidence-Based Practices Scientific Procedures
to Guide Instruction

• A module for pre-service and in-service
  professional development
• MN RTI Center
• Author Ann Casey, Ph.D.
• www.scred.k12.mn.us click on RTI Center

2
MN RTI Center Training Modules

• This module was developed with funding from the
  MN legislature
• It is part of a series of modules available from
  the MN RTI Center for use in preservice and
  inservice training

2
3
Overview

• Why do we need to know about evidence based
  practices? Education is a fad-ridden profession
• 3 levels of evidence based practice
• Criteria to judge whether a practice is
  research-based
• Evaluating research

4
Education is a fad-ridden profession

• We mistake opinion for fact
• Something written in a professional journal may
  be research but often is just opinions of the
  author
• Theories are just that - one has to do the
  research to prove or disprove a theory

5
What constitutes evidence (research) that a
practice is effective?

• Ellis Founts classification system (1994)
• Level 1 , Basic Research
• Level 2, Test of theory in real classroom
• Level 3, Program evaluation or large scale data
  collection

6
Level 1 - Basic Research

• Descriptive studies and qualitative studies are
  level 1
• Often use correlations to show the relationship
  between variables under study (e.g relationship
  between achievement and _______)
• Correlational data can not disprove a theory -
  instead correlations suggest the degree of
  relationship between variables

7
Level 2 - classroom application

• Use your hypothesis (theory) about what might be
  effective and try it out in a classroom
• In the best controlled circumstances, you have
  students/teachers randomly assigned to the
  intervention group and to a control group who
  doesnt get the intervention - then compare
  results

8
Level 3 - Large scale trials

• Evidence that a particular intervention works not
  just in one school or with certain types of
  students but works in a wide variety of settings
  (replication)
• Level 3 research is the kind of data we need to
  feel confident in the interventions we use with
  students

9
Fads - jump from Level 1 right into mainstream
acceptance

• Examples
• Theory of Multiple Intelligences
• Integrated curriculum
• Block scheduling
• Piaget
• Learning styles

10
Criteria to judge whether a practice is
research-based

• IRA recommendations
• Objective, 2 people come to same conclusion about
  the data
• Valid, data represent tasks we expect children to
  accomplish
• Reliable, data remain unchanged if collected on
  different day or by different person
• Systematic, data collected according to rigorous
  design of experimentation or observation
• Refereed, data that have been approved for
  publication by a panel of independent reviewers

11
What we need to do

• Encourage our teachers to use curriculum and
  instruction that have evidence to support their
  use
• Discourage teachers from picking and choosing
  based on their personal preference
• Grossen, To be a profession is to have a
  professional-knowledge base comprised of shared
  procedures that work.

12
Evaluating Research

• Theoretical framework.
• What was the theoretical basis of the program
  being considered? Is it related to learning
  theory and best practices in the field of study
  e.g. reading??
• Research design.
• Did the study have a control and an experimental
  group? Are there other reasons they may have
  gotten these findings?

13
Evaluating Research Continued

• Research methodology.
• What was the treatment for the experimental
  group? What activities were implemented? How many
  schools were in the study? In what settings did
  the research take place? Was the context similar
  to our school demographics?
• Implementation and replication.
• Is the treatment program understandable so that
  it can be replicated in our school? What kinds of
  resources were used in the study, and do we have
  the capacity to provide them?

14
Evaluating Research Continued

• Evidence of results.
• What assessment data were used to measure the
  treatment? If achievement testing, how comparable
  is this to our assessment system? Has the
  hypothesis been adequately tested in order to
  justify the conclusions? What statistical
  measurements were used, and do we have the
  capability to replicate them?

15
What is Quality Evidence?
16
Evaluating Research Continued

• Approved research.
• To what extent has the research been accepted by
  a peer-reviewed journal or approved by a panel of
  independent experts? Is the research considered
  rigorous, objective, and scientific to meet the
  SBR criteria?
• After completing these processes, the school
  leaders make a commitment to translate specific,
  applicable research into practices in their own
  setting. With these steps, the school has begun
  to incorporate research into its culture and to
  increase the likelihood of strengthening teaching
  and improving learning.

17
What makes a journal article research/evidence
based?

• It has an introduction to the topic
• It has a subheading referred to as the Method or
  Procedures section - this section tells you
  exactly everything you need to know about how
  they implemented the intervention

18
What makes a journal article research/evidence
based? Continued

• Another subheading entitled Results - where
  youll find the tables, graphs and statistical
  results
• Final subheading entitled Discussion section -
  this is where the author gets to say what they
  found out and why they think it turned out the
  way it did (their interpretations of the study.)

19
Accessing reviews of programs done by other
people

• What Works clearinghouse
• http//ies.ed.gov/ncee/wwc/
• Best Evidence Encyclopedia
• http//www.bestevidence.org/
• Florida Center for Reading Research review of
  programs
• http//www.fcrr.org/FCRRReports/reportslist.htm

20
Be a smart consumer of these reviews of programs

• Be sure you dig into the information enough to
  know what the intervention program is focused on
  or good at
• Even if it is a well reviewed program, if it
  target comprehension and your students need an
  intervention that targets phonics it wouldnt be
  a good choice
• Different sites (e.g., what works vs. best
  evidence vs. fcrr) use different criteria
• All of these sites use reasonable criteria, but
  they are not identical so you can get differing
  reviews. If that happens you can dig in and
  figure out why by reading the criteria or reading
  the reviews in more depth
• If a program was not reviewed it doesnt mean it
  is bad, just that it was not reviewed!

21
What else would you do?

• Look for the research yourself
• Google Scholar, ERIC, PsychInfo, EBESCO, etc.
• Use guidelines from Horner et al. 2005 or Gersten
  et al. 2005 to guide your decisions about whether
  the research was well done and look for
  different levels of research (basic, classroom,
  larger scale)
• Look at individual effectiveness with your
  students- collect progress monitoring data to
  show if it is working or not!

22
Activity Using program review internet sites

• 1. Compare and contrast the information (or
  lack) on what works, best evidence and fcrr sites
  on at least 2 of the following reading
  interventions
• Read Well, FastForword, Earobics, Read Naturally,
  PALS (from Vanderbilt)
• 2. If the intervention was not reviewed or if the
  reviews were conflicting, go to direct sources
  (ERIC, EBESCO, etc) and see if you can find any
  actual research studies on the intervention.
• 3. Would you recommend the interventions you
  examined as evidence based? why or why not? If
  so, for what students/skill focus?

23
Articles included with this module

• Institute of Education Sciences (ED), Washington,
  DC. (2003). Identifying and implementing
  educational practices supported by rigorous
  evidence A user friendly guide. Retrieved from
  ERIC.
• International Reading Association, Newark, DE.
  (2002). What is evidence-based reading
  instruction? A position statement of the
  international reading association Retrieved from
  ERIC database.
• Lembke, E. S., Stormont, M. (2005). Using
  research-based practices to support students
  with diverse needs in general education
  settings. Psychology in the Schools, 42(8),
  761-763. Retrieved from http//dx.doi.org/10
  .1002/pits.20110.

24
Articles included with this module, Contd

• Gersten, R., Fuchs, L. S., Compton, D., Coyne,
  M., Greenwood, C., Innocenti, M. S. (2005).
  Quality indicators for group experimental and
  quasi- experimental research in special
  education. Exceptional Children, 71(2), 149.
  For full text
• Horner, R. H., Carr, E. G., Halle, J., McGee, G.,
  Odom, S., Wolery, M. (2005). The use of
  single-subject research to identify
  evidence-based practice in special education.
  Exceptional Children, 71(2), 165-179. For full
  text

25
Articles included with this module

• National Inst. for Literacy, Washington, DC.,
  RMC Research Corp., Arlington, VA. (2005). What
  is scientifically based research? A guide for
  teachers. National Institute for Literacy.
  Retrieved from ERIC.
• Stanovich, P. J., Stanovich, K. E. (2003).
  Using research and reason in education How
  teachers can use scientifically based research
  to make curricular instructional decisions.
  Retrieved from ERIC database.
• Try to find and include pdf of the Gersten and
  the Horner articles.

26
References

• The Access Center. (2009). K8 access center.org
  Electronic references.  Retrieved (what date
  should I put here?) from http//www.k8accesscente
  r.org/index.php
• Ellis, A., Fouts, J. (1994). Research on school
  restructuring. Princeton, NJ Eye on Education.

DRAFT May 27 2009
26
27
Resources

• www.rti4success.org
• www.rtinetwork.org
• http//ies.ed.gov/ncee/wwc/
• http//www.bestevidence.org/
• http//www.fcrr.org/FCRRReports/reportslist.htm
• http//www.nectac.org/topics/evbased/evbased.asp

28
Quiz Question Essay Questions

• 1. List and explain the 3 levels of research
• 2. If you were asked to find a research based
  intervention for students with decoding problems,
  what would you do? And what criteria would you
  use to determine if the program/intervention was
  truly research based?

29
Quiz, Contd

• 3. Level 1 basic research is (a)
• A. Coming up with a hypothesis and doing a lit
  review
• B. Descriptive Study
• C. Research that shows causation
• 4. Which of the following is not a helpful way to
  judge whether a practice is research-based
• A. Research on it has to be published in a
  refereed journal
• B. Research on it has to be objective
• C. The research data must have been collected
  with rigorous designs of experimentation/observati
  on
• D. It has to have testimonials discussing
  whether it was useful to teachers or not

30
Quiz, Contd

• 5. Level 2 classroom application is not
• A. Preferably done in a controlled setting
• B. Testing your hypothesis in an actual
  classroom
• C. Done in a lab setting
• 6. T or F When looking at internet sites that
  review different curriculum or intervention
  programs, If a program is not reviewed you
  should assume it is not good.
• 7. Level 3 large scale trials are
• A. Important for us to feel confident in the
  intervention.
• B. Evidence that the intervention works in more
  than one setting.
• C. Replication
• D. All of the above

31

• Note The MN RTI Center does not endorse any
  particular product. Examples used are for
  instructional purposes only.
• Special Thanks
• Thank you to Dr. Ann Casey, director of the MN
  RTI Center, for her leadership
• Thank you to Aimee Hochstein, Kristen Bouwman,
  and Nathan Rowe, Minnesota State University
  Moorhead graduate students, for editing work,
  writing quizzes, and enhancing the quality of
  these training materials