ScientificallyBased Research: Choosing Effective Programs

1
Scientifically-Based Research Choosing
Effective Programs

• Michelle Mehlberg
• South Dakota Department of Education
• Michelle.mehlberg_at_state.sd.us
• 605-280-3614

2

• Early in the 17th century, two astronomers
  competed to describe the nature of our solar
  system.
• Galileo built a telescope and found new planets
  and moons. Francesco Sizi ridiculed Galileo's
  findings. There must be only seven planets, Sizi
  said. After all, there are seven windows in the
  head--two nostrils, two ears, two eyes, and a
  mouth. There are seven known metals. There are
  seven days in a week, and they are already named
  after the seven known planets. If we increase the
  number of planets, he said, the whole system
  falls apart. Finally, Sizi claimed, these
  so-called satellites being discovered by Galileo
  were invisible to the eye. He concluded they must
  have no influence on the Earth and, therefore, do
  not exist.
• Sizi's most valuable contribution to history may
  have been to remind us that true understandings
  of the world, and how it works, cannot be based
  on pure thought alone, no matter how logical,
  creative, or contemporary such thought may seem.
• True understandings require some measure of
  science and the willingness to seek information
  when making decisions.

3

• Teachers use the concepts of rigorous research
  and evaluation in profoundly practical ways

4
Effective teachers use scientific thinking in
their classrooms all the time

• use experimental logic when they plan for
  instruction
• evaluate their students' previous knowledge
• construct hypotheses about the best methods for
  teaching
• develop teaching plans based on those hypotheses
• observe the results
• and base further instruction on the evidence
  collected.

5

• Teachers can further strengthen their instruction
  and protect their students' valuable time in
  school by scientifically evaluating claims about
  teaching methods and recognizing quality research
  when they see it.

6
NCLB Act -

• Enacted in 1965 - redefines the federal role in
  K-12 education.
• Stipulates that federally funded programs and
  practices must be grounded in "scientifically
  based" research.
• School leaders who depend on federal funding are
  now required to be aware of the nature of the
  research that
• guides their programs and practices. In
  particular, federal funding decisions will be
  determined in part by whether programs and
  practices have a basis in "scientific research."

7
To meet the NCLB definition of "scientifically
based," research must

• employ systematic, empirical methods that draw on
  observation or experiment
• involve rigorous data analyses that test the
  stated hypotheses and justify the general
  conclusions
• rely on measurements or observational methods
  that provide valid data across evaluators and
  observers, and across multiple measurements and
  observations
• be accepted by a peer-reviewed journal or
  approved by a panel of independent experts
  through a comparatively rigorous, objective, and
  scientific review

8
Recognizing Effective Research
9
Questions to Ask

• Has the study been published in a peer-reviewed
  journal or approved by a panel of independent
  experts?
• Have the results of the study been replicated by
  other scientists?
• Is there consensus in the research community that
  the study's findings are supported by a critical
  mass of additional studies?

10
Independent peer review

• Subjects a paper to scrutiny by scientists in the
  relevant field of specialization.
• Two ways.
• a paper submitted for publication in a
  peer-reviewed journal is examined by other
  scientists in the field before an editor (usually
  an expert in the field) passes judgment on it.
• review by an independent panel of experts who,
  using rigorous criteria, determine whether the
  findings of the paper are credible.
• Provides a baseline of quality control
• It is relatively easy for teachers to determine
  if a paper has been published in a peer-reviewed
  journal it can be more difficult to determine
  whether a panel review (without publication) has
  occurred unless it is specified in the paper.

11
Peer Review Journals

• Examples of journals that conduct peer reviews
  and contain empirical evidence about teaching
  techniques.
• The American Educational Research Journal,
• the Journal of Educational Psychology,
• and Reading Research Quarterly
• Examples of journals that contain original
  thought but do not publish peer-reviewed
  research
• Phi Delta Kappan and
• Educational Leadership

12
Replication

• Look for evidence that an instructional technique
  has been proven effective by more than one study.
  
• Research finding presented in a way that enables
  other researchers to reach the same results when
  they repeat the experiment.
• Public and open to challenge. It is held
  tentatively, subject to change based on contrary
  evidence.

13
Consensus within a research community

• Community of scientists in a field comes to
  agreement over time that sufficient evidence has
  converged to support one theory over another.
• Scientists do not evaluate data from a single,
  perfectly designed experiment. They evaluate data
  from many experiments, each containing some flaws
  but providing part of the answer.

14
The basic principles of the scientific method

• Science progresses by investigating testable
  problems.
• A testable theory yields predictions that could
  possibly be proven wrong.
• Scientific knowledge has passed some minimal
  tests.
• Data and theories are considered in the public
  domain, or included in the research base, only
  after a peer review, either by a journal or a
  panel.
• Published data and theories allow for replication
  and criticism by other scientists.
• Theories are tested by systematic observation
  bound by the logic of true experiments.
• Correlational studies, useful when experiments
  can not be carried out, only help rule out
  hypotheses.
• Researchers use many different methods to reach
  conclusions. Most often, they draw conclusions
  only after a slow accumulation of data from many
  studies.

15

• The NCLB Act (2002) defines scientifically based
  research as "research that involves the
  application of rigorous, systematic, and
  objective procedures to obtain reliable and valid
  knowledge relevant to education activities and
  programs."

16
Choosing Effective Programs How do we
know???????
Success For All
Accelerated Reader
Reading Recovery
Waterford
Failure-Free Reading
Accelerated Literacy Learning
Literacy First
Essential Learning Systems
17
Choosing Effective Programs Questions to Ask

• Is the program based on educational theory
  derived from research?
• What information is provided about implementation
  and replicability?
• Is there evidence of a significant positive
  effect on student achievement?

18
Theoretical Base

• What are the ideas behind this program?
• Is there a clear, non-technical description of
  the central idea and objectives of the practice
  or program?
• How does it work?
• Is there a clear description of the instructional
  activities that are central to this program or
  practice?
• Why does it work?
• Is the program clearly tied to an established
  learning theory, e.g. language acquisition?

19
Implementation and Replicability

• Has this program or practice been widely used?
• How many schools have used this practice or
  program?
• Did the schools using it fully implement the
  practice or program?
• What is the effect if the implementation varies?
• Are we prepared to implement the program as
  described?

20
Implementation and Replicability

• Where is this reform likely to work?
• In what settings has it been implemented?
• Are the schools where it has been implemented
  similar to ours?
• In size
• In location (rural, urban)
• In demographics (ethnic/racial, poverty, student
  disabilities)

21
Implementation and Replicability

• Under what circumstances is it most effective?
• Has improved student achievement been
  convincingly demonstrated in a variety of
  settings?
• Large group, small group, one-on-one
• Highly qualified teachers or paraprofessionals
• During regular school day, extended day
• Days per week, hours per day
• Additional resources
• Do these settings match ours?

22
Evidence of Significant Positive Effect on
Student Achievement

• Are there studies looking at the impact on
  students of this particular program?
• Are these studies of high quality?
• Are there at least 5 high quality studies?
• Do at least 4 of the 5 studies show improved
  student achievement?
• If yes, are the findings significant in at least
  3 of those 4 studies?

23
NO
24
YES
Yes to All
Yes to All But One
25
Yes to All
26
SBR Review Tool
27
In Summary

• As teachers make the important decisions about
  what and how to teach, they must evaluate the
  claims associated with educational strategies and
  programs. And in the classroom, they must
  constantly assess and reassess the value of
  programs and their impact on students.

28
SBR Resources

• Florida Center for Reading Research
• http//www.fcrr.org/
• IDEA Institute for the development of
  Educational Achievement
• http//reading.uoregon.edu/curricula/index.php
• Center for Implementing Technology in Education
• http//www.cited.org/
• National Center on Response to Intervention
• http//www.rti4success.org/
• Center on Instruction
• http//www.centeroninstruction.org/
• Doing What Works
• http//dww.ed.gov/disclaimer.cfm?/index.cfm
• What Works Clearinghouse
• http//ies.ed.gov/ncee/wwc/
• The Center for Comprehensive School Reform and
  Improvement
• http//www.centerforcsri.org/
• Vaugh-Gross Center for Reading and Language Arts
• http//www.texasreading.org/utcrla/

29

• Center for Research on Education, Diversity, and
  Excellence (CREDE),  http//www.cal.org/crede/
• Center for Research on the Education of Students
  Placed At Risk (CRESPAR), http//www.csos.jhu.edu/
  CRESPAR/index.htm
• Center for the Improvement of Early Reading
  Achievement, http//www.ciera.org
• Charles A. Dana Center, University of Texas,
  http//www.utdanacenter.org
• Educational Research and Improvement Reports and
  Studies, United States Department of Education,
  http//www.ed.gov/about/pubs/intro/index.html?src
  gu
• Educational Resources Information Center, United
  States Department of Education,
  http//www.eric.ed.gov/
• Eisenhower National Clearinghouse,
  http//www.Enc.org
• Institute of Education Sciences, United States
  Department of Education, http//www.ed.gov/about/o
  ffices/list/ies/index.html
• National Center for Early Development and
  Learning,  http//www.fpg.unc.edu/ncedl/
• National Center for Improving Student Learning
  and Achievement in Math and Science,
  http//www.wcer.wisc.edu/NCISLA/
• National Institute for Literacy,
  http//www.nifl.gov/
• National Research and Development Center on
  English Learning and Achievement,
  http//cela.albany.edu
• Northwest Regional Educational Laboratory,
  http//www.nwrel.org/

30
Questions?