Determining Impact of Treatments on Student

1
Determining Impact of Treatments on
Student Achievement in Math and Science
Presentation for the 2003 Invitational Urban
Indicator Data and Evaluation Workshop
June 6, 2003 Atlanta, Georgia
Carla J. Thompson, EdD Associate Professor of
Statistics and Teacher Preparation External
Evaluator, Oklahoma City Public Schools NSF
USP Tulsa Community College Tulsa, Oklahoma
2
OKLAHOMA Oklahoma City Public Schools
NSF Urban Systemic Project
3
Presentation Outline Atlanta, June 6, 2003
1. Legislation and Policy Mandates for Research
2. Implementing and Measuring Standards-based
Education
3. Determining Impact of SBE Differences and
Contributions
4. Specific Contributions of SBE Practices to
Achievement
5. Next Steps for Research
4
1 .
Legislation and Policy Mandates for Research
5
WORLD REPORTS
T I M S S
Germany
Norway
Japan
6
REACTIONS
Oklahoma's PASS
NCTM NSTA
Texas's TASS
7
no child left behind
2001
8
selected NCLB policies....
..closing the achievement gap
.improving teacher quality
influences on SBE ?
specific mandates for math and science education
9
2 .
Implementing and measuring Standards-based
education ?
10

Standards for Math and Science
Content Standards
Teaching Standards
Professional Development Standards
Assessment Standards
Program System Standards
NCTM and NSTA
11
Standards-based Math Education
Teaching Standards
Content Standards
Number and Operations Algebra Geometry Measurement
Data Analysis Probability Problem
Solving Reasoning and Proof Communication Connecti
ons Representation
Equity Principle Curriculum Principle Teaching
Principle Learning Principle Assessment
Principle Technology Principle

12
Standards-based Science Education
Teaching Standards
Content Standards

Unifying concepts and processes in
science Science as inquiry Physical Science Life
Science Earth and space science Science and
technology Science in personal and
social perspectives History and nature of
science
Inquiry-based Responsive to diversity Assessment
driven Appropriate time, space and
resources Social communities Involved in
planning
13
Implementing Standards-based Education
Reports and Journals
Multiple Assessments
Technology
Manipulatives
Data Analysis
Cooperative Learning
Projects-based
14
NSF Urban Systemic Program provides
FORMAL and INFORMAL
Professional Development
15
Over 700 teachers have received formal training
from the NSF USP during 2001 and 2002
1
16
NSF USP P.I.E.
Partners In Excellence provide

• Lesson Modeling
• One-on-one teacher consulting
• Classroom Observations
• Resource Information
• 5. Workshop training

17
OKC teachers have received over 1000 hours of
informal training from the NSF USP during the
first two years of the program.
18
Watch for these SBE activities in the classroom !
Awareness Checlists for Teachers and
Administrators
19
Sample Descriptions of Standards-based Classroom
Activities
Additional Information for Implementation

• Hands on activities using concrete manipulative
  materials to perform
• problem-solving and laboratory exploratory
  activities in math and science
• Small group discussion using pairs and small
  groups of students to discuss
• and verbally report processes and findings in
  math and science
• Use of technology using computers, and other
  technologies to search for
• resources, investigate processes, conduct data
  analyses, and explore
• laboratory simulations and problem-solving tasks
  in math and science
• Calculators for problem-solving using
  calculators in solving real world
• applications in simulation and laboratory
  settings related to math and science
• Journal and report writing using journals and
  reports to summarize investigative and
  problem-solving processes as well as to promote
  reflective writing related to attitudes and
  feelings about math ad science

20
Sample Descriptions of Standards-based Classroom
Activities (continued)

• Inquiry and scientific method using discovery
  and hypothesis testing
• approaches to problem-solving and laboratory
  explorations in math and science
• Multiple types of assessment using
  observations, portfolios, self-reporting
• tools, journal reflections, and other measures
  of performance in math
• and science
• Models and experimentation using modeling,
  demonstrations, and
• experimentation with concrete materials to
  simulate processes and solutions
• for problem-solving and laboratory explorations
  in math and science
• Student-directed projects using
  student-initiated topics and student-developed
• questions to promote student-directed projects
  in math and science
• Real world applications using real world data,
  issues, and questions in
• problem-solving and laboratory explorations in
  math and science

21
Non-Standards based Activities

• Students listen to lecture
• Students work on worksheets
• Students take tests or quizzes
• Students work on homework from textbook
• 5. Students work on independent practice

22
3 .
What is the impact of Standards-based
education in OCPS ?
23
..... measuring SBE
Observing Standards-based education in schools
requires trained professionals !
NSF USP P.I.E.
Partners In Excellence
24
Teacher Practice
25
Observed Standards-based activities in OCPS
(2002-2003)
26
Observed Standards-based activities in selected
OKC schools (2002-2003)
27
3 Determining Impact of SBE Differences and
Contributions
.tuning up with t tests F tests R tests
28
Relational Database Standards-based Education
Classroom Implementation
Professional Development
Classroom Observations (monitored by assessment)
Formal and Informal PD (monitored by assessment)
ITBS/CRT/SAT9 Data-driven Decision-Making
Student Performance/Achievement
29
Connecting the first two components of the
relational database
Professional Development
Classroom Implementation
?
Student Achievement
30
Finally adding the third component..
student achievement
ITBS
31
Differences
t test analyses
SBI versus non-SBI
Contributions
regression analyses
Specific SBI strategies in the classroom
32
Example of SBE Impact (t tests)
33
Example of SBE Impact (t test)
34
Example of SBE Impact (t tests)
35
Example of Impact of Standards-based Content
Prepared Teachers (t tests)
36
Middle school math certification is a critical
need for OKCPS !
KEY CONCERN
Low math achievement of middle school
students generally indicates.
Low math content knowledge of middle school
teachers
37
Example of Informal Teacher Training Impact (t
tests)
38
Stepwise Multiple Regression Model
Math and Science Achievement (ITBS)
Classroom Observations SBI and non-SBI N 15
schools N 359
Dependent Variables
Independent Variables
Math(SBI) m1, m2, m3, m4, m5, m6, m7,
m8 Science(SBI) s1, s2, s3, s4, s5, s6, s7,
s8 Non-Sandards ns1, ns2, ns3, ns4, ns5, ns6,
ns7, ns8
Math ITBS scores Science ITBS scores
39
Independent Variables for Regression
Math SBI
Science SBI
Non-Standards
S1 inquiry-based S2 hands-on activities S3 lab
experiments S4 pairs/small groups S5
Internet S6 computers S7 calculators S8
self-assessments
NS1 lecture NS2 text/homework NS3
worksheets NS4 textbook work NS5
tests/quizzes NS6 independent work NS7
unrelated work NS8 restricted movement
M1 projects-based M2 manipulatives M3
journals/writing M4 pairs/small groups M5
Internet usage M6 computers/technology M7
calculators M8 self-assessments
Dependent Variables for Regression
Math and Science Achievement
ITBS Scores
40
Contributions of specific SBI practices on
achievement
Math SBI
Science SBI
Non-Standards
S1 inquiry-based S2 hands-on activities S3 lab
experiments S4 pairs/small groups S5
Internet S6 computers S7 calculators S8
self-assessments
NS1 lecture NS2 text/homework NS3
worksheets NS4 textbook work NS5
tests/quizzes NS6 independent work NS7
unrelated work NS8 restricted movement
M1 projects-based M2 manipulatives M3
journals/writing M4 pairs/small groups M5
Internet usage M6 computers/technology M7
calculators M8 self-assessments
Math ITBS (N359) M2, M8, M1
R2 3 of variance in ITBS
None of the non-Standards activities were found
to be contributors to ITBS
Science ITBS (N359)
R2 5 of variance in ITBS
S8, S6
None of the non-Standards activities were found
to be contributors to ITBS
41
Contributions of specific SBI practices on
achievement (by Gender)
Math SBI
Science SBI
Non-Standards
S1 inquiry-based S2 hands-on activities S3 lab
experiments S4 pairs/small groups S5
Internet S6 computers S7 calculators S8
self-assessments
NS1 lecture NS2 text/homework NS3
worksheets NS4 textbook work NS5
tests/quizzes NS6 independent work NS7
unrelated work NS8 restricted movement
M1 projects-based M2 manipulatives M3
journals/writing M4 pairs/small groups M5
Internet usage M6 computers/technology M7
calculators M8 self-assessments
Math ITBS Males(N154) M7
Females(N167) M8, M2
R2 2 of variance in ITBS
R2 4 of variance in ITBS
None of the non-Standards practices contributed
to ITBS
Science ITBS
R2 5 of variance in ITBS
Males (N 154) S6, S8
Females (N167) S8, S6
R2 3 of variance in ITBS
None of the non-Standards practices contributed
to ITBS
42
Contributions of specific SBI practices on
achievement (by Ethnicity)
Math SBI
Science SBI
Non-Standards
S1 inquiry-based S2 hands-on activities S3 lab
experiments S4 pairs/small groups S5
Internet S6 computers S7 calculators S8
self-assessments
NS1 lecture NS2 text/homework NS3
worksheets NS4 textbook work NS5
tests/quizzes NS6 independent work NS7
unrelated work NS8 restricted movement
M1 projects-based M2 manipulatives M3
journals/writing M4 pairs/small groups M5
Internet usage M6 computers/technology M7
calculators M8 self-assessments
Math ITBS White(N105) M8, M2
R2 4 of variance in ITBS
Black/Hispanic(N 215) M2
R2 2.5 of variance in ITBS
NS1 (White group)
R2 5 of variance in math ITBS
Science ITBS
White(N105) S8, S1
R2 6.5 of variance in ITBS
Black/Hispanic(N215) S6, S8
R2 7 of variance in ITBS
NS7, NS1 (White group)
R2 8 of variance in ITBS
43
Contributions of specific SBI practices on
achievement (by Grade)
Math SBI
Science SBI
Non-Standards
S1 inquiry-based S2 hands-on activities S3 lab
experiments S4 pairs/small groups S5
Internet S6 computers S7 calculators S8
self-assessments
NS1 lecture NS2 text/homework NS3
worksheets NS4 textbook work NS5
tests/quizzes NS6 independent work NS7
unrelated work NS8 restricted movement
M1 projects-based M2 manipulatives M3
journals/writing M4 pairs/small groups M5
Internet usage M6 computers/technology M7
calculators M8 self-assessments
Math ITBS Middle School (N149) M2, M1
R2 3 of variance in ITBS
High School (N175) M8, M4, M2
R2 7.5 of variance in ITBS
None of the non-Standards practices contributed
to ITBS scores
Science ITBS Middle School (N149) S2, S1
R2 2.5 of variance in ITBS
R2 3 of variance in ITBS
High School (N175) S6, S1
None of the non-Standards practices contributed
to ITBS scores
44
Summary Comments on Contributions of SBI to Math
and Science Achievement
Greatest contributors to math achievement are
self-assessment and manipulatives followed by
projects/reports and calculator practices.
Greatest contributors to science achievement are
computers, inquiry, assessment followed by
hands-on practices.
Non-Standards based instruction (lecture)
contributes to white students math
and science achievement
IMPLICATIONS ?.
45
5 .
Next steps for research ...
46
These data-driven examples of successful SBE
strategies employed by the OKCPS NSF USP in math
and science reflect the essence of a relational
database driving decision-making
Classroom Implementation
Professional Development
Classroom Observations (monitored by assessment)
Formal and Informal PD (monitored by assessment)
ITBS/CRT/SAT9 Data-driven Decision-Making
Student Performance/Achievement
47
What can we do to close the gap ?
Examine effects of Standards-based strategies for
math and science. Model and prepare teachers to
use Standards-based activities. Provide formal
teacher preparation in content (math/science
certification).
Use data as a positive
driver for change
48
Next steps for research efforts

• Continue to monitor and assess the implementation
  of
• Standards-based education in classrooms.
• Develop and monitor multiple types of student
  performance self-assessment measures in the
  classroom.
• Assess and compare varying types of
  hands-on/manipulative
• materials used in the classroom.
• Assess and compare varying types of inquiry
  materials/activities
• used in the classroom.
• Examine the implementation of SBE content impact
  in the classroom.
• Continue to empirically examine all of the
  components of
• relational databases.

49
closing gaps improving teacher quality
Standards-based Education
50
Visualize the future ....
Standards- based Education
empirically connected to learning