Education, Evidence, and Policy

1
Education, Evidence, and Policy

• Cecilia Elena Rouse, Ph.D.
• Member of the Council of Economic Advisers
• June 8, 2009

2
The Council of Economic Advisers was established
by Congress in the Employment Act of 1946

• There is hereby created in the Executive Office
  of the President a Council of Economic Advisers
  (hereinafter called the Council). The Council
  shall be composed of three members who shall be
  appointed to appraise programs and activities
  of the Government and to formulate and recommend
  national economic policy to promote employment,
  production, and purchasing power under free
  competitive enterprise.

3
It shall be the duty and function of the Council
--

• to assist and advise the President in the
  preparation of the Economic Report
• to gather timely and authoritative information
  concerning economic developments and economic
  trends.
• to appraise the various programs and activities
  of the Federal Government.
• to develop and recommend to the President
  national economic policies to foster and promote
  free competitive enterprise, to avoid economic
  fluctuations or to diminish the effects thereof,
  and to maintain employment, production, and
  purchasing power
• to make and furnish such studies, reports
  thereon, and recommendations.

4
Educators
Scientific Learning's family of programs and
products are for anyone who wants to improve
language, reading and overall communication
skills. From children who struggle with basic
language skills or attention problems to adults
who want to improve reading comprehension and
organizational skills, Scientific Learning's
programs and products have something for
everyone.
5
Rouse, Krueger, and Markman Evaluation of Fast
ForWord

• 4 schools in a large urban school district in the
  Northeast
• Eligibility based on low reading score on Fall
  2001 State Assessment (Bottom 20)
• 513 students in grades 3-6
• Random assignment within grade and school
• Administered CELF-3 and Reading Edge Test SFA
  Assessments

6
CELF-3 Test Gains Among FFW Participants
N 43 Effect size0.3s
p-value0.008
7
Reading Edge Test Gains Among FFW Participants
N 244 Effect size0.7s p-value0.000
8
But, what would have happened had the children
not participated in FFW (i.e., whats the
counterfactual)?
9
CELF-3 TestGains Among FFW Participants and
Controls
N 89 Net Effect size0.04s
p-value0.73
10
SLCs Reading Edge TestGains Among FFW
Participants and Controls
N 485 Net Effect size0.13s p-value0.098
11
Post FFW SFA Assessment FFW Participants vs.
Experimental Controls
N 197 FFW Participants 176 Experimental
Controls Effect size0.05s p-value0.583
12
Post FFW SFA Assessment FFW Participants vs.
Non-experimental and Experimental Controls
Experimental effect size0.05s (p-value0.583)
Non-experimental effect size-0.3s
(p-value0.000) N 197 FFW Participants 3850
Non-experimental Comparisons 176 Experimental
Controls
13
Non-experimental vs. Experimental Estimated
Effects of FFW on Gain in SFA Assessments
Non-experimental N4048 Net Effect
size0.04s p-value0.11 Experimental
N373 Net Effect size0.01s
p-value0.60
14
Example of Results that Led to Push for College
for All
15
Chicagos College Prep for All

• Instituted in 1997
• Requires all students to take
• Four years of English Three years of Math,
  science, and social science
• Particular classes Algebra I, geometry, Algebra
  II (math) survey literature, American
  literature, European literature, world literature
  (English) biology, earth science, and chemistry
  or physics (science) world studies, U.S.
  history, and one elective (social science)

16
First Results from Chicagos College Prep for
All (Mathematics)
17
Limits of most rigorous studies for policy

• Heterogenous treatment effects
• Size of likely impacts are difficult to
  interpret/translate
• Difficulty or ease of bringing to scale
• Typically doesnt illuminate the mechanism by
  which a particular program/policy works
• Longer-run impacts are usually difficult because
  of cost (especially time).

18
So Where Does this Leave Us?

• Current evidence-base for most education
  programs/reforms is extremely thin.
• Need more high-quality impact evaluations to help
  guide everyday decisions.
• Itll take cooperation from all to improve the
  quality of education research on impacts.

19
What Is PART?

• Standardize performance assessments across
  government
• Not used enough in decision-making
• Reform will emphasize cross-agency program goals
  and transparency

20
How PART Processes Unfold
Presidents Budget
Inform Decisions
Congressional Action
Define Program Activity
Assessment (OMB Agency)?
Improvement Plans Created
OMB Internal Consistency Check
Formal Agency Appeals Process
Appropriations, Authorization, Oversight
Executive Branch
Periodic Reassessment
Share Best Practices
ExpectMore
Build Public Trust