THOUGHTS ABOUT GATEWAY 2001

1
THOUGHTS ABOUT GATEWAY 2001

• MARK ST. JOHN
• FEBRUARY 2001

Inverness Research Associates Box 313 Inverness,
Ca 94937 415669-7156
2
ENGINEERING THE DESIGN AND USAGE OF INNOVATIVE
CURRICULUMDESIGN CHOICES AND TENSIONS
3
DEVELOPERS CHOICES

• DIMENSIONS AND DEGREE AND NATURE OF INNOVATION
• TOPICS/COVERAGE
• INQUIRY
• CONTEXTUALIZATION
• TECHNOLOGY
• INTEGRATION
• ASSESSMENT
• STRUCTURE/APPROACH

4
DEVELOPERS CHOICES

• THE GRAIN SIZE (OR QUANTUM) OF CURRICULAR CHANGE
• ACTIVITY (INFUSION)
• MODULE (REPLACEMENT)
• WHOLE COURSE (STRUCTURAL)
• PROGRAM (CULTURAL)

5
Likelihood of various scenarios for introducing
new instructional materials comparison between
science and mathematics
6
IMPLEMENTATION CENTER CHOICES AND TENSIONS

• Work at wholesale or retail level
• Focus on particular curricula or on more generic
  change processes
• Develop Leaders to do the work or develop leaders
  through the work (and who?)
• Work with districts who are part of NSF projects
  or districts that are not part of NSF projects
• Focus on dissemination or on implementation

7
The Nature and Stages of Curricular Work
High Fidelity and High Quality Usage of Materials
System support for and wide Spread Usage of
Materials
Increasing cost, complexity,time, expertise
Selection and Adoption of Materials
Awareness of and Interest in Materials
Development of High Quality Materials
8
PICK TWO OUT OF THREE

• QUICK
• CHEAP
• HIGH QUALITY

9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
TRADE OFFS
16
MISCONCEPTIONS

• Of students
• Of the public
• Of Ourselves

17
Investing in Curriculum Development and
Dissemination

• But To What End?
• Widespread Mainstream Acceptance and Usage?
• A Means for Professional Development and
  Leadership Training?
• A Feasibility Proof? Leading Edge Development?
• Curricular Research and Knowledge Generation?
• Political Advocacy?

18
THE SCALE OF OUR EFFORTS

• On the order of 1 to 10 per student per year in
  the United States
• New Materials in Math and Science -- 10 out of
  6000 (.1)

19
Investments Made in the Development and
Implementation of Curriculum

• These are essentially investments made in the
  strengthening of the infrastructure that supports
  good instruction.
• Resources --such as curriculum --are one, and
  only one, component of the infrastructure needed
  to support good instruction.

20
The Nature of Investments Made In NSF Curricular
Projects
STUDENT ACHIEVEMENT
HIGH QUALITY INSTRUCTION
INSTRUCTIONAL INFRASTRUCTURE
IMPROVEMENT INFRASTRUCTURE
21
INFRASTRUCTURE

• The foundational set of capacities and resources
  that are necessary for the healthy functioning of
  the system.
• Infrastructure enables other activities
• Infrastructure bridges discontinuities
• Best evaluated by considering the effects of its
  absence

22
The Instructional Infrastructure
Well Designed Curriculum and Instructional
Materials
Competent And Well-prepared Teachers
Appropriate, Well-designed Assessments
Supportive Policies
Safe and Sane Environment
Financial Support
23
Curricular Capacities of Schools and Districts

• Vision
• Of teaching and learning
• Of curriculum
• Of a curricular change process
• Leadership Expertise
• in math/science
• In curricular implementation
• In all aspects of reform
• Materials and Resources
• Finances
• Support Systems

24
THINK BASF
You dont make the all of the curriculum they
use You make the curriculum they use better
25
The Nature of Investments Made In NSF Curricular
Projects
STUDENT ACHIEVEMENT
HIGH QUALITY INSTRUCTION
INSTRUCTIONAL INFRASTRUCTURE
IMPROVEMENT INFRASTRUCTURE
26
The Improvement Infrastructure
Curriculum Development Capacity And Support for
Usage
Professional Development Capacity
Assessment and Evaluation Capacity
Policy Development Capacity
Financial Capacity
27
CURRICULUM AND LEADERSHIP
REFORM LEADERSHIP
INNOVATIVE CURRICULUM
SUPPORTED NETWORKS
28
CONNECTIONS
CURRICULUM IMPLEMENTATION CENTERS
CURRICULUM DEVELOPERS
SYSTEMIC INITIATIVES
LSCs (Pilot LSCs)
Centers
29
INFRASTRUCTURE AS A WEB

• Infrastructure is strongest when its components
  all interact and support each other
• Subways, airports and highways
• Water, sewage, drainage and electrical systems
• Schools, universities, libraries

30
ACCOUNTABILITY

• In the schools and public climate
• For our own work

31
AN IMPORTANT QUESTION FOR ALL OF US HOW
SHOULD WE HOLD ACCOUNTABILITY ACCOUNTABLE?
32
The Logic (and Reality?)of StandardsBased
Accountability SystemsAs a Way to Improve the
System
ESTABLISH STANDARDS (Specify what students
should know and be able to do)
DEVELOP AND IMPLEMENT CONGRUENT ASSESSMENTS
USE THE RESULTS OF TESTS FOR FEEDBACK SET
REWARDS AND SANCTIONS FOR INDIVIDUAL
DISTRICTS/SCHOOLS/TEACHERS
RESULTS WILL MOTIVATE -- AND GUIDE -- A PROCESS
OF SCHOOL IMPROVEMENT
33
EVALUATING ACCOUNTABILITY AS AN
INTERVENTIONFive Key Questions

• What is the evidence that Accountability, as
  currently practiced, is
• Productive?
• guiding improvements, developing a supportive
  culture, and generating greater system capacity
• Scientifically sound?
• measures and inferences are valid
• Equitable?
• processes are fair and also equally serving of
  all groups
• Responsible for increased student learning?
• as assessed by independent measures
• Cost-effective?
• benefits are worth all the costs

34
Evaluating Investments In Educational
ResourcesTHE CHALLENGE for Funders, Program
Leaders and EvaluatorsTo Design A Rigorous
Assessment of Appropriate Outcomes versusA
Pseudo Rigorous Assessment of Inappropriate
Outcomes
35
The fundamental paradox

• The ultimate purpose and goal of investments made
  in the development, dissemination and
  implementation of curricular resources is to
  improve student learning
• But measuring changes in student learning is NOT
  the appropriate measure of the value of the
  investment made in those new educational
  resources.

36
So, if the focus is not on student
achievement,then what?

• The evaluation of investments made in educational
  resources --such as curriculum -- is most
  appropriately focused on assessing the degree to
  which and the ways in which the investment
  strengthens the capacity of the system to
  initiate and sustain a high quality and ongoing
  improvement effort.

37
THE NEW CHALLENGE How to Evaluate Investments
Made In INFRASTRUCTURE?

• Investments that seek to improve infrastructure
  create expertise and resources that build the
  capacity of the system to carry out a wide range
  of functions
• Ironically, to evaluate investments in
  infrastructure it is ultimately necessary to
  assess the consequence of their absence
• Appropriate evaluation of infrastructure
  investments focus on assessing the degree to
  which and the ways in which the capacity of the
  system is enhanced

38
Does the Investment Make a Difference?(A Turing
Test)
SYSTEM A (INFRASTRUCTURE A New Resources)
Blind Expert Observers
SYSTEM B (INFRASTRUCTURE B)
Comparative Judgments
39
A FrameworkFor Evaluating Investments In
Educational Resources

• Quality of resources
• Usage of resources
• User perceptions and experience
• Analysis of value-added to the instructional
  infrastructure
• Analysis of niche and connectedness within
  the system

40
Evaluating Investments In Educational
Resources(Think of the criteria you might use in
evaluating a new tool).

• Is there a need for the tool?
• Is there a market for the tool?
• Who could potentially use this tool?
• Has the tool been tested in real settings?
• Can the tool serve multiple functions?
• Is the tool practical in terms of all the costs
  involved?
• What are the threshholds for use in terms of
  training, logistics, power etc?
• Is the tool robust in its range of usage?
• What are the critical competitors?
• What is the expected lifetime of the tool?
• What risks are involved in the use of the tool?
• What are the potential benefits of this new tool
  (for both teachers and students)?

41
INVERNESS RESEARCH ASSOCIATES
WEBSITE www.inverness-research.org