DR K12 Evaluation: Introduction for Advisors

1
DR K-12 Evaluation Introduction for Advisors

• Kathy Haynie, 10/11/07

www.khaynieconsulting.com
2
Evaluation Positioning / Process

• Creating a shared understanding of the position
  of the evaluation
• What is the purpose of the evaluation?
• Where is the evaluation located in the context
  of the broader project ? How will it interact?
• Considering the evaluation process
• Evaluation Design and Planning
• Data Gathering
• Making Meaning (analysis, synthesis,
  interpretation)
• Use of Results

3
DR K-12 Evaluation
Final Results
Cycle 2 Development 2010-2011
Formative Evaluation Results
Cycle 1 Assessment Development 2008-2009
Baseline Assessment Development 2006-2007
4
Qualities of the Evaluation

• Formative
• Improvements to the process of using PADI / ECD
  for test design, in the context of the MCAT-II
  science assessment
• Summative
• Quality Improvements
• Efficiencies/Costs
• Technological Benefits

5
Evaluation Team

• Lead Evaluator
• Dr. Kathy Haynie
• Internal Panel
• Dr. Edward Haertel (lead)
• Dr. Cathy Kennedy, Dr. Nancy Songer
• External Panel
• Dr. Jamal Abedi (lead)
• Dr. Greg Chung, Dr. Kristen Huff
• External Independent Content Advisors
• Dr. Chris Swoyer, Dr. Steve Robinson

6
Role of Advisors

• Structuring design/delivery system according to
  current developments in large-scale testing,
  science education, psychology of learning,
  science of assessment, and technological
  developments ECD, PADI Design System
• In each of the design layers, looking for
  opportunities to enhance validity and efficiency
  of the process
• Everyone on the Advisory Panel bringing their
  expertise to bear in this
• Developing human expertise/capacity on the
  evaluation team, DR K-12 project team (and,
  indirectly, with test developers)

7
External Project Evaluator
External Project Evaluator
External Panel of Advisors, Content Advisors
Internal Panel of Advisors
Project Team
8
Year 1 Advisor Scope of Work

• Initial Web-Ex Meeting (today!)
• Theory of Change (December 2007)
• Review document and provide written feedback
• Participate in 2-3 hour Web-Ex meeting
• Evaluation Plan (March 2008)
• Review plan and provide written feedback
• Participate in 2-3 hour Web-Ex meeting
• Review Annual Report (August 2008)
• Advice based on monthly evaluation newsletters

9
Theory of Change

• Design begins with creating a Theory of Change
• Systematic, visual way to present connect parts
  of a project, how a project will work

Resources / Goals
Short-term Outcomes
Long-term Outcomes
Activities

• Why is a Theory of Change important?
• Project Planning elements relationships
• Project Management consensus, monitoring
• Project Evaluation activity links to outcomes

10
Developing a Theory of Change

• Key stakeholders participate in a series of
  discussions
• Delineate program design and available resources
• Specify major program goals, attributable to this
  program intervention
• State program theory (why it will work)
• Specify strategies, objectives, major
  activities for each goal
• Provide measurable short-term outcome(s) for each
  action, including assumptions or conditions for
  success
• Provide clear logic linking the measurable
  outcome(s) to the specific action
• Provide long-term outcomes, associated with
  program objectives and goals
• Delineate, generally, major assumptions/conditions
  for success

11
Project Constraints

• Test specifications for MCA-II science
• Look and feel of MCA-II test items
• Test development cycles steps and timing
• MCA-II 5th and 8th grade science for general
  student population (accommodated MTAS)
• Existing technologies supporting MCA-II
• Security for operational test items