Mississippi E2T2 Grantee Evaluation Institute

1
Mississippi E2T2 GranteeEvaluation Institute

• May 28, 2008

2
Introductions

• Mississippi Department of Education
• Office of Instructional Technology
• Robin Silas, Rsilas_at_mde.k12.ms.us
• Lee Bray, Lbray_at_mde.k12.ms.us

3
Housekeeping

• Cell phones turned off, please.
• Breaks
• Restrooms
• Parking lot
• Cards on table for questions
• Other

4
Institute Objectives

• Create a logic map that reflects how your project
  works.
• Identify evaluation questions that will help you
  judge the implementation and impact of your
  project.
• Identify methods and measures for collecting data
  about your project.
• Develop benchmarks to measure progress of your
  project.
• Understand how to use results from your formative
  evaluation to make decisions about your project.
• Develop an evaluation management plan.

5
Todays Agenda

• Welcome and Introductions
• Who we are
• Why we are here
• A brief overview of formative evaluation
• Lets talk about our E2T2 Projects
• Taking time to explore the critical components of
  our projects.
• Examining how the critical components add up to
  success

6
Introductions

• SERVE Center _at_ UNCG
• Beth Thrift, bthrift_at_serve.org
• Kathleen Mooney, kmooney_at_serve.org
• Holli Bayonas, hbayonas_at_serve.org
• Nita Matzen, matzennj_at_appstate.edu

7
Introductions

• Grantees
• Introduce the members of your team,
• Including name and position.
• In one or two sentences, share one thing that
  will be different for students as a result of
  your project

8
Introductions

• Getting to know everyone
• a
  little better.

9
Capacity for Applying Project Evaluation (CAPE)
http//www.serve.org/Evaluation/Capacity/
10
SERVEs Role in the Mississippi E2T2 Projects

• Help schools gain knowledge and understanding
  about formative evaluation
• Help schools learn how to evaluate their
  projects
• Plan evaluation
• Collect data
• Analyze and interpret data
• Use findings to make decisions
• Communicate with Stakeholders

11
The Mississippi E2T2 Grantees Role

• Actively participate in professional development
  on project evaluation.
• Review and revise project evaluation plan as
  appropriate.

12
The Mississippi E2T2 Grantees Role

• Implement the evaluation plan
• Collect data
• Analyze and interpret data
• Use findings to make decisions
• Communicate with stakeholders
• Keep MDE informed of the projects progress, any
  problems encountered, and any major changes in
  project activities.

13
What is Evaluation?

• Evaluation is the systematic investigation of
  merit or worth
• Uses a thoughtful and purposeful approach.
• An appraisal or judgment of value, based on the
  best evidence available.
• Is done for specific reasons.

14
Formative EvaluationThe Process and Key
Decisions
V-6
15
Every Evaluation hasthese Components

• Planning
• Data collection
• Data analysis and interpretation
• Results

16
Why Formatively Evaluate Education Projects?

• Wadesboro Elementary School
• Wadesboro, NC

17
Why Formatively Evaluate Education Projects?

• MonitorandAdjust

18
Why Formatively Evaluate Education Projects?

• Why would your district or school want to
  evaluate one of its projects or programs?
• Group brainstorm, then share

19
Why Evaluate Education Projects

• Does your project match both the needs and
  interests of your students?
• Determine if weve accomplished the goals and
  objectives in our project and to what degree
  quality and quantity
• Determine effectiveness of project did your
  outcomes meet your expectations?

20
Why Evaluate Education Projects

• See if student outcomes have improved.
• What changes need to be made to the project.
• How the students felt about the project after
  its concluded.
• Monitor how resources are being used and how
  effectively.

21
Why Evaluate Education Projects

• Evaluate to pinpoint specific problems (students
  are not progressing or project is not going the
  way you expected) so they can be addressed.
• Determine whether benefits warrant the cost.

22
What is Formative Evaluation?

• When the cook tastes the soup, thats formative
  evaluation. When the guests taste the soup that
  is summative evaluation Bob Stake.
• Formative evaluation is the purposeful appraisal
  of a project, to make ongoing decisions about
  project implementation.
• As cited in Patton, M. Q., Utilization-Focused
  Evaluation The New Century Text. Edition 3.
  Thousand Oaks, CA Sage, 1997 p. 69.

23
Understanding Your Project

• As a team
• Review your proposal
• Look at each of the questions on the following
  slides
• Discuss possible responses
• Record consensus responses on large sticky note
  cards, one item per card

24
Understanding Your Project

• Ultimately, what good is your project going to do
  for your students?

25
Understanding Your Project

• Ultimately, what good is your project going to do
  for your students?
• What are the major activities that youre going
  to do?
• (How are you going to spend your time, energy,
  and money?)

26
Understanding Your Project

• Ultimately, what good is your project going to do
  for your students?
• What are the major activities that youre going
  to do?
• If youre successful at implementing these
  activities, what will your project accomplish
  this school year?

27
Understanding Your Project

• Arrange your sticky note cards in a logical
  order on your table or flip chart paper.
• Work from left to rightfrom major activities to
  ultimate good for students.

28
Evaluation Concepts

• Goals
• Goals tell the ultimate purpose(s) of the project
  or program. They answer the question What
  difference will the project or program make in
  the long run?
• In education, goals usually say what long-term
  impact the project will have on learners and
  learning.

29
Evaluation Concepts

• Objectives
• Focus on project outcomes.
• Answer the question How will learning,
  (teaching, the school, etc.) be improved as a
  result of this project?
• Describe what you hope will happen as a result of
  the project.

30
Evaluation Concepts

• Objectives for technology projects often focus
  on
• What teachers do (instructional practice) as a
  result of the project
• What students do (student learning) as a result
  of the project
• What the school/community/learning environment is
  like as a result of the project.

31
Evaluation Concepts

• Strategies
• Identify the major components of the project
• Are larger in scope than day-to-day project
  activities and
• Provide details about how the project funds are
  spent.

32
Evaluation Concepts

• Strategies for technology projects often focus
  on
• Professional development
• Infrastructure
• Personnel
• Technology resources
• Collaboration/communication with the community

33
Evaluation Concepts

• Inputs
• Are the contexts or conditions that influence
  project activities and strategies.
• Should be considered when planning project
  activities, e.g., policies, plans, needs
  assessment data.

34
Understanding Your Project

• Brainstorm a list of things that influence your
  project (e.g. policies, needs assessment data,
  etc.), put it to the left of your strategies, and
  label them inputs.

35
Logic Maps

• A logic map is a graphic representation of the
  relationships among the key elements of a project
  (goals, objectives, strategies, and inputs).
• Helps to articulate the key elements of the
  project.
• Can lead to evaluation efficiency and
  effectiveness.
• Promotes stakeholder buy-in by helping clarify
  how the project works.
• Coffman, J. (1999). Learning from Logic Models.
  Cambridge, MA Harvard Family Research Project.

IV- 1-8
36
Healthy LivingLogic Map Example
IV-9
37
Understanding Your Project

• Start with the ultimate benefit to students, on
  the right side these will become the goals of
  your project, so label that column.
• Label the column of items describing how you
  invest your time, energy, and money as your
  strategies, on the left.
• The outcomes are possible objectives of your
  project, so put a label above them.

38
Understanding Your Project

• Draw lines to show the linkages between the
  inputs and strategies, between the strategies and
  objectives, and between the objectives and goals,
  as your team sees fit. Strive for consensus.

39
Understanding Your Project

• Team Glossary
• Your definitions of key concepts and terms
  related to your project

40
Using If-Then ReasoningTo Check Your Logic Map
41
Understanding Your Project
www. ScienceCartoonsPlus.com
42
Using If-Then Reasoning to Check Our Logic

• What do you notice?
• How do we use if-then reasoning to check the
  relationship between components of the example
  logic map?

43
Understanding Your Project

• Use If-Then reasoning to check the logic of each
  linkage in your map.
• Work out any differences between the logic map in
  your proposal and your new Inspiration document.
• This should represent a consensus understanding
  of how your project works, to turn major
  project activities into good for your students.

44
Understanding Your Project

• Some hints to make your logic map useful
• Resist the temptation to link everything.
• You can color code your linkages.
• You can have more than four columns, if it helps
  explain your thinking.
• Use concise language. Make each element
  straightforward.
• Check for dead ends each element (box) should
  link to at least one other element.

45
Museum Walk

• One team member stays at your table
• Other team members walk around and visit other
  teams tables to learn about their projects
• Ask questions if you do not understand something
• Write any questions on the note cards provided at
  each table

46
Understanding Your Project

• As a team, consider these questions as you
  resolve the remaining differences
• What is the same? These probably stay
• What is different? Choices must be made
• What do you keep?
• What do you set free?
• What is truly important? Are there too many
  goals? Or objectives?
• What is reasonable for a one year project?

47
Understanding Your Project

• Using Inspiration
• Demo and tips
• Symbol Palette inserting symbols
• Links creating, editing
• Tool Bar zoom, font, symbol/link color
• Insert your concepts
• Work from left to rightfrom spending money to
  ultimate good for students
• Duplicate the structure on the tables/flip chart

48
Checking Your Logic Map

• Using Inspiration
• Use the Logic Map Self Check as a guide for
  reviewing your logic map.

IV-10
49
Todays WRAP-Up

• Keep in mind
• Evaluation is an iterative process.
• Q A
• Daily reflection
• Plus/Delta