Evaluation: Exploring a Reporting Framework

1
Evaluation Exploring aReporting Framework The
Community Nutrition Education Logic Model

• Helen Chipman
• Program Coordinator, FSNEP, CSREES/USDA
• Prepared for ASNNA Post-Conference, Washington DC
• 28 February 2003

2
Getting to the Same Page

• Purposes To answer questions
• What are we getting for our nearly 400 million
  dollar investment (199 million food stamp
  administrative dollars)?
• Are we making a difference with our target
  population?
• Why Developed
• Needed for CES/Land Grant System a national
  report that reflects impact without loosing the
  richness of program diversity

3
Getting to the Same Page

• Framework to report what happens in the short,
  medium and long term, with respect to
  individuals/households, institutions/communities,
  social structures/policies
• Outcomes core areas identified by FNS
  commissioned white papers
• Theoretical basis, use of well known model

4
Getting to the Same Page

• Developed within Extension/Land Grant System
  others had opportunity to provide input,
  recommendations incorporated into the model (all
  networks, youth evaluation workgroup, FNS/Food
  Stamp Program - nationally)
• FNS has accepted use of the CNE Logic Model, with
  disclaimer not exclusively Food Stamp Nutrition
  Education Logic Model

5
Getting to the Same Page

• Does not replace reports/information requested by
  FNS
• Information requested should be readily available
  for transfer to other reports, whoever the
  stakeholder is
• Pilot effort learn what states have refine
  based on what is already in place
• Identify gaps (in states and nationally) -
  strengthen planning/implementation

6
Getting to the Same Page

• Is something where we had nothing!
• Is live the www.csrees-fsnep.org website
  contains
• Two training modules 1) logic models in general
  2) CNE Logic Model specifically
• The CNE Logic Model (module 2, section 3)
• Worksheet for states to use in preparing state
  reports (module 2, section 3)

7
The State Report

• For Extension/Land Grant System others can use
  if desired
• Will be used to communicate regionally and
  nationally about Food Stamp Nutrition Education
  to tell the story
• Information from FY 2002 make it strong and
  complete, but brief
• States can send 1 or 2 reports (may or may not
  send separate network report)

8
The State Report

• State coordinators/directors or their designees
  to complete
• Email attachment to secretary of Helen Chipman
• Contractor will aggregate into regional and
  national reports also provide feedback to states

9
The State Report

• Projected timeline
• April 15, 2003 report due to Helens secretary
• June 30, 2003 feedback to states
• August 15, 2003 regional and national reports
  available on the national website
• Actual times depend on what we receive and
  contractual arrangements we are able to make to
  develop the reports

10
State Reports Network Considerations

• Are considerable number contracted through the
  Extension/Land Grant System important to report
• Socio-ecological model partially collapsed to
  reduce complexity (institutions/communities)
• Endpoints model doesnt resolve the ongoing
  discussion determined from FNS guidance/research

11
Steps Taken to Explain How to Complete the
Worksheet

• Asked Joyce Counihan to review the worksheet
  comments incorporated into web-based document
• Reviewed Food Stamp Nutrition Education Networks
  Partners for Better Health Publication
• Completed the worksheet based on information
  state Networks provided, national statistics, and
  some extrapolation

12
Situation Statement Description of States Food
Stamp Population

• State population 6 million people
• State poverty rate 15
• State food stamp participation 60 of 1,000,000
  eligible average of 185 food stamp
  benefits/household/month
• Children seniors disproportionately represented
  ____ children ____ seniors ____ single
  parent homes
• Gender, race, household composition

13
Situation Statement Description of States Food
Stamp Population, cont.

• Dietary Quality Substantial percentages of
  households falling short of RDAs 69 of iron,
  79 of folate
• Food Security In 2001, 50 of food stamp
  participants experienced some level of food
  insecurity, which they attributed to lack of
  financial resources (ref) 30 thousand
  experienced outright hunger (ref)

14
Situation Statement Description of States Food
Stamp Population, cont.

• Rising demands at food banks emergency food
  providers around the state attribute the increase
  in demand to working families the faltering
  economy, and

15
Situation Statement Description of Issues of
Concern

• Dietary Quality
• Poor iron absorption despite high iron intakes
  question link to enriched and fortified foods
• Increased obesity rates
• Decreased meals together as families
• Low intake fruits and vegetables
• General poor nutrition among seniors

16
Situation Statement Description of Issues of
Concern, cont.

• Food Security
• Lack of food access
• Transportation issues
• Other barriers (list)
• Food Safety
• Identify
• Shopping Behavior/Food Resource Management
• Identify

17
Situation Statement States Objectives for FY
2002

• Encourage increased fruits and vegetable
  consumption by limited resource individuals
• Increase consumption of low-fat, calcium-rich
  foods
• Increase physical activity of limited resource
  individuals by 3
• Connect state objectives to Healthy People 2010

18
Situation Statement States Objectives for FY
2002

• Increase participation in Food Stamp Nutrition
  Education
• Increase participation in the Food Stamp Program
  by eligible persons
• Strengthen the abilities of community nutrition
  coalitions to increase healthful nutrition and
  physical activity practices among low-income
  audiences

19
Inputs Financial Resources Budgeted Dollars

• FNS Funds 499,000
• Matching Funds, state 260,000
• Matching Funds, local/other 241,000
• Total 1,000,000

20
Inputs Financial Resources Budgeted Dollars,
comments

• State funds State government and university,
  state grants
• Local funds County government, local network
  partner funds, local grants
• Does not include private in-kind contributions,
  other federal funds
• Additional state/local contribution of 50,000
  not included in cost share

21
Inputs Planning Process

• Conduct needs assessment/analyze the situation
• Segment and select the target audience
• Conduct formative research
• Set goals and objectives
• Develop marketing and communication strategies
• Conduct concept and content pre-testing

22
Inputs Planning Process, cont.

• Develop promotion plan and communication
  materials
• Pre-test, refine, and produce materials
• Implement program/conduct social marketing
  campaign
• Conduct process and outcome/impact evaluation at
  all stages
• Revise as needed

23
Inputs Materials Curriculum
Name of Curriculum Check if new Check if in NAL
Nutrition, Good For You
Jump Start your Bones X
Five a Day through Theatre and Role Play X
24
Inputs Materials Educational Media

• Include content that is not as extensive as a
  specific curriculum and not part of social
  marketing campaign

25
Inputs Materials Social Marketing Campaign

• Jump N Jive... Come Alive with Fruit -- includes
  tip sheets, interactive displays, posters,
  newsletter articles, school lunch menu backs,
  food demonstrations and food sampling, stickers
  and balloons Contact (name, institution/agency,
  state, email address, phone)

26
Inputs Materials Social Marketing Campaign

• Pick a Better Snack -- promotes fruits and
  vegetables as snacks includes public service
  announcements recipes, newspaper/newsletter
  articles, videos Contact (name,
  institution/agency, state, email address, phone)

27
Inputs People Expertise

• Steering committee professionals with expertise
  in nutrition, management, marketing, finances,
  education.
• Network composition number of persons,
  description of expertise, etc.

28
Inputs People Accountability

• Routine fiscal, personnel, affirmative action
  audits within university/ institutional system
  quarterly reports to grants and contracts office
  time and effort reporting (plan confirmation/
  other)
• Periodic reporting to/from partner agencies
  (describe)

29
Inputs People Accountability, cont.

• Accountability procedures within partner
  infrastructures (describe)
• Annual face-to-face meeting with state food stamp
  fiscal office representative

30
Inputs People Intra-Institutional
Relationships (STATE Level)

• Collaboration between your state Food Stamp
  agency and other state agencies that administer
  FNS and other USDA programs
• Team Nutrition/Food Stamp Program Partnership
  Why needed, how compliment, but do not duplicate
• WIC FSNEP Partnership Why needed, how
  compliment, but not duplicate
• EFNEP Same
• Others

31
Inputs People Intra-Institutional
Relationships (STATE Level), cont.

• Head Start Consortium access to low-income
  parents of young children
• Fruit Vegetable Growers Association provide
  foods for social marketing campaign to low income
  families
• 5 African American Organizations, 3 Tribal
  Organizations, 2 Faith-Based Organizations
  access to limited resource population

32
Outputs Individual/Household Activities
Direct Methods Direct Methods Indirect Methods Indirect Methods
X Education class X Public service announcements
Workshop X Billboards
Group discussion X Newsletters
One-on-One intervention X Other (describe) Educational Fairs
Other (describe)
33
Outputs Individual/Household Participation
Method Teaching Contacts If possible, provide further breakdown of direct contacts If possible, provide further breakdown of direct contacts If possible, provide further breakdown of direct contacts
Gender Counts Ethnicity Counts Adult/ Youth Counts
Direct 37,650
Indirect 900,000 Distributed to low-income neighborhoods based on average clientele served at respective sights Distributed to low-income neighborhoods based on average clientele served at respective sights Distributed to low-income neighborhoods based on average clientele served at respective sights
34
Outputs Individual/Household Participation
Comments

• Direct methods 12 lessons per child 3-5 years
  old in head start classes
• Indirect methods posters, bookmarks, signage,
  billboards newsletters, calendars, radio, and
  newspapers

35
Outputs Institution/Community Activities
Community Assessment X Integration of Services
X Community Awareness Campaigns Other please list
36
Outputs Institution/Community Participation
Sector Sector Sector Sector Sector
Public Private Non-Profit Schools Private Other
Number of Partnerships 10 2,000
37
Outputs Institution/Community Comments

• School districts primarily middle school and
  high schools
• Private supermarkets, neighborhood markets, and
  warehouse stores

38
Outputs Social Structures, Policies, Practices
Activities
Impact Seminars X Expert review/comment on federal, state, and/or local public policies
Public Forums X Other please list Bus tour with policy makers
39
Outputs Social Structures, Policies, Practices
Participation

• Many Faces of Food and Agriculture bus tour
  with continuing dialogue included network
  stakeholders, industry leaders, and public
  officials to provide first-hand look at food
  production, delivery, and consumption issues in
  low-income area of state, with special attention
  to examining impact of issues and the system on
  low income citizens

40
Impact Statement -- 1

• Objective 3-5 year old children try new foods
• Indicator ID DQ 1
• Core Element Dietary Quality
• Level of Intervention Individual/Household
• and/or Community/Institutions
• Time Frame Short Term

41
Impact Statement 1, cont.

• In 2002, preschool children who attended head
  start or preschool participated in a 12-week
  new-foods intervention class. Of the 432
  children that participated in a tasting party
  evaluation at the conclusion of the class, 86
  tried all four of the novel foods 96 tried two
  of the novel foods.

42
Impact Statement 1, cont.

• What is the significance of this impact, e.g. why
  is it important that children try new foods?
• How representative is this number of preschool
  age children in the state geographic area,
  pilot project, convenience sample?
• Does the number represent all children who
  participated in 12-week intervention?

43
Impact Statement 1, cont.

• How many children shared the information with
  their parents later?
• What was the impact for parents did they
  request additional information, did they offer
  the new foods at home?
• Did the new foods become part of the
  institutions menu options?
• Any evidence that the foods are being eaten
  repeatedly after exposure?

44
Impact Statement 1, cont. Data Collection Method

• Teachers tracked childrens responses at
  conclusion of 12-week intervention
• likes it
• doesnt like it
• didnt try it
• At end of class or after class had been completed?

45
Impact Statement 1, cont. Tools

• Survey
• Developed through formative research?
• How validated?

46
Impact Statement -- 2

• Objective Individuals eat more healthy by
  eating breakfast
• Indicator ID DQ 1
• Could also be DQ 18 or 19
• Core Element Dietary Quality
• Intervention Individual/Household
• Could also be Community/Institution
• Time Frame Medium Term

47
Impact Statement 2, cont.

• Between 2000 and 2002, 74 of participants became
  aware of the campaign and 12 reported increasing
  their frequency of eating breakfast as a result
  of the campaign.

48
Impact Statement 2, cont.

• Why is eating breakfast important? Tie back to
  dietary quality.
• Who was targeted general public, low income
  public, others? Clarify.
• How do you know you reached your target audience?
• How frequently were they eating breakfast? Any
  baseline comparisons of how much change this
  represented?

49
Impact Statement 2, cont.

• What about community organizing impact numbers
  and types of partners who joined to promote
  eating breakfast those who were part of the
  social marketing campaign?
• What did the 300 public and private partners do
  differently because of Food Stamp Nutrition
  Education?
• Any lasting commitment or further action?

50
Impact Statement 2, cont. Data Collection Method

• Secondary data
• Telephone contacts
• In-depth interviews
• How wide scale was the campaign the whole
  state a targeted area, etc.?
• Describe more specifically how you arrived at the
  -- random sampling, convenience, etc.

51
Impact Statement 2, cont. Tools

• Local Behavior Risk Factor Surveillance System
  (BRFSS) (behavior change)
• Telephone Surveys (awareness and behavior change)
• In-Depth Interviews (process evaluation)
• Which were developed for project?
• Validation?

52
Program of Excellence

• Project Title Senior Nutrition Education at
  Congregate Food Sites
• Core Element Addressed
• Dietary Quality
• Food Security
• Objective
• Strengthen the abilities of community nutrition
  coalitions to increase healthful nutrition
  practices among targeted lowincome seniors.

53
Program of Excellence, cont.

• Timeline
• Start October 2001
• End September 2002
• Location
• 50 congregate meal sites
• Where?

54
Program of Excellence, Project Description

• Issue of Concern
• Poor nutrition among seniors due to lack of
  knowledge, changing environment (living alone),
  change tastes, etc.
• Description of Strategy
• Train nutrition managers at meal sites. Cost
  shared with Area Agencies on Aging, who paid
  staff to attend training and covered expenses
  related to nutrition education on site. Food
  Stamp Nutrition Education funds were used to
  cover training material costs.

55
Program of Excellence, Project Description, cont.

• Description of Audience and Participation
• Primary audience nutrition managers at
  congregate meal sites
• Secondary audience lowincome seniors who eat
  at congregate meal sites

56
Program of Excellence, Project Description, cont.

• Materials
• Take Charge of Your Health The Active Older
  Adult Speakers Kit, developed by Wellness Inc.
  with support from the Georgia Nutrition Network
• The Kit included easytofollow speaker notes,
  master transparencies and handouts, which
  facilitated presentations that would not have
  occurred otherwise due to the lack of time,
  resources, and knowledge among nutrition
  managers. Materials are creative and engaging.

57
Program of Excellence, Project Impact

• Project Impact
• Community Organizing New Partnership A local
  network project used Take Charge of Your Health
  at congregate meal sites and lowincome senior
  housing sites in partnership with a local food
  bank. The food bank provided food items,
  generally fresh produce, to program participants.
  Nutrition information and recipes about the
  donated food items were provided as part of the
  nutrition education session.

58
Program of Excellence, Project Impact, cont.

• Project Impact
• Having food available increased program
  participation and facilitated increased knowledge
  of local nutrition resources and reduced food
  insecurity.
• Nutrition education was provided to over 23,000
  lowincome seniors at meal sites using the new
  materials. Area Agencies on Aging included
  information from the curriculum into their
  newsletters that reached 122,500 lowincome
  seniors.

59
Program of Excellence, Project Impact, cont.

• Project Impact
• Followup questions would be about sustainability
  of the community organizing and changes the
  seniors were making in their own lives. If you
  had that additional information, should be a nice
  Program of Excellence example. Give enough
  information that someone else would be able to
  replicate, if desired.

60
Program Improvement

• Reach the target audience with relevant nutrition
  messages.
• Expand and enhance partnership base.
• Keep partners engaged in the process.
• Conduct meaningful, results based evaluation to
  achieve effective impact with participants and
  assure continued state and local support.

61
Topics for Future Research

• Calcium intake and ???
• Fortification and iron deficiency
• Obesity and food insecurity
• Other ???