Food and Nutrition Technical Assistance II Project FANTA2

1
What is Layers?
2
What is Layers?

• Layers is a monitoring system developed to help
  USAID Missions fulfill their responsibilities to
  monitor Food for Peace Act, Title II Programs.
• According to DCHA/FFP enabling regulation
• USAID Missions are expected to monitor CS's
  management of the commodities and use of grant
  funds. () Oversight and monitoring should
  include regularly scheduled visits to
  distribution centers and warehouses

3
How is Title II site monitoring usually done?

• How does USAID decide which site(s) to visit?
• When/how frequently do USAID staff visit?
• What does USAID look for during the site visit?
• What kinds of conclusions can USAID make from the
  site visits about the quality of CS programs?
• How does the site visit benefit the CS?

4
Common Approach to Title II Monitoring

• Title II sites are periodically visited by USAID
  staff to verify compliance on commodity storage,
  disposal and distribution.
• If discrepancies are found, the partner is
  notified.
• Some weaknesses of the typical approach
• Often only commodities are monitored, not program
  activities.
• Site selection is based on convenience, meaning
  there is no systematic sampling.
• Only the sites visited are assessed there is no
  program-wide monitoring.
• This monitoring system is designed for USAIDs
  use and little helpful feedback is given to
  partners.
• There is a lack of methodological guidance to
  field monitors on sampling and what to look for
  in a site visit.

5
Advantages of Layers

• Allows monitoring of program activities as well
  as commodity management
• Draws from a random sample of sites, which allows
  findings to be generalized to the entire program
• Standardized monitoring indicators
• Shares findings with partners to improve program
  performance

6
Lot Quality Assurance Sampling (LQAS)

• LQAS is a sampling method developed to control
  the quality of manufactured goods produced in
  lots. LQAS is a type of stratified random
  sample.
• LQAS takes a small random sample and tests the
  sample for quality.
• The sample will tell you if program activities
  (health, agriculture, etc.) are meeting a
  performance benchmark or not.
• The sample size is chosen so that there is a high
  probability of determining what indicators in a
  given activity are meeting or not meeting the
  performance benchmark.

7
What do Layers results tell us?

• Layers makes a determination about whether the
  MYAP meets the performance benchmark on each
  indicator.
• The determination is representative of the MYAPs
  performance at all of its sites for a given
  activity, not just those sites that were assessed
  during the survey.
• A MYAP may receive a No on some indicators and
  a Yes on others within the same activity type.
• Does the MYAP meet the performance benchmark for
  an indicator?
• YES - Meets the benchmark We assume the MYAP is
  performing adequately.
• NO - Does not meet the benchmark There is strong
  evidence that the MYAP is not performing
  adequately.

8
Limitations of Using LQAS in Layers

• Layers only shows whether a CS has met the
  benchmark for an indicator or not. It does not
  provide point estimates (i.e. of sites
  where.)
• LQAS is good at accurately identifying a CS that
  is above the upper benchmark or below the lower
  benchmark. Errors are more likely to occur with
  LQAS when a CS falls between the two benchmarks.
• The Yes/No result for indicators where the
  planned sample size was not met have higher error
  rates. In particular, the chance of erroneously
  concluding that the CS is performing below the
  benchmark increases, so these results should be
  treated with caution.

9
Statistical Analysis Under LQAS

• The math required for LQAS is done with
  easy-to-use calculators found on the FANTA-2
  website, at
• http//www.fantaproject.org/layers/reference.shtm
  l
• The calculator tells us
• How many sites to sample (e.g., 25)
• How many of those sites must receive a yes
  (e.g., 21) in order to meet the benchmark for
  that indicator (called the decision rule).
• Alpha and beta error rates based on sample size
  and upper and lower thresholds

10
(No Transcript)
11
Monitoring Indicators Used by Layers

• Using LQAS, Layers can monitor many aspects of a
  program to ensure the overall success of the CS
  in delivering services.
• Pre-existing indicators/questionnaires can be
  modified for each country. New indicators can be
  created.
• A wide array of indicators are possible.

12
Sample Layers Indicators
13
Layers Automated Data Entry Method

• Typical Survey
• Large number of paper questionnaires
• Requires data entry and cleaning
• Time-intensive
• Prone to error
• Layers
• Automated data entry
• Reduced data entry error
• Saves time and money
• PDAs require training and IT support

14
(No Transcript)
15
(No Transcript)
16
(No Transcript)
17
Layers Data Analysis

• Analysis with LQAS is simple
• Count the number of yes responses for each
  indicator.
• Apply the decision rule in each case.
• of Yes responses decision rule YES - meets
  benchmark
• of Yes responses lt decision rule NO - does
  not meet benchmark

18
Questions?
Comments?
Short Break?
19
An Illustrative Timeline for Installing Layers in
a new Country Assuming data collection is done in
one month by an external contractor
20
Layers Steps

• USAID and CSs
• Sampling
• Questionnaire adaptation
• USAID
• Enumerator and field supervisor training
• Data collection and syncing
• Generating results tables
• Communicating results to CSs

21
1. Initial Planning

• A. Internal Preparations
• Introduce Mission staff to Layers and assemble a
  team
• In-house vs. external implementation
• Budget and procurement
• Acquire technology (PDAs, software, central data
  storage, laptops)
• Scope of the survey (which activities will be
  assessed)
• B. Meet with CSs
• Present Layers
• Scope of the survey
• Ask for Total Universe Spreadsheet (TUS) (list of
  all activity sites)
• Discuss purpose and plan for follow-up Layers
  meetings

22
2. Sampling

• Collect TUSs from CSs
• Determine sample sizes and decision rules using
  the LQAS Sample Size Calculator
• Randomly select sites to be visited and
  replacement sites
• Create one complete list of all activity sites to
  be visited during Layers

23
3. Questionnaire Adaptation

• Existing questionnaires in MS Word and PPCC
• WM Warehouse Management
• FD Food Distribution
• MCHN Maternal and Child Health and Nutrition
• AG Agriculture
• FFW-RC Food for Work - Road (and Path)
  Construction and Rehabilitation
• FFW-RM Food for Work - Road (and Path)
  Maintenance
• FFW-DC Food for Work - Dams and Canals for
  Agriculture
• FFW-W Food for Work - Wells
• FFE Food for Education
• TN Tree Nurseries
• With CSs, review and adapt questionnaires as
  needed
• Write Field Manual
• Agree upon performance benchmarks
• Pre-test questionnaires

24
4. Enumerator and Supervisor Training

• A. Classroom
• Background on Title II programs in-country and
  purpose of Layers
• Conducting interviews/recording observations
  using the
• Layers questionnaires on the PDA
• Clarification of roles and responsibilities
  (field supervisors, IT Specialist, technical
  sector specialists, enumerators)
• Properly executing field procedures, including
  backing up
• B. Field Practice
• 2-3 sites per activity type (questionnaire)
• Practice PDAs, identify final corrections/revision
  s to questionnaire(s)

25
5. Data Collection and Syncing

• Enumerators and field supervisors collect data
• Survey Manager, field supervisors and IT
  Specialist review quality of data and ensure
  proper methods are followed for syncing and
  backing up

26
6. Generating Results Tables

• Export data from PPCC to MS Excel
• Copy comments to the Comments Template
• Reduce data to the key indicators
• Tabulate results
• Create the Layers Results Table

27
7. Communicating Results to CSs

• Internal meeting to discuss results
• Share results with CSs
• Ask CSs for an improvement plan
• Follow up on improvements
• Plan the next round of Layers

28
Sample LAYERS Report Letter

• 1. Findings for commodity warehouses
  (documentation, management and storage)
• Successes
• Routine documentation procedures at MCHN sites
  are followed appropriately.
• Storage management is appropriate storage sites
  are clean, weatherproof and safe and the food
  stored in those sites is kept correctly.
• Challenges
• The ledgers did not concur with existing
  inventories in many sites. This is most serious
  and needs urgent attention.
• Minor problems related to ventilation, rodent
  infestation and use of the warehouse for other
  purposes were detected in a few EHN sites.
• Letter ends with a Summary of Recommendations.

29
This presentation is made possible by the
generous support of the American people through
the support of the Office of Health, Infectious
Disease and Nutrition, Bureau for Global Health,
and the Office of Food for Peace, Bureau for
Democracy, Conflict and Humanitarian Assistance,
United States Agency for International
Development (USAID) under terms of Cooperative
Agreement No. GHN-A-00-08-00001-00, through the
Food and Nutrition Technical Assistance II
Project (FANTA-2), managed by the Academy for
Educational Development (AED). The contents are
the responsibility of AED and do not necessarily
reflect the views of USAID or the United States
Government.
Food and Nutrition Technical Assistance II
Project (FANTA-2) Academy for Educational
Development 1825 Connecticut Ave., NW
Washington, DC 20009 Tel [phone] Fax
202-884-8432 E-mail fanta_at_aed.org Website
www.fanta-2.org