Objectives of Session Three

1
Objectives of Session Three

• Focus Group and Survey Assignment
• Formative Design Proposal
• Construct Validity
• Reliability in Data
• In-class Data Exercise

2
Focus Group/Survey Project

• Select topic and moderator(s)
• ½ hour no penalty if you do not finish your
  guide
• Short ice-breaker, background on topic, and
  questions
• Be prepared with follow-up questions
• Handout relating to survey design

3
Sources for Formative Design Proposal

• Conduct an internet search using keywords for
  your program area of interest and evaluation
  terms
• Points of Light Foundation - http//www.pointsofli
  ght.org/
• United Way - http//national.unitedway.org/
• Electronic Policy Network - http//movingideas.org
  /
• Evaluation Center - http//www.wmich.edu/evalctr/i
  ndex.html
• Michigan Non-profit link - http//comnet.org/index
  .html
• Community Foundation of CNY - http//www.cnycf.org
  /nonprofit/grantees.cfm

4
Construct Validity

• Does the measure capture the construct of
  interest? How well does the measure capture the
  process at hand?
• -Face validity
• -Content validity
• -Convergent v. Discriminant validity

5
Achieving Construct Validity

• Propose different possibilities for valid
  measures or operationalizations of the following
  concepts
• Quality of an individuals diet
• Customer satisfaction with visit to the Dept. of
  Motor Vehicles
• Racial or ethnic identity
• Physical limitations

6
Reliability

• Free of error
• Consistency in measurement and in coding
• Data cannot be valid unless it is reliable, but
  reliable data can be invalid.

7
Internal and External Validity

• Internal Validity Does the design allow us to
  reach causal conclusions?
• External Validity Are your findings
  generalizable?

8
Improving Validity and Reliability of Data

• Pretest your questions or data collection
  techniques
• Use widely trusted measures
• Correlations between coders/respondents
• Develop multiple measures
• Use qualitative findings to check quantitative
  findings and vice versa

9
In-Class Data Exercise

• Data collected from Open Cupboard, nonprofit that
  opened 10 food pantries 6 months ago
• What can the data tell us about service delivery?
  
• Different measures of initial implementation and
  service delivery -- no impact or outcomes
• Frequencies
• Means and variance
• Correlation
• Which measures are most valid? Most reliable?

10
Readings for Next TimeWeiss, Chapters 4 and 7