Evaluation in Practice: A Methodological Approach Chapter 3 Measurement

1
Evaluation in PracticeA Methodological
ApproachChapter 3 Measurement

• Christopher Stream, PhD

2
Introduction

• Program Evaluation is an empirical exercise
• Choice of programs is subjective
• Methods used to examine programs is not
• Clearly, measurement is an important element of
  empirical inquiry

3
Conceptualization the Process of
Operationalization

• Concepts are the abstract/underlying properties
  of variables
• The more abstract the concept, the more difficult
  it is to measure.
• In evaluation research, some concepts are easy to
  measure, some are not.
• Examples?

4
Conceptualization the Process of
Operationalization

• Once the concept has been identified, the first
  step in measurement is to conceptually define it.
  
• This brings us to Operational Definition.
• We operationalize a concept we define how it is
  seen in reality.

5
Conceptualization the Process of
Operationalization

• When you operationalize a concept, you name the
  variable that you will measure.
• Measurement refers to the rules for assigning
  numeric value to a variable
• Fred Kerlinger quote on page 32

6
Conceptualization the Process of
Operationalization

• In order to specify the rules, we need to
  correctly identify the unit of analysis.
• This is the object of the observation measured at
  an aggregated level.
• e.g. individual persons, neighborhoods, cities,
  states
• The variable must be both conceptually and
  operationally measured at the same level of
  aggregation or unit of analysis
• What does this mean? Examples?

7
Conceptualization the Process of
Operationalization

• Measurement deals with rules rules concerning
  how to construct good measures.
• All measures must have categories which are
  mutually exclusive and collectively exhaustive.
• What does mutual exclusivity mean?
• What does collectively exhaustive mean?

8
Conceptualization the Process of
Operationalization

• A final aspect of measurement concerns the
  ecological fallacy.
• This involves asserting individual level behavior
  from aggregate level measures.
• Example of this?

9
Levels of Measurement

• Levels of measurement explain how data are
  measured categorically or continuously.
• Categorical data are measured in a group (less
  than 2 years, 2 to 5 years).
• Nominal In nominal measurement the numerical
  values just "name the attribute uniquely. No
  ordering of the cases is implied. categorically
  or continuously. (gender, region, country)
• Ordinal categories with implied scales and you
  can rank order (military rank, satisfaction
  levels)
• Continuous data are measured with specific
  numbers (2 years or 5 years).
• Interval variables in a numeric scale without a
  true zero interval measurement the distance
  between attributes does have meaning
  (temperature, achievement test scores)
• Ratio In ratio measurement there is always an
  absolute zero that is meaningful. This means that
  you can construct a meaningful fraction (or
  ratio) with a ratio variable (annual income,
  vacation)

10
Levels of Measurement
11
Special Case of Dummy Variables

• Dummy variables designate the presence or absence
  of a characteristic.
• They are often used as interval level variables
  in regression.
• Dummy variables are coded with a 1 indicating
  the presence of an attribute and a 0 for the
  absence.
• Examples?

12
Special Case of Dummy Variables

• What if the nominal variable of interest has more
  than two categories?
• See pages 35-36 for example
• Rule of thumb in the absence of a compelling
  theoretical choice, leave out the potential dummy
  variable with the smallest number of cases.

13
Validity and Reliability of Measures

• The issues of validity and reliability of
  measures is important
• Because we tend to measure concepts that are not
  easily observed in reality

14
Validity

• Definition Validity is the strength of our
  conclusions, inferences or propositions. More
  formally, Cook and Campbell (1979) define it as
  the "best available approximation to the truth or
  falsity of a given inference, proposition or
  conclusion."
• In short, were we right? Let's look at a simple
  example. Say we are studying the effect of strict
  attendance policies on class participation.
• In our case, we saw that class participation did
  increase after the policy was established. Each
  type of validity would highlight a different
  aspect of the relationship between our treatment
  (strict attendance policy) and our observed
  outcome (increased class participation).

15
Validity

• Types of ValidityThere are four types of
  validity commonly examined in social research.
• 1. Conclusion validity asks is there a
  relationship between the program and the observed
  outcome?
• Or, in our example, is there a connection between
  the attendance policy and the increased
  participation we saw?
• 2. Internal Validity asks if there is a
  relationship between the program and the outcome
  we saw, is it a causal relationship?
• For example, did the attendance policy cause
  class participation to increase?

16
Validity

• 3. Construct validity is the hardest to
  understand in my opinion. It asks if there is
  there a relationship between how I
  operationalized my concepts in this study to the
  actual causal relationship I'm trying to study/?
• Or in our example, did our treatment (attendance
  policy) reflect the construct of attendance, and
  did our measured outcome - increased class
  participation - reflect the construct of
  participation?
• Overall, we are trying to generalize our
  conceptualized treatment and outcomes to broader
  constructs of the same concepts.
• 4. External validity refers to our ability to
  generalize the results of our study to other
  settings.
• In our example, could we generalize our results
  to other classrooms?

17
Reliability

• Definition Reliability is the consistency of
  your measurement, or the degree to which an
  instrument measures the same way each time it is
  used under the same condition with the same
  subjects.
• In short, it is the repeatability of your
  measurement. A measure is considered reliable if
  a person's score on the same test given twice is
  similar. It is important to remember that
  reliability is not measured, it is estimated.
• There are two ways that reliability is usually
  estimated test/retest and Equivalent Forms.

18
Conclusion

• Measurement is a key aspect of program
  evaluation.
• It involves defining concepts of interest,
  organizing rules for the assignment of numbers to
  the concept (make it empirical) and ensuring the
  validity and reliability of the measures.

19
Conclusion

• As you read the examples of program evaluations
  in this book, pay attention to some of the
  creative ways the researchers have measured
  outcomes and carefully reported evidence
  concerning the validity and reliability of their
  measures.
• Measurement is an art and a science.