Title: The Theory of Sampling and Measurement
1The Theory of Sampling and Measurement
2Sampling
- First step in implementing any research design is
to create a sample. - We cannot study the theoretical population of all
conceivable events (e.g., events that have not
occurred), nor can we usually study all instances
of actual events. We select some instances to
study and not others. Those we include are our
sample. - How our sample is selected is critical for
external validity or generalizability.
3Groups in Sampling
Who do you want to generalize to?
4Groups in Sampling
The theoretical population
5Groups in Sampling
The theoretical population
What population can you get access to?
6Groups in Sampling
The Theoretical Population
The study population
7Groups in Sampling
The theoretical population
The study population
How can you get access to them?
8Groups in Sampling
The theoretical population
The study population
The sampling frame
9Groups in Sampling
The theoretical population
The study population
The sampling frame
Who is in your study?
10Groups in Sampling
The theoretical population
The study population
The sampling frame
The sample
11Types of Samples
- Probability Sampling
- Simple random
- Stratified random
- Cluster or area random
- Non-Probability Sampling
- Accidental
- Modal instance
- Expert
- Snowball
- Case study (intentional selection)
12The Sampling Distribution
Average
Average
Average
...is the distribution of a statistic across an
infinite number of samples.
The sampling distribution...
13Population Parameter
The population has a mean of 3.75...
1
5
0
...and a standard unit of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
This means
Self esteem
About 64 of cases fall between 3.5 - 4.0.
About 95 of cases fall between 3.25 - 4.25.
about 99 of cases fall between 3.0 - 4.5
14Sampling Distribution
The population has a mean of 3.75.
1
5
0
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
Self-esteem
15Sampling Distribution
The population has a mean of 3.75...
1
5
0
...and a standard error of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
Self-esteem
16Inferring Population from Sample
The sample has a mean of 3.75...
1
5
0
...and a standard deviation of .25.
1
0
0
Frequency
5
0
0
4
.
5
4
.
0
3
.
5
3
.
0
This means
Self esteem
64 chance true population mean falls between 3.5
- 4.0.
95 chance true population mean falls between
3.25 - 4.25.
99 chance true population mean falls between 3.0
- 4.5
17Figure 3.4 Labor Repression and Growth in the
Asian Cases, 1970-1981
18Figure 3.5 Labor Repression and Growth in the
Full Universe of Developing Countries,1970-1981
19Measurement
- Operationalization is the process of translating
theoretical constructs into observable
indicators. - Construct validity and reliability are the
criteria we use to evaluate how well you have
operationalized your concepts. - Both matter regardless of the level of
measurement and whether you are using qualitative
or quantitative indicators.
20The Hierarchy of Levels
Ratio
Absolute zero
Interval
Distance is meaningful
Ordinal
Attributes can be ordered
Nominal
Attributes are only named weakest
21Nominal Measurement
- The values name the attribute uniquely.
- The name does not imply any ordering of the cases.
22Ordinal Measurement
- When attributes can be rank-ordered
- Distances between attributes do not have any
meaning.
23Interval Measurement
- When distance between attributes has meaning, for
example, temperature (in Fahrenheit) -- distance
from 30-40F is same as distance from 70-80F - Note that ratios dont make any sense -- 80F is
not twice as hot as 40F.
24Ratio Measurement
- Has an absolute zero that is meaningful
- Can construct a meaningful ratio (fraction), for
example, number of clients in past six months
25Construct Validity
- Key problem is that we have abstract theoretical
construct power, democracy, development,
corruption, etc. that we can never observe
directly. - Yet, to test propositions requires that we have
some indicator for the construct or at least
have proxies that we can argue are capturing some
attributes of the construct. - Our indicator is an analogy (to an analogy).
26Assessing Construct Validity
- Translation Validity
- Face Validity plausible on its face
- Content Validity matches lists of attributes
- Criterion-related Validity
- Predictive Validity predicts accurately
- Concurrent Validity distinguishes appropriately
between groups - Convergent Validity
- Discriminant Validity
27The Convergent Principle
- Alternative measures of a construct should be
strongly correlated.
28How It Works
Theory
You theorize that the items all reflect
self-esteem.
29How It Works
Theory
1.00 .83 .89 .91 .83 1.00 .85 .90 .89 .85 1.00
.86 .91 .90 .86 1.00
The correlations provide evidence that the items
all converge on the same construct.
Observation
30Convergent Validity in Measures of Democracy
- 1985 polity2 pollib civlib
reg - -------------------------------------------------
- polity2 1.0000 -0.9148 -0.8770
-0.8601 - pollib -0.9148 1.0000 0.9176
0.8440 - civlib -0.8770 0.9176 1.0000
0.8053 - reg -0.8601 0.8440 0.8053
1.0000
31Convergent Validity in Measures of Education
- 1985 1 2 3 4 5 6
- -------------------------------------------------
------------------ - Ed. spending 1.0000 -0.1217 0.2415
0.3563 0.0214 0.0195 - Illiteracy () -0.1217 1.0000 -0.5797
-0.7306 -0.8569 -0.6196 - Cohort to Grade 4 0.2415 -0.5797 1.0000
0.4419 0.6553 0.3654 - Grade School 0.3563 -0.7306 0.4419
1.0000 0.6230 0.3612 - Secondary School 0.0214 -0.8569 0.6553
0.6230 1.0000 0.7576 - College 0.0195 -0.6196 0.3654
0.3612 0.7576 1.0000
32The Discriminant Principle
- Measures of different constructs should not
correlate highly with each other.
33How It Works
Theory
Locus-of-control construct
LOC1
LOC2
34How It Works
You theorize that you have two distinguishable
constructs.
Theory
Locus-of-control construct
LOC1
LOC2
35How It Works
Theory
Locus-of-control construct
LOC1
LOC2
rSE1, LOC1 .12
The correlations provide evidence that the items
on the two tests discriminate.
rSE1, LOC2 .09
rSE2, LOC1 .04
rSE2, LOC2 .11
Observation
36We have two constructs. We want to measure
self-esteem and locus of control.
Theory
Self-esteem construct
Locus-of-control construct
SE1
SE2
SE3
LOC1
LOC2
LOC3
For each construct, we develop three scale items
our theory is that items within the construct
will converge and Items across constructs will
discriminate.
37Theory
Self-esteem Construct
Locus-of-control construct
Green and red correlations are Convergent yellow
are Discriminant.
SE1
SE2
SE3
LOC1
LOC2
LOC3
Observation
38Theory
Self-esteem construct
Locus-of-control construct
SE1
SE2
SE3
LOC1
LOC2
LOC3
The correlations support both convergence and
discrimination, and therefore construct validity.
Observation
39What Is Reliability?
- The repeatability of a measure
- The consistency of a measure
- The dependability of a measure
40True Score Theory
Observed score
True ability
Random error
e
X
T
41The Error Component
e
X
T
Two components
er
es
42The Revised True Score Model
er
X
es
T
43Random Error
Frequency
The distribution of X with no random error
X
44Random Error
The distribution of X with random error
Frequency
The distribution of X with no random error
Notice that random error doesnt affect the
average, only the variability around the average.
X
45Systematic Error
Frequency
The distribution of X with no systematic error
X
46Systematic Error
The distribution of X with systematic error
Frequency
The distribution of X with no systematic error
Notice that systematic error does affect the
average we call this a bias.
X
47If a Measure Is Reliable...
We should see that a persons score on the same
test given twice is similar (assuming the trait
being measured isnt changing).
X1
X2
48If a Measure Is Reliable...
But, if the scores are similar, why are they
similar?
X1
X2
Recall from true score theory that...
T e1
T e2
49If a Measure Is Reliable...
The only thing common to the two measures is the
true score, T. Therefore, the true score must
determine the reliability.
X1
X2
T e1
T e2
50Reliability Is...
a ratio
variance of the true scores
variance of the measure
var(T)
var(X)
51Reliability Is...
a ratio
variance of the true scores
variance of the measure
We can measure the variance of the observed
score, X. The greater the variance, the less
reliable the measure.
52This Leads Us to...
- We cannot calculate reliability exactly we can
only estimate it. - Each estimate attempts to capture the
consequences of the true score in different ways.
53We want both Reliability and Validity
54Reliability and Validity
Reliable but not valid
55Reliability and Validity
Valid but not reliable
56Reliability and Validity
Neither reliable nor valid
57Reliability and Validity
Reliable and valid
58Assignment 1
- Assess the validity and reliability of the IRIS-3
International Country Risk Guide. - Can examine a single instance, compare instances,
analyze the full variation in the dataset,
compare with additional measures, or use any
other form of assessment. May use outside sources
of data, history, or analysis (but document). - The only restriction is that the paper must be
empirical and examine issues of validity and
reliability. - 3-5 pages. Be concise.
- Due Monday 10/24 at beginning of class.