Sampling Demystified

1
Sampling Demystified

• Research Methods for Public Administrators
• Dr. Gail Johnson

2
Steps in the Research Process

• Planning
• 1. Determining Your Questions
• 2. Identifying Your Measures and Measurement
  Strategy
• 3. Selecting a Research Design
• 4. Developing Your Data Collection Strategy
• Developing the Sampling Strategy
• 5. Identifying Your Analysis Strategy
• 6. Reviewing and Testing Your Plan

2
3
Why Sample?

• Sometimes it is possible to gather data from
  every file, every street, every person in the
  population of interest.
• When you can work with the entire population of
  files, streets, or peoplewe call it a census.
• Everyone is included because everyone counts

4
Why Sample?

• When you have the resources to gather data from
  the entire population thats the gold standard.
• Butthe world is not often organized in a way
  that makes this easy
• Often there is limited time, staff and money to
  gather data from the entire population.
• So researchers use samples

5
Why Sample?

• It requires a basic faith that the sample is a
  fairly accurate reflection of the population
• Think about it Doctors dont have to take all
  our blood in order to analyze it.
• They take a sample.

6
Two Big Sampling Options

• Nonrandom Samples
• Quota
• Accidental
• Snow-ball
• Judgmental
• Convenience
• Random Samples
• Based on probability theory where
• every item (person, transaction, street, house,
  whatever) has an equal chance of being selected

7
Option Nonrandom Sampling

• Used when it is not possible or desirable to do a
  random sample
• Can be more focused specifically chosen
• Select 6 executive directors 3 men and 3 women
• Select all the transactions from the busiest day
  of the month
• Can observe traffic through town during rush hour
  (430-6pm Monday through Friday).

8
Types of Nonrandom Samples

• Quota
• Set a specific number we will interview 10 men
  and 10 women at the health clinic
• We will call 50 Democrats and 50 Republicans
• First come basis and once quota is met, we stop
• If we reach 50 Republicans, we then continue to
  dial for Democrats until we obtain the 50 quota.

9
Types of Nonrandom Samples

• Accidentalperson on the street think
  Jay-walking
• While chaotic, it does not meet the definition of
  random
• If I survey people outside the big box store on a
  Saturday morning, not everyone in the community
  will have had an equal chance of being selected

10
Types of Nonrandom Samples

• Snow-ball
• This is useful when researchers really do not
  know who to include. So they start with the few
  they think have the information and then ask,
  who else should we talk to?
• Ideally, they continue until no new names are
  given

11
Types of Nonrandom Samples

• Judgmental (sometimes called a Purposive sample)
• Definite choices based on criteria that is
  meaningful given the situation
• I might decide to conduct focus groups with the
  heads of the largest nonprofits in my countyand
  I will specifically select them by name
• United Way, Community Services, Big Brother and
  Sister, the Food Bank, Affordable Housing
  Coalition, Interfaith Works, Early Learning
  Coalition, etc

12
Types of Nonrandom Samples

• Convenience
• I might send a link to a cyber survey to
  everyone in my social network (myspace, facebook,
  twitter) because it is easy but they do not
  represent the larger community
• I might survey everyone in my classes but they do
  not represent all the students in the school

13
Limitations of Nonrandom Samples

• Risk of bias
• Why were these people (files, streets, classes,
  whatever) selected but not others?
• Are they substantially different from the
  ones not selected?

14
The Inherent Caveat of Non-Random Samples

• The results of non-random samples cannot be
  generalized to the larger population.
• Results are always limited to
• Of the people who participated in the focus
  groups. or
• Of the three classes we observed.. or
• Of the 100 people we interviewed at the corner
  of walk and dont walk..

15
Non-Random Samples Can Be Useful Despite
Limitations

• In a study about teenaged mothers, non-random
  selection made sense
• We wanted a mix of ages and ethnic backgrounds
• Because we were selecting only a few teenaged
  mothers from each program to participate in focus
  groups, it was unlikely that we would have gotten
  the desired mix through random sampling.

16
Non-Random Samples

• Qualitative research
• Can yield very useful and important information
• Researchers should explain what they did, their
  rationale and the limitations of any conclusions
  based on this data
• If other research results are similar, it adds
  strength to their results

17
Option Random Sample

• A random sample means that each person (or item)
  has an equal chance of being selected
• Note this is not the same as random assignment
  we talked about in classic experiments
• Freshman in a psychology class may be randomly
  assigned to two different groups, but the results
  are not generalizable to all freshman, all
  college students, or all people

18
Random Sampling Three Benefits

1. It eliminates bias in selecting participants
2. It enables researchers to make estimates about
   the larger population based on what is learned
   from the sample (jargon term generalizability)
3. It enables the researchers to estimate sampling
   error (we will get to this in a minute)

19
Random Sampling The Challenges

• To locate a complete listing of the entire
  population from which to select a sample
  (sometimes called a complete enumeration)
• For example, there is no listing of all MPA
  students in the United States
• To select a large enough sample so the results
  will be statistically meaningful
• As a general rule, the larger the sample, the
  more resources are needed

20
Sample Concepts The Jargon

• Population
• the total set of units
• Census
• A complete count of the population
• Sample
• a subset of the population
• Sampling Frame
• list from which to select your sample

21
Sampling Concepts The Jargon

• Sample Design
• methods of sampling
• probability or non-probability
• Parameter
• characteristic of the population
• Statistic
• characteristic of a sample

22
Types of Random Samples

• Simple Random sample
• Stratified Random Sample
• Proportionate and Disproportionate
• Multi-Stage/Cluster Samples

23
Simple Random Sample

• A subset of the entire population
• Example
• A sample of all graduates of the teachers college
• A sample of all state employees

24
Simple Random Sampling Process

• Obtain a complete listing of the entire
  population
• Assign each case a number
• Randomly select the sample
• Given a population of 200 students, randomly
  select the first 25 whose numbers are between 001
  and 200 on a random numbers table
• See http//ts.nist.gov/WeightsAndMeasures/Publica
  tions/upload/h133_appenb.pdf

25
Mini-Random Number Table

• Using the 1st three digits, select all the
  numbers between 001 and 200

11164 12639 75061 00298
21215 99756 03024 04554
10431 00431 09532 48819
36002 12130 28060 07159
73941 87912 16936 35713
26
Plan B Systematic Sampling

• If a complete enumeration (list) is not
  available, use a systematic sample with a random
  start
• We randomly select the staring point, and then
  select every 20th or 50th file or street
  (whatever you population of interest)
• If you have 300 files in boxes, you might decide
  to randomly begin with the 19th file, and then
  select every 25th until you have a sample of 100
  files.

27
Random Sampling for Phone Surveys

• Why not use the phone book?
• Well, not everyone who has a phone is listed.
• HUD estimates that between 30-50 of city
  dwellers do not have listed phone numbers

28
Random Sampling For Phone Surveys

• To conduct telephone surveys, a computer randomly
  generates phone numbers with the appropriate area
  codes
• Jargon random digit dialing
• Efforts are made to include cell phones

29
Random Sampling For Phone Surveys

• Those without phones will be excluded, which is a
  limitation that might matter
• Researchers obtain a substantial number of phone
  numbers some wont be working, some might be fax
  numbers
• In a local community assessment, we obtained
  3,000 numbers in order to get a sample size of
  400 completed surveys
• You have to kiss a lot of frogs before you find
  the prince

30
More Complexity Stratified Random Sample

• What happens when one group is very small in the
  population and is therefore not likely to show in
  the sample in large enough numbers?
• A stratified random sampling process is used.

31
Process of Selecting a Stratified Random Sample

• Population is separated into strata (or groups)
• Each strata is randomly sampled
• Example Male and Female CEOs
• A simple random sample of men and a simple random
  sample of women are selected

32
Stratified Random Sample

• Ensures that we have enough men and women in each
  group to use statistical techniques, like tests
  for statistical significance (well get to this
  later)
• Stratified random samples tend to be larger than
  if a simple non-stratified random sample is used

33
Proportionate Stratified Sample

• The sample has the same percent distribution as
  the population

Gender Population Percent Sample Percent
Men 800 80 80 80
Women 200 20 20 20
Total 1,000 100
34
Disproportionate Stratified Sample

• The sample has a different percent distribution
  than the population

Gender Population Percent Sample Percent
Men 800 80 50 50
Women 200 20 50 50
Total 1,000 100
35
Disproportionate Stratified Sample

• This is used when one group (or strata) is so
  small that it will not yield statistically useful
  results
• The key point to remember is that when the
  researchers want to generalize back to all 1,000
  employees, they will need to weight the data so
  the proportions are back in line with population
• The weighted data would show the results where
  80 are men and 20 are women

36
No Listing, More ComplexityMulti-Stage Sampling

• Suppose we want to observe classroom activities
  to measure the amount of time spent on hands-on
  learning activities.
• Randomly select classrooms
• and then
• Randomly select days of the week and then
• Randomly select times of day
• Then observe all the children in those classes at
  those times.

37
More ComplexityCluster Samples

• Useful when you dont have a complete listing of
  the entire population.
• If you want to survey parents of primary school
  children in your country, you probably dont have
  a list.
• You will want to select a few primary schools and
  then select a random sample of students who can
  bring the survey home to their parents

38
Cluster Samples

• In war torn countries and natural disasters, a
  cluster sample approach is used to estimate the
  number of civilians who are killed.
• Researchers select a specific number of
  geographic areas and then randomly select
  streets, and then select a house as a starting
  point, and then select a set number of homes on
  that block

39
Combinations

• Random and non-random methods can be combined.
• Judgmental sample of schools
• Select 2 from the poorest communities and 2 from
  wealthiest communities.
• Then select a random sample of students.

40
How Might We Select a Random Sample to Measure
Traffic?

• We want to observe amount of traffic on the road
  from the village to major town.
• Randomly times of year?
• Randomly select times and days of week?
• Randomly select observation points or select a
  single observation point along the road?

41
Discussion

• You want to find out how likely it is that
  graduates of MPA programs in the U.S. will apply
  for jobs in the federal government as compared to
  consulting companies.
• Remember there is no complete list of all MPA
  students

42
Discussion

• What are the likely questions you would ask? What
  are the likely options for data collection?
• Which one do you think is the best given your
  circumstance? Why?
• Given you choice of data collection, how would
  you construct a random sampling plan?

43
Takeaway Lesson

• The decisions about whether to use a sampleand
  whether it should be random or nonrandomdepends
  on the situation.
• If it is possible to collect data from the
  population, that avoids concerns about selection
  bias and errors associated with sampling.
• Researchers should fully disclose their sampling
  procedures, their rationale, any problems in the
  process and the limitations.

44
Creative Commons

• This powerpoint is meant to be used and shared
  with attribution
• Please provide feedback
• If you make changes, please share freely and send
  me a copy of changes
• Johnsong62_at_gmail.com
• Visit www.creativecommons.org for more information