UAPP 702: Research Design for Urban

1
UAPP 702 Research Design for Urban Public
PolicyClass NotesBabbie, The Practice of Social
Research, Chaps.45

• Danilo Yanich
• School of Public Policy Administration
• Center for Community Research Service
• University of Delaware

2
Ch. 4 Research DesignPurposes of Research

• Exploration typically done for three purposes
• to satisfy the researchers curiosity and desire
  for better understanding
• to test the feasibility of undertaking a more
  extensive study
• to develop the methods to be employed in a
  subsequent study
• Description describe situations and events
• Census is good example of descriptive research
• Explanation the why? of events, situations,
  behavior, attitudes, etc.

3
Logic of Nomothetic Explanation

• Nomothetic explanation refers to the accounting
  of many variations in a given phenomenon
• In contrast to
• Idiographic explanation that seeks an in-depth
  understanding of a single case

4
Criteria for Nomothetic Causality

• Correlation the variables must be correlated
• Time order the cause takes place before the
  effect
• Non-spurious the variables are non-spurious
• Spurious relationship a coincidental statistical
  correlation between two variables, shown to be
  caused by some third variable

5
Correlation

• Some relationship---or correlationbetween the
  variables must exist before we can consider
  causality
• Correlation empirical relationship between two
  variables such that
• Changes in one are associated with changes in the
  other
• Particular attributes of one variable are
  associated with particular attributes of the other

6
False Criteria for Nomothetic Causality

• Complete causation
• Causation is incomplete and probabalistic
• Exceptional cases
• Exceptional cases do not disprove general overall
  pattern of causation
• Majority of cases
• Causal relationship may be true even if they
  dont apply to the majority of cases
• Example lack of supervision delinquency
• as long as unsupervised juveniles are more likely
  to be become delinquent, social science can say
  there is a causal relationship

7
Necessary and Sufficient Causes

• Necessary cause represents a condition that must
  be present for the effect to follow
• Ex must be female to become pregnant
• Ex must take college courses to get a
  degreebut
• Simply taking courses is not a sufficient cause
• Must take the right ones

8
Necessary and Sufficient Causes

• Sufficient cause represents a condition that, if
  it is present, guarantees the effect in question
• Not saying that sufficient cause is only possible
  cause for effect
• Ex skipping exam in course would be sufficient
  cause for failing, but students could fail in
  other ways, too
• So, cause can be sufficient but not necessary

9
Units of Analysis

• No limit to what or whom can be studied
• Common social science units of analysis
• Individuals
• Groups
• Organizations
• Social artifacts.
• Important what you call a given unit of
  analysis is almost irrelevantbut you must be
  clear what that unit is
• Are you studying marriages or marriage partners?
• Crimes or criminals?
• Historic buildings or the process for selecting
  them?
• Efficiency of the hotel or the satisfaction of
  customers?

10
Ecological Fallacy

• Ecological in this context refers to groups or
  sets or systems, something larger than
  individuals.
• Fallacy is to assume that something learned about
  such a unit says something about the individuals
  comprising that unit.
• Babbie uses example of data that shows which
  precincts supported a female candidate
• Some census data for each precinct that shows
  that precincts with relatively young voters gave
  her more support
• Could not assume that young voters were most
  likely to support a female candidate...
• That iswe cannot assume that age affects support
• The unit of analysis was the precinct, NOT the
  individuals in the precinct

11
Reductionism

• Tendency to explain everything in terms of a
  particular, narrow set of concepts
• Remember paradigms that predispose researcher to
  a particular explanation
• Definition of order by coercion, shared values,
  exchange

12
Ch. 5 Conceptualization, Operationalization
Measurement

• Conceptualization
• The refinement and specification of abstract
  concepts
• A specific agreed-upon meaning of the concept
  under study
• Ex. compassion does not exist in any sense that
  we can measure in an objective sense
• Operationalization
• The development of specific research procedures
  (operations) that will result in empirical
  observations representing those concepts in the
  real world

13
What social scientists measureTable 5.1, p. 129
Examples
Direct observables Physical characteristics of a person being observed/interviewed
Indirect observables Characteristics of a person as indicated by answers given in a self-administered questionnaire
Constructs Level of alienation, as measured by a scale that is created combining several direct and/or indirect observables
14
Indicators and Dimensions

• Indicator
• An observation that we consider as a reflection
  of the variable under study
• Ex attending church as an indicator or
  religiosity
• Dimension
• A specific aspect of a concept
• Ex action aspects of religiosity (attending
  church, giving money) and contemplative aspects
  (prayer, etc)

15
Operational definition

• Specifies precisely how a concept will be
  measured
• Operationalization
• The development of specific research procedures
  (operations) that will result in empirical
  observations representing those concepts in the
  real world

16
Progression of measurement steps

• Conceptualization
• ?
• Nominal definition
• ?
• Operational definition
• ?
• Measurements in the real world

conceptual funnel
17
Progression of measurementTable 5.2, p. 136
Measurement step Example social class
Conceptualization What are the different meanings and dimensions of the concept social class?
Nominal definition For our study, we will define social class as representing economic difference specifically, income
Operational definition We will measure economic differences via responses to the survey question What was your annual income, before taxes, last year?
Measurements in the real world The interviewer will ask What was your annual income, before taxes, last year?
18
Operationalization Choices

• Range of variation Must be clear about the range
  of variation in any concept that interests you.
• Babbie uses as an example studying certain ranges
  of income, i.e., using 100,000 as the floor for
  the highest income group rather than a higher
  amount
• Attitudes toward nuclear power...might use a
  range of favor it very much to dont favor it
  at all...
• But, that would leave out the people who are
  opposed to it.
• Variations between extremes Get as much detail
  in the measurement as possible.
• Can always aggregate data (that is, combine
  precise attributes) into more general
  categories...
• But can never separate out any variations that
  were lumped together during observation and
  measurement.

19
Two important qualities of variables Exhaustive
Mutually Exclusive

• Exhaustive For the variable to have any utility
  in research, must be able to classify every
  observation in terms of one of the attributes
  composing the variable
• Babbie uses example of political party
  affiliation that specifies just Democrat or
  Republican
• When that would leave out others who do not
  identify with either
• Use other or no affiliation to make it
  exhaustive.
• Mutually exclusive Must be able to classify
  every observation in terms of one and only one
  attribute.
• Babbie uses defining employed and unemployed in
  such a way that nobody can be both at the same
  time
• Refer to Graber social type variable...farmer,
  ner-do-well, etc. Family Court gender
  variable.

20
Levels of measurement (NOIR)

• Nominal variables whose attribute have only the
  characteristics of exhaustiveness and mutual
  exclusivity
• Examples gender, religious affiliation,
  birthplace, etc
• Ordinal variables with attributes that can
  logically rank-order the different attributes
  represent relatively more or less of a variable.
• Examples social class, conservatism, alienation,
  prejudice, coolness
• Interval variables in which the actual distance
  separating them can be expressed in meaningful
  standard variables
• Examples temperature, intelligence tests
• Ratio variables that have all of the
  characteristics of the previous levels of
  measurement AND are based on a true zero point
• Examples age, length of residence in a home,
  duration of news story, etc.

21
Implications of levels of measurement

• Requirements of analytical techniques
• Certain analytical techniques require variables
  that meet certain minimum levels of measurement
• Must plan analytical techniques according to the
  level of measurement at which you will gather
  your data.
• Should anticipate drawing research conclusions
  appropriate to the levels of measurement used in
  your variables.
• Caution Seek highest level of measurement
  possible because...
• Although you can reduce a ratio measure to
  ordinal...
• You cannot convert an ordinal measure into a
  ratio measure...
• It is a one-way street

22
Criteria of measurement quality

• Precision and accuracy
• Precisionfineness of the distinction made
  between the attributes that compose a variable
• Saying that a woman is 43 years old is more
  precise than saying that she is in her forties
• Degree of precision is dictated by your research
  requirements
• If your research question does not require her
  precise age, then additional effort to gather it
  precisely is wasted
• However, if your needs are unclear, be more
  precise rather than less
• Do not confuse precision with accuracy
• Saying that someone was born in Stowe, VT is
  more precise than born in New England
• Butsuppose the person in question was born in
  Boston
• The more general description of New England is
  less precise, but accurate

23
Criteria of measurement quality, p.2

• Reliability
• Whether a particular technique, applied
  repeatedly to the same object, yields the same
  result every time
• Example Measuring weight using two different
  persons estimates versus a scale
• Reliability does NOT ensure accuracy
• Suppose the scale is set five pounds too light
• Measurement would be reliable each time, but it
  would also be wrong each time
• Ways to cross-check the reliability of measures
• Test-retest method
• Split-half method
• Using established measures (Miller book is useful
  here)
• Reliability of research workers

24
Criteria of measurement quality, p.3

• Validity
• Refers to the extent to which an empirical
  measure adequately reflects the real meaning of
  the concept under consideration
• Social research does operate on agreements about
  the terms we use and the concepts they represent

25
Criteria of measurement quality, p.4

• Testing validity
• Face validity empirical measures that jibe with
  our common understanding of a concept
• Ex. Grievances worker morale
• Criterion-based validity based on external
  criterion
• Ex. College board scores student success in
  college

26
Criteria of measurement quality, p.5

• Testing validity
• Construct validity based on logical
  relationships among variables
• Ex. Marital fidelity marital satisfaction
• Content validity refers to how much a measure
  covers the range of meanings in a concept
• Ex test of math ability cant be limited to
  addition alone

27
Criteria of measurement quality, p.6

• Tension between reliability validity
• Often a trade-off between the two because
  resources limit the research
• Ex. Measuring morale by spending days on assembly
  line talking w/ workers seems a more valid
  measure of morale than counting grievances
• If there is no clear agreement on how to measure
  a conceptmeasure it several ways
• Ex. Recidivism, court success, hotel efficiency,
  etc.
• Concept does not have any meaning other than what
  we give it.
• Only justification to give concept a particular
  meaning is utility

28
Basic Research Outline

• The Social Problem
• Present a clear, brief statement of the problem,
  with concepts defined where necessary
• Show that the problem is limited to bounds
  amenable to treatment or test
• Describe the significance of the problem with
  reference to specific criteria

Source Miller, Delbert C. 1991. Handbook of
Research Design and Social Measurement, 5th
Edition. Newbury Park Sage Publications, pp.
15-16.
29
Basic Research Outline, p.2

• The Theoretical Framework
• Describe the relationship of the problem to a
  theoretical framework
• Demonstrate the relationship of the problem to
  previous research
• Present alternate hypotheses considered feasible
  within the framework of the theory.

30
Basic Research Outline, p.3

• The Research Question/Hypotheses
• Clearly state the research questions or the
  hypotheses selected for test. (Null and
  alternate)
• Indicate the significance of test hypotheses to
  the advancement of research and theory.
• For policy research state how research might
  inform policy.
• Define concepts or variables (preferably in
  operational terms).
• Describe possible mistakes and their
  consequences.
• Note seriousness of possible mistakes.

31
Basic Research Outline, p.4

• Design of the Experiment or Inquiry
• Describe ideal design or designs with particular
  attention to the control of interfering variables
• Describe selected operational design
• Specify statistical tests including dummy
  variables

32
Basic Research Outline, p.5

• Sampling Procedures
• Describe experimental and control samples
• Specify method of drawing or selecting sample

33
Basic Research Outline, p.6

• Methods of Gathering Data
• Describe measures of quantitative variables
  showing reliability and validity when these are
  known. Describe means of identifying qualitative
  variables
• Include descriptions of questionnaires or
  schedules
• Describe interview procedure
• Describe use made of pilot study, pretest, trial
  run.

34
Basic Research Outline, p.7

• Working Guide
• Prepare working guide with time and budget
  estimates
• Estimate total person-hours and cost

35
Basic Research Outline, p.8

• Analysis of Results
• Specify methods of analysis

36
Basic Research Outline, p.9

• Interpretation of Results
• Discuss how conclusions will be fed back into
  theoryOR
• Inform policy/practice.

37
Basic Research Outline, p.10

• Publication or Reporting Plans...Communication
  Plans
• Monograph, Executive summary
• Testimony to policy makers.
• Presentations to institutions, non-governmental
  agencies, media, public.
• Journal publication

38
The Policy Research Process
Present a clear, brief statement of the problem, with concepts defined where necessary. Over half of the criminal cases in Delaware exceed the Supreme Courts standard for the time from arrest to disposition (plea, verdict, etc.).
Show that the problem is limited to bounds amenable to treatment or test. An analysis of the period from arrest to disposition of criminal cases in Delawares Superior Court during a randomly chosen calendar year will provide the required data to examine the issue.
D. Yanich example using model in Miller,
Delbert C. (1991). Handbook of Research Design
and Social Measurement. Fifth Edition. Newbury
Park, CA Sage Publications, pp15-16
39
The Policy Research Process, p.2
Criterion Comment
Timely The Constitution requires that justice is delivered in a timely manner. To the extent that Delaware is not in compliance with its own 120 standard, it jeopardizes that requirement.
Practical problem The costs, the ethics, the legal liability for operating a system in violation of its own standard.
Wide population All citizens bear the cost of a dysfunctional court system, whether in taxes or large policy choices.
Influential/Critical population Main audience for the research is the Delaware General Assembly and the agents of the court.
Research gap Never has been a comprehensive look at the case processing in Delaware
40
The Policy Research Process, p.3
Criterion Comment
Generalizations Can NOT generalize to populations (court systems) beyond Delaware.
Sharpens concept Offers a more detailed examination of case processing through critical phases
Practical implications Practice and policy will change as a result of the research.
Improve data analysis instrument The courts never had a data-gathering instrument to understand case processing. The research will provide a base-line.
Data gathering constrained by time One calendar year is precisely geared to acquire the critical data within a manageable time period.
Fruitful exploration The research extends the analysis of court processing.
41
The Policy Research Process, p.4
Clearly state the research question/hypotheses selected for test. Null hypothesis There is no difference between the cases that are disposed within 120-day mandate and those that exceed it. Research hypothesis Differences exist between the cases that comply and do not comply with the 120-day mandate along case and courts culture dimensions.
Indicate the significance of test hypotheses to the advancement of research and theory. For policy research state how research might inform policy. A systematic examination of the case processing activity of Delawares Superior Court will give policy-makers a baseline from which to make changes in the courts policy and practice.
42
The Policy Research Process, p.5
Define concepts or variables (preferably in operational terms). Contained in coding instructions in which all variables are operationalized Examples unit of analysiscase instant offensecrime for which case is prosecuted criminal historynumber of previous convictions
Describe possible mistakes and their consequences. Possible mistakes focus on validity and reliability issues.
Note seriousness of possible mistakes. Validity or reliability mistakes are fatal to the research process.