Choosing a Research Question

1
Choosing a Research Question

• Week 2
• 311
• 2001

2
The Research Process

• Identify Research Question

Choose Method
Experiment
Quasi-experiment Ex post facto
Locate needed instrumentation
Develop Instrumentation
Develop data analytic procedures
Choose data analytic strategy
Implement Study
Communicate Findings
3
Type of Research
Where does your research question fit in?
4
Developing a research Question

• Questions are the engine that drive the research
  enterprise
• method, sample, settings, variables all follow.
• Many papers get rejected because they do not have
  a discernible question.
• An assessment of the questions importance also
  determines publication - more than methodology or
  presentation style.

5
Choice of Question

• Internal Factors
• Curiosity - I wonder what would happen if?
• Compassion -
• Confirmability - easy replication
• Conformity - fad
• Predilection - skill, temperament, contacts or
  training

6
Choice of Question

• External Factors
• a question determined by someone other than the
  researcher (e.g. sponsor)
• cost - the data were cheap and easy to obtain
  (e.g., AWIRS)
• resource availability (opportunism)
• reward system - certain types of research are
  rewarded
• propinquity - what are the interests of potential
  collaborators

7
Significance comes from ...

• Activity Contacts with colleagues and
  organisations rather than thinking in isolation
• Intrinsic interest and motivation excitement and
  commitment

8
Lack of significance ...

• Expedience cheap, quick and easy
• Lack of interest in the question motivated by
  external outcome

9
Where do questions come from?

• Theory based questions (13) (highly regarded)
• tests of predictions
• comparison of theories
• Applied problem (3) (not highly regarded)
• problems defined by and limited to a single
  organisation or tightly defined problem
• Build on existing literature (84) (not highly
  regarded)
• use different subject population
• different operationalisation of 1 or more
  variables
• include different levels of a variable studied
  previously
• simultaneously study variables that have only
  been examined independently
• include moderators or mediators
• add an additional variable

10
The form of the question

• the description of a single variable
• operationalisation of a construct and aspects of
  its distribution
• (not as common as it should be!)
• the interrelationships between variables
• is there an association between variables?
• is there a causal relationship
• under what conditions are they associated and in
  what way (moderator, or mediator)
• does variable Z add to the prediction of y after
  x has been extracted?

11
Choice of methodology

• Often the research question dictates the method
• is it true that 75 of employees are dishonest?
  (epidemiology)
• I believe that dishonesty is determined by peer
  group pressure - are there high rates of
  dishonesty among certain types of employees?
  (still lacks explanation)

12
Choice of methodology2 variables

• Are employees of low moral fibre likely to be
  dishonest?
• introduces 2 variables if the variables are
  measured simultaneously there is little other
  than correlations can be done unless one can
  induce low/high moral fibre
• experimental methods (even quasi) of some kind
  are needed to infer causality

13
Choice of methodology 3 variables

• Do peers who vary in moral fibre influence
  dishonesty amongst individuals who vary in moral
  fibre
• this introduces a whole new set of issues that
  dictates methods and analysis
• does peer level of moral fibre influence
  dishonesty independently of individual level of
  moral fibre
• does peer level of moral fibre influence
  individual moral fibre which in turn influences
  dishonesty (mediation)
• is the strength of the moral fibre-dishonesty
  relationship differentially influenced by level
  of peer moral fibre (moderation)
• how well can dishonesty be predicted from moral
  fibre of individuals and peers (additive)
• Such questions influence the decision on methods
  to observe or manipulate, to measure once or
  repeatedly

14
The unit of analysis

• Should we look at the individual level or at the
  unit level, (faculty, department, university) or
  at multiple levels
• why are there differences among departments?
• are all the departments within a company the
  same?
• Ecological fallacy (Robinson 1950)
• Simpsons paradox (Simpson, 1951)

15
Two Fallacies

• Ecological Fallacy drawing conclusions about
  individual people from data that refer only to
  aggregates.
• For instance, assume that you measured
  satisfaction scores of a particular shopfloor and
  found that they had the highest average score in
  the factory. Later you run into one of the
  employees from that group and you think to
  yourself "she must be satisfied." Just because
  she comes from the group with the highest average
  doesn't mean that she is automatically satisfied.
  She could be the most dissatisfied in a group
  that otherwise consists of satisfied employees
• Exception Fallacy Occurs when you reach a group
  conclusion on the basis of exceptional cases.
  This is the kind of fallacious reasoning that is
  at the core of a lot of sexism and racism. The
  stereotype is of the guy who sees a woman make a
  driving error and concludes that "women are
  terrible drivers."

16
Critical minimal criteria

• the ultimate goals of the research will be clear
  to sponsors and to consumers of the research if
  the following are specified
• the participants, setting and variables of
  interest (extent of generalisation)
• the nature of the relationship that is being
  investigated and when it is expected to be found
  (correlational vs causal)
• the form of the relationship (mediation,
  moderation)
• the unit of analysis (individual, group,
  organisation)

17
Simpsons Paradox

• Simpson's paradox occurs when an association
  between two variables is reversed upon observing
  a third variable.
• The following data were collected on a
  university's admissions to its professional
  schools.
• Admit Deny
• Male 490 210
• Female 280 220
• Is there a case for discrimination here?

18
Are women discriminated against?

• 490/700 males admitted 70
• 280/500 females admitted 56
• However, when we look at the individual schools,
  we observe quite a different picture
• Business
  Law
• Admit Deny Admit
  Deny
• Male 480 120 Male 10 90
• Female 180 20 Female 100 200

19
Or is it the men?

• In the business school 80 of males were admitted
  and 90 of females.
• A higher proportion of females (33) make it to
  the law school compared to 10 of men.
• We observe the relationship between gender and
  admittance is reversed when the individual
  schools are considered.

20
Simpsons paradox

• Simpson's paradox is a classic example of a
  confounding variable. It occurs in this example
  because a higher proportion of women apply to the
  law school.
• a confounding variables is a difference between
  the treatment groups besides the treatment itself
• The law school is much harder to get into.
• Even though the law school admits a slightly
  higher proportion of women, the fact the law
  school admits so few people results in a lower
  proportion of women being admitted to the
  university overall.

21
Using proportions to understand

• Business Law
• Male 600 100 (600/70086)
• Female 200 300 (200/50040)
• Confounding variables present difficulty when
  they have an effect on the response variable
• Admit Deny
• Business 660 140 business school admits
  660/80082.5
• Law 110 290 law school admits
  110/40027.5
• Simpson's paradox occurs in this example because
  women apply more often to the school that admits
  few people.

22
The sample

• Is an appropriate sample(s) readily available?
• Can they be randomly allocated?
• Is the sample size large or small (could make
  tests of significance dubious)?

23
The Contextual Constraints

• the design of the study is affected
• which in turn affects the statistics
• which in turn affects the strength of the
  conclusions one can draw