Evaluating Research

1
Evaluating Research

• This lecture ties into chapter 17 of Terre
  Blanche
• We know the structure of research
• Understand designs
• We know the requirements of good research
• Now we can evalute a study
• Is it good? Can we believe its conclusions?

2
Validity in designs

• If a design is not valid, then the conclusions
  drawn are not supported
• Like not doing research at all
• Validity is evaluted before the design is run,
  and problems solved/worked around
• Validity of designs come in 2 parts
• Internal validity (can the design sustain the
  conclusions?)
• External validity (can the conclusions be
  generalized to the population effectively?)

3
Internal validity

• Each design is only capable of supporting certain
  types of conclusions
• Eg. Only experiments can support conclusions
  about causality
• Says nothing about if the results can be applied
  to the real world
• Generally, the more controlled the situation, the
  higher the internal validity

4
External Validity

• Can the findings of the study be generalized?
• Do they speak only of our sample, or of a wider
  group?
• Says nothing about the truth of the result being
  generalized
• Generally, bigger samples with valid measures
  lead to better external validity

5
Examining the validity of designs

• Each of the three major design types has
  different internal/external validity requirements
• We can examine the aims of each of these to
  determine how much weight should be given to
  internal and external validity

6
Validity of descriptive designs

• Aim accurately describe the world
• The central purpose is to describe the population
• External validity issue is central
• Internal validity is not irrelevant
• Measurement instruments
• Poor scales will paint the wrong picture

7
Validity of descriptive designs (2)

• Two important questions to ask
• How good was the sample?
• Were correct sampling techniques used?
• Does the sample represent the population
• Is the target population defined?
• How good was the measurement?
• Were the variables selected representative of the
  behaviours?
• Were reliable/valid scales used?

8
Validity of relational designs

• Aim discover relationaships between variables
• Observation of at least 2 variables
• Generally, only external validity is an issue
• Want the relationships to generalized
• Main internal validity problem is causality
• Causality cannot be sustained by a relational
  design

9
Internal validity of relational designs

• Few problems, if you remember that it cannot show
  causation
• Why?
• Mediator/Moderator variables (change the strength
  of a relationship without taking part)
• Problem of direction
• What is causing what?
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

10
External Validity of Relational Designs

• Focuses on naturally occuring relationships
• Thus external validity is ver important
• Populations must be carefully defined
• Samples selected to represent them well
• Measures must be both reliable valid
• Unreliable / invalid measures can lead to no
  picture/false picture

11
Validity of Experimental designs

• Aim Establish that a relationship is causal
• External validity is fairly important
• Especially to apply findings (eg. cures)
• Most important Internal validity
• Only by sticking to the strict design of the
  experiment is it possible to show causality

12
Control in experiments

• To show that the IV causes the DV, the only
  difference between control and experimental group
  must be the IV manipulation
• Any other differences between the groups is a
  confound
• Something else that may have caused the DV

13
Control in experiments

• The best way to ensure that there are no
  differences between experimental and control
  groups is to assign subject carefully
• Eg. have equal number of males and females in
  both groups
• How do we know that the groups are evenly split
  on all charactertistics?
• Intelligence, gender, personality type, optimism.

14
Random Assignment

• It is impossible to manually create even groups
• We do not know all the characteristics that may
  confound the experiment
• Answer Random assignment
• Ensures that evenly balanced groups will result,
  but only if the sample size is fairly large (n gt
  30)

15
Campbells Scheme

• Detecting what threatens the validity of an
  experiment is difficult
• Donald Campbell devised a system to make the
  system easier
• Designed for experiments, but can be applied to
  all designs
• Gives a list of possible threats to internal and
  external validity
• Check the design to these criteria

16
Threats to internal validity

• 1. Co-varying events
• Another, unseen variable might be causing the
  effect we are seeing
• 2. Maturation
• Changes over time can be caused by a natural
  learning process
• 3. Reactivity (testing effect)
• People realize that they are being studied, and
  respond they way they think is appropriate

17
Internal validity threats

• 4. Instrument decay
• Instruments with low reliability lead to
  inaccurate findings/missing phenomena
• 5. Regression to the mean
• Studying extreme scores can lead to inflated
  differences, which would not occur in moderate
  scorers
• 6. Subject mortality
• If subjects drop out, it creates a bias to those
  who didnt

18
External validity threats

• 1. Subject selection
• Selecting a sample which does not represent the
  population well will prevent generalization
• 2. Operationalization of the variables
• We take a variable (wide scope) and
  operationalize it (narrow scope) will we find
  the same results with a different
  operationalization of the same variable?