AMSc Research Methods Research approach IV: Experimental [1]

1
AMSc Research MethodsResearch approach IV
Experimental 1

• Jane Reid
• jane_at_dcs.qmul.ac.uk

2
Overview of lectures

• Tuesday 4th December Jane
• Lecture on experimental design
• Introduction to experimental design exercise
• Thursday 6th December Jane
• Experimental design presentations
• Tuesday 11th December Marie-Luce
• Lecture on statistical testing
• Introduction to statistical testing exercise
• Thursday 13th December Marie-Luce
• Statistical testing presentations

3
Background

• Aim to develop and/or test theories about human
  characteristics and behaviour
• Origins in experimental psychology
• Experimental work can be carried out at different
  stages of the system development process, e.g.
• Design, i.e. to provide requirements capture
  information
• Implementation, i.e. for formative evaluation of
  prototypes
• Evaluation stage, i.e. for final evaluation, in
  terms of functionality or usability, of finished
  system

4
Scenario

• Imagine you wish to test 2 different schemes for
  helping young children to learn new vocabulary
• The 2 schemes are
• Cards with the individual words written on them
  (W cards)
• Cards with the individual words and an
  illustrative picture on them (WP cards)
• You want to examine how effective these cards are
  at different recall times of 2 minutes, 10
  minutes and 30 minutes

5
Experimental hypotheses 1

• A theory developed about a relationship between 2
  or more factors is an experimental hypothesis
• Hypotheses are developed in pairings
• A hypothesis that predicts the presence of an
  effect or association is an alternative
  hypothesis
• E.g. The type of vocabulary scheme employed will
  have an effect on the number of words recalled
• A hypothesis that predicts the absence of an
  effect or association is a null hypothesis
• E.g. The type of vocabulary scheme employed will
  have no effect on the number of words recalled

6
Experimental hypotheses 2

• A hypothesis can predict an effect or
  association
• In one direction - a one-tailed hypothesis
• E.g. The WP vocabulary scheme will result in
  more words being recalled than the W scheme (WP
  gt W)
• In either direction - a two-tailed hypothesis
• E.g. The type of vocabulary scheme employed will
  result in a different number of words being
  recalled (WP gt W or W gt WP, i.e. W ltgt WP)

7
Hypothesis testing 1

• An experiment is designed to answer one (or more)
  experimental hypothesis(es)
• A hypothesis cannot be proved it can only be
  upheld on the basis of the evidence presented
• Dangers of hypothesis testing
• Type I error
• The null hypothesis is incorrectly rejected, i.e.
  the alternative hypothesis is incorrectly upheld
• E.g. finding that WP gt W, when really W WP

8
Hypothesis testing 2

• Type II error
• The null hypothesis is incorrectly upheld, i.e.
  the alternative hypothesis is incorrectly
  rejected
• E.g. finding that W WP when really W gt WP

9
Variables 1

• An experimenter attempts to eliminate unwanted
  effects from confounding variables, i.e. factors
  other than those being investigated
• E.g. age, family background, educational
  background, gender...
• Independent variables
• Can sometimes be manipulated directly by the
  experimenter
• E.g. vocabulary scheme
• Can sometimes only be manipulated indirectly by
  assignment of participants
• E.g. gender

10
Variables 2

• Each manipulation of an independent variable is
  an experimental condition
• E.g. W and WP vocabulary schemes
• Dependent variables
• These are the variables thought to be affected by
  changes in the independent variables
• E.g. number of words recalled after a set time
• The effect on the dependent variables is measured
  during the experiment

11
Experimental design 1

• The basis for the experiment
• Differential designs
• Examination of the difference in effect of 2 or
  more experimental conditions (of an independent
  variable) on a dependent variable
• E.g. The effect of choosing the W or WP
  vocabulary scheme on the number of words recalled
• Correlational designs
• Examination of a possible association between 2
  variables
• E.g. Any association between the ability to learn
  new vocabulary and the ability to spell

12
Experimental design 2

• Not possible to form a hypothesis about which
  variables have an effect on which other
  variables, so
• No concept of independent/dependent variables
• A cause-and-effect relationship does not always
  exist between the variables anyway

13
Designs with one independent variable 1

• Using a control condition
• The 2 experimental conditions are the control
  condition and the independent variable
• E.g. a standard, existing vocabulary scheme using
  no cards
• The effect of the independent variable is
  measured against the baseline of the control
  condition results
• Using 2 experimental conditions
• The effects of the 2 conditions are measured
  against each other
• E.g. W and WP vocabulary schemes

14
Designs with one independent variable 2

• Using 3 or more experimental conditions
• Possible to examine trends across 3 or more
  conditions
• E.g. The effectiveness of a vocabulary scheme
  after different amounts of recall time 2
  minutes, 10 minutes, 30 minutes

15
Designs with 2 or more independent variables

• Each independent variable may have a separate
  effect - a main effect
• E.g. effect of vocabulary scheme choice, effect
  of recall time
• There may also be an interaction effect due to
  the combination of 2 or more independent
  variables
• E.g. One vocabulary scheme (W) may lead to better
  recall after a short period of time (2 minutes)
  while the other (WP) may lead to better recall
  after a longer period of time (10 minutes)
• Extension to 3 or more independent variables is
  possible, but such designs may quickly become
  complex
• E.g. Include the effect of gender in the
  experiment

16
Participants 1

• When different participants are assigned to each
  experimental condition, this is an unrelated or
  between-subjects design
• E.g. Different children assigned to each
  vocabulary scheme
• Advantage 2 closely related experimental
  conditions can be tested with different
  participants
• Disadvantage possible bias in results due to
  unwanted effects from other variables

17
Participants 2

• When the same participants are assigned to each
  experimental condition, this is a related or
  within-subjects design
• E.g. The same children perform under all 3 recall
  conditions
• Advantage no confounding variables possible to
  study an effect over time
• Disadvantage care needed in experimental design
  to avoid unwanted learning effects, e.g. by using
  counterbalancing

18
Participants 3

• When different participants are assigned to each
  experimental condition, but matched on important
  characteristics, this is a matched subjects
  design, and is treated as a related subjects
  design
• E.g. Children assigned to different groups are
  matched on age, gender...
• Advantage attempt to combine the advantages of
  the related and unrelated subjects designs
• Disadvantage very difficult (impossible?) to
  identify and match on all significant variables

19
Data measurement

• For statistical analysis, data must be
• Quantitative
• Objective
• Levels of measurement
• Nominal - categories of data
• Ordinal - rank ordered data
• Interval - numerical data, measured on a
  continuous scale, with equal intervals between
  measurement points
• Ratio - interval data with an absolute zero

20
Measures

• For naturally quantitative data
• Results in interval or ratio data
• E.g. time, number of errors
• For qualitative data that can be measured in a
  quantitative way (or classified as such by the
  experimenter)
• Results in ordinal data
• E.g. confidence, satisfaction
• For qualitative data
• Results in nominal data
• E.g. preference for one interface over another

21
Instruments 1

• Questionnaires
• Structured closed (fixed) questions and response
  categories
• Semi-structured some closed questions and some
  open (flexible) questions
• Unstructured open questions
• Can be administered before or after the
  experiment (pre- and post- questionnaires)
• Advantage experimenter does not need to be
  present
• Disadvantage no interaction, so no discussion -
  unexpected data may be missed

22
Instruments 2

• Interviews
• Similar possibilities to questionnaires
• Advantage interaction, so unexpected information
  may be captured
• Disadvantage participants may be less willing to
  express themselves honestly
• Observation
• Experimenter observes the participant during the
  experiment, and may video- or audio-tape the
  experiment
• Advantage can pick up on transitory events, e.g.
  hesitations
• Disadvantage Hawthorne effect, i.e. participants
  behave differently when they are aware of being
  observed

23
Instruments 3

• Think aloud protocols
• Participants are asked to think aloud during
  the experiment
• Advantage can pick up on thoughts and feelings
  of the participant
• Disadvantage an unnatural process, so may
  distract the participant from his task
• Automatic collection of data
• E.g. software logging

24
Data analysis

• Descriptive statistics
• Describe the raw data in numerical or graphical
  form
• Inferential statistics
• Statistical tests and significance testing
• Allow conclusions (inferences) to be drawn from
  results and generalisations to be made
• BRING A CALCULATOR WITH YOU NEXT WEEK!