Lecture 3: Experimental Research Methods

1
CMPDLLM002 Research Methods

• Lecture 3 Experimental Research Methods

2
So far

• Lecture 1
• Introduction
• Nature of research
• Classifying Research
• The field of research,
• The purpose of research
• The approach to research,
• Lecture 2
• Types of research methods
• Qualitative research
• Survey research
• Action research
• Ethnographic research
• Case-study research
• Observation
• Quantitative Research
• Sampling
• Measurement Data Collection
• Experimental Research

3
In This Session

• In this talk we are going to consider three types
  of experimental design
• Pre-Experimental Design
• True Experimental Design
• Quasi-Experimental Design
• and relate them to IS Research through the medium
  of a simple IS example.

4
Outline of our Approach

• Briefly consider the underlying philosophy behind
  experimental design
• Describe the IS example
• Explore the experimental design methods in some
  detail
• Compare and contrast the methods
• Point to other ideas worth investigation

5
Philosophy Underlying Experimental Research

• Experimentation is a process of observation, to
  be carried out in a situation especially brought
  about for that purpose.
• The embryology of the chick - (Aristotle BCE 340)
• Scientific observation is controlled
  investigation (Nagel 1961)
• Etc.
• Experiments must strike a balance between
• richness of worldly realism
• tightness of control
• Experiments are undertaken to produce new
  knowledge
• The practical use of knowledge establishes its
  value
• knowledge is what works or satisfies us (James
  1907)
• New knowledge is derived by reasoning from
  experiments

6
Reasoning From Experiment
Scope

Inference
Reliability
Real World
Experiment
Results
Theory
Knowledge
7
Information System Example

• The main feature of experimental research is the
  deliberate control and manipulation of conditions
  (one or more independent variables),
• the model of the natural sciences.
• We will be interested in the situation in which
  we wish to investigate for instance
• whether changes in font size of the on screen
  text window influence the effects of fatigue that
  occur when users carry out routine tasks using a
  graphical user interface.

8
Pre-Experimental Design

• The One Group Pre-Post Test
• O1 Observes/measures the baseline level of the
  dependent variable before exposure to the
  treatment condition (e.g.task efficiency level).
• X Exposure to the treatment condition/independent
  variable (e.g. reduce window font size).
• O2 Observes/measures the level of the dependent
  variable (task efficiency level) after exposure
  to treatment condition.

9
Pre-Experimental Design

• Problems With the One Group Pre-Post Test
• IF Treatment is given to Subject
• THEN Response occurs.
• T IMPLIES R (modus ponens)
• vulnerable to the logical fallacy of affirming
  the consequent.
• Logic tells us that a false antecedent can imply
  a true consequent.

10
True Experimental Design

• Pre-Post Test Control Group Design
• Experimental Group
• Control Group
• There are two major features of this design
• Randomisation
• ensures that variation between the subjects is
  equalised between the groups
• Control Group
• ensures that the conclusions drawn about the
  experimental group and the effect of the
  independent variable are valid

11
Pre-Post Test Control Group Design 1

• Variant 1 of the standard PPT Control Design
• To address the concern that
• the control group O2 measure may be influenced by
  the prior exposure to the O1 measure.
• Experimental Group
• Control Group 1
• Control Group 2
• Example

12
Pre-Post Test Control Group Design 2

• Variant 2 of the standard PPT Control Design
• Extends Variant 1 with
• internal checking of the possible effect of the
  O1 observation on the experimental group.
• Experimental Group 1
• Experimental Group 2
• Control Group 1
• Control Group 2
• Example

13
Quasi-Experimental Design

• Non-Equivalent Control Group Design
• This design uses experimental and control groups
• Often we have to use existing (intact) groups
• These groups are technically not equivalent
• because they have not been equated using
  randomisation procedures
• Disadvantage
• the performance levels of the groups may be
  different irrespective of, or in addition to, any
  effect of the independent variable

14
Experimental Design Process 1

• Purpose
• Rigour and reliability
• Experiments require planning and organization to
  produce reliable results.
• An experiment has two variables types
• independent variable is the one being tested,
• dependent variable is the result or change being
  observed.
• Constants materials and conditions that do not
  change.
• A control
• a part of the experiment that is not treated
• serves as a standard for comparison with the
  objects that are treated.

15
Experimental Design Process 2
Source Ref 1
16
Some Useful Definitions

• Null Hypothesis
• In statistics, a null hypothesis is a hypothesis
  set up to be nullified or refuted in order to
  support an alternate hypothesis. When used, the
  null hypothesis is presumed true until
  statistical evidence in the form of a hypothesis
  test indicates otherwise. In science, the null
  hypothesis is used to test differences in
  treatment and control groups, and the assumption
  at the outset of the experiment is that no
  difference exists between the two groups for the
  variable being compared.
• http//en.wikipedia.org/wiki/Null_hypothesis

17
Further Topics

• Validity of experiments
• internal validity
• external validity
• Example
• Laboratory experiments in IS
• Experiments on individuals in IS
• Small group research in IS
• Field experimentation in IS
• Validity in Information Systems Experiments
• Considerations in Individual and Small Group
  Experimental Research

18
Next Session

• Survey Research
• Lecture 4
• http//www.cms.livjm.ac.uk/taleb/lectures/cmssem00
  3
• Lecture Overview
• Definition
• Planning and undertaking survey research
• Information sources, search method
• Guidelines
• Documentation and presentation

19
Extra Slides