Lecture 3: Experimental Research Methods

1
CMPDLLM002 Research Methods

• Lecture 3 Experimental Research Methods

2
So far 1

• Introduction
• Nature of research
• Definitions
• Research as a process
• Defining the research problem, Research protocol
  design,
• Undertake research, and document
• Classifying Research
• The field of research,
• The purpose of research
• To review existing knowledge, to describe some
  situation or problems, the construction of
  something novel, explanation.
• The approach to research,
• Theoretical, Laboratory experiment, field
  experiment, case-study and survey.
• The nature of research
• Pure basic research, Testing of existing theory,
• description of the state-of-the-art, specific
  problem solution.

3
So far 2

• Types of research methods
• Qualitative research
• Survey research
• Action research
• Ethnographic research
• Case-study research
• Observation
• Quantitative Research
• Sampling
• Measurement Data Collection
• Experimental Research

4
In This Session

• In this talk we are going to consider three types
  of experimental design
• Pre-Experimental Design
• True Experimental Design
• Quasi-Experimental Design
• and relate them to IS Research through the medium
  of a simple IS example.

5
Outline of our Approach

• Briefly consider the underlying philosophy behind
  experimental design
• Describe the IS example
• Explore the experimental design methods in some
  detail
• Compare and contrast the methods
• Point to other ideas worth investigation

6
Philosophy Underlying Experimental Research

• Experimentation is a process of observation, to
  be carried out in a situation especially brought
  about for that purpose.
• The embryology of the chick - (Aristotle BCE 340)
• Scientific observation is controlled
  investigation (Nagel 1961)
• Etc.
• Experiments must strike a balance between
• richness of worldly realism
• tightness of control
• Experiments are undertaken to produce new
  knowledge
• The practical use of knowledge establishes its
  value
• knowledge is what works or satisfies us (James
  1907)
• New knowledge is derived by reasoning from
  experiments

7
Reasoning From Experiment
Scope

Inference
Reliability
Real World
Experiment
Results
Theory
Knowledge
8
Information System Example

• The main feature of experimental research is the
  deliberate control and manipulation of conditions
  (one or more independent variables), the model of
  the natural sciences.
• We will be interested in the situation in which
  we wish to investigate for instance
• whether changes in font size of the on screen
  text window influence the effects of fatigue that
  occur when users carry out routine tasks using a
  graphical user interface.

9
Pre-Experimental Design

• The One Group Pre-Post Test
• O1 Observes/measures the baseline level of the
  dependent variable before exposure to the
  treatment condition (e.g.task efficiency level).
• X Exposure to the treatment condition/independent
  variable (e.g. reduce window font size).
• O2 Observes/measures the level of the dependent
  variable (task efficiency level) after exposure
  to treatment condition.

10
Pre-Experimental Design

• Problems With the One Group Pre-Post Test
• IF Treatment is given to Subject
• THEN Response occurs.
• T IMPLIES R (modus ponens)
• vulnerable to the logical fallacy of affirming
  the consequent.
• Logic tells us that a false antecedent can imply
  a true consequent.

11
True Experimental Design

• Pre-Post Test Control Group Design
• Experimental Group
• Control Group
• There are two major features of this design
• Randomisation ensures that variation between the
  subjects is equalised between the groups
• Control Group ensures that the conclusions drawn
  about the experimental group and the effect of
  the independent variable are valid

12
Pre-Post Test Control Group Design (Var 1)

• Problem
• the control group O2 measure may be influenced by
  the prior exposure to the O1 measure.
• Experimental Group
• Control Group 1
• Control Group 2

13
Pre-Post Test Control Group Design (Var 2)

• Experimental Group 1
• Experimental Group 2
• Control Group 1
• Control Group 2
• This adds a further level of sophistication and
  internal checking of the possible effect of the
  O1 observation on the experimental group.

14
Quasi-Experimental Design

• Non-Equivalent Control Group Design
• This design uses experimental and control groups
• Often we have to use existing (intact) groups
• These groups are technically not equivalent
• because they have not been equated using
  randomisation procedures
• Disadvantage
• the performance levels of the groups may be
  different irrespective of, or in addition to, any
  effect of the independent variable

15
Further Topics

• Validity of experiments
• internal validity
• external validity
• Laboratory experiments in IS
• Experiments on individuals in IS
• Small group research in IS
• Field experimentation in IS

16
Discussion Topics

• Validity in Information Systems Experiments
• Considerations in Individual and Small Group
  Experimental Research

17
Next Session

• Survey Research
• Lecture 4
• http//www.cms.livjm.ac.uk/taleb/lectures/cmssem00
  3
• Lecture Overview
• Definition
• Planning and undertaking survey research
• Information sources, search method
• Guidelines
• Documentation and presentation