Title: M.Sc. Computing Research Methods 06-Experimental Research
1M.Sc. ComputingResearch Methods06-Experimental
Research
- Adrian J Pullin
- FML208
- pullina_at_hope.ac.uk
- hopelive.hope.ac.uk/imc/staff/pullina
2Plagiarism Warning
- The majority of the material in this presentation
would constitute plagiarism if it were submitted
for assessment
3What is Science?
- Science is best defined as a careful,
disciplined, logical search for knowledge about
any and all aspects of the universe, obtained by
examination of the best available evidence and
always subject to correction and improvement upon
discovery of better evidence. What's left is
magic. And it doesn't work. Randi, James,
n.d. - James Randi has an international reputation as a
magician and escape artist, but today he is best
known as the world's most tireless investigator
and de-mystifier of paranormal and
pseudoscientific claims.
4Originality (1)
- Setting down a major piece of new information in
writing for the first time - Continuing previous original work
- Carrying out original work designed by supervisor
- Providing single original technique, observation
or result in otherwise unoriginal but competent
research - Having many original ideas, methods or
interpretations performed by others under your
direction - Originality in testing someone elses idea
- Empirical work that has not been done before
- Making a synthesis that has not been made before
- Using already known material but with new
interpretation
5Originality (2)
- Doing something in your country that has
previously been done in other countries - Corollary comparing something in your country
with the same thing in another country AP - Taking a particular technique and applying it in
a new area - Bringing new evidence to bear on an old issue
- Being cross-disciplinary and using different
methodologies - Looking at areas that people in that discipline
have not looked at before - Adding to knowledge in a way that has not been
done before - Phillips and Pugh, 2000
6What is Scientific Method?
- Observe some aspect of the universe
- Invent a tentative description, called a
hypothesis, that is consistent with what you have
observed - Use the hypothesis to make predictions
- Test those predictions by experiments or further
observations and modify the hypothesis in the
light of your results - Repeat steps 3 and 4 until there are no
discrepancies between theory and experiment
and/or observation - Wudka, Jose 1998
7What is a Theory?
- When consistency is obtained the hypothesis
becomes a theory and provides a coherent set of
propositions which explain a class of phenomena. - A theory is then a framework within which
observations are explained and predictions are
made - Wudka, Jose 1998
8What Is Scientific Method? (Diag)
Wudka, Jose 1998
9Steps in the Scientific Method
Dolphin L, 1992
10Scientific Method Text Version
- Working from previous knowledge
- Formulate research question
- State hypothesis
- Test hypothesis
- Design experiment
- Conduct experiment
- Analyse data
- (Iterate)
- Publish results
- Peer review
- After Kenyon, P, n.d.
11Triangulation
- Note triangulation on previous slides
- Concept is credible because a range of sources
state the same thing - In this case, a massive range, of which this is
only a very small sample
12Why Scientific Method?
- Unprejudiced
- Dont have to believe researcher
- Redo experiment and determine if results are true
or false - Conclusions will hold irrespective of state of
mind, religious persuasion, state of
consciousness of investigator and/or subject - Faith (belief that does not rest on logical proof
or material evidence) does not determine whether
theory is adopted - Theory is accepted not on prestige or convincing
powers of proponent, but on results obtained
through observations and/or experiments which
anyone can reproduce - (? Do you really believe this? - AP)
- Results obtained using the scientific method are
repeatable - Most experiments and observations are repeated
many times - If original claims not verified, origin of
discrepancies is studied - Wudka, Jose 1998
13Caveat
- Getting the same results is not the same as
drawing the same conclusions
14Experimental ResearchHolah, M 2005 next 10
slides
- Most scientific of all methods
- Method of choice
- At least in some areas
- Provides controlled situations
- An experiment is a study of cause and effect
- Involves deliberate manipulation of one variable,
while trying to keep all other variables constant
15Laboratory Experiments (1)
- Try to keep all aspects of the situation constant
except one - The one we are looking at
- Aspect we vary is called the independent variable
(IV) - Change this in a very precise way
- The aspect we measure, the dependent variable
(DV), changes as a result - If our thesis is correct
- DV depends on IV
16Laboratory Experiments (2)
- All other variables which might affect the
results and therefore give us a false set of
results are called confounding variables - Or random variables
- E.g.
- Differences in instructions given by an
experimenter - Could be overcome by standardising instructions
- Differences between participants, e.g. in their
age - Eliminated as a variable by using a single age
group
17Laboratory Experiments (3)
- Change one variable (the IV)
- Measuring another (the DV)
- Control all others, as far as possible
- Experimental method allows us to draw conclusions
with far more certainty than any non-experimental
method - If IV is only thing that is changed then it must
be responsible for any change in DV
18Example (Say HCI)
- Divide participants into 2 groups
- Experimental group and control group
- Introduce a change for experimental group and not
control group - A control group is a group for whom the
experimenter does not change IV - Experimental and control groups must be matched
on all important characteristics - Age, sex, experience etc.
19Advantages of Laboratory Experiments
- Only means by which cause and effect can be
established - IV is thought of as cause and (DV) as effect
- Allows for precise control of variables
- Control group enables experimenter to isolate one
key variable (IV) and observe its effect on some
other variable (DV) - Experiments can be replicated
- Cannot generalise from one set of results
- Repeating experiment, with same results obtained,
gives more confidence that theory being tested is
valid - Experimental method consists of standardised
procedures and measures which allow it to be
easily repeated - Experiments yield quantitative data
- Can be analysed using inferential statistical
tests - Permit statements about how likely the results
are to have occurred through chance
20Limitations of Laboratory Experiments (1)
- Artificiality experiment is not typical of real
world situations - Most experiments done in laboratories
- Strange, contrived environments
- People asked to perform unusual or bizarre tasks
- Unusual equipment set-up
- Bias George Miller (1962) estimated 90 of U.S.
experiments used college students - Accessible and cheap
- Results still tend to be generalised to
population as a whole
21Limitations of Laboratory Experiments (2)
- Not possible to completely control all variables
- Variables at work of which the experimenter is
unaware - Ethics
- Experiments with people nearly always involve
deceiving participants to some extent - Educational experiments are still experimenting
on children - (Aside need approval from head teacher/parents
and possibly a CRB check) - Normative data
- Grouping people together and trying to cancel out
individual differences to look at a group norm is
limited in how much it can tell us because it
ignores what is special about people
22Field Experiment
- Experiments in a more natural setting
- The field
- IV still deliberately manipulated by researcher
- Not possible to have such tight control over
variables in the field - Advantage of being far less artificial than the
laboratory
23Natural Experiment
- Take advantage of natural situation to carry out
investigation - Circumstances cannot be manipulate
- E.g. school may decide to switch to a new CBL
package - Effects compared with similar school using
different package - Not true experiment
- Unable to manipulate or control variables
- Quasi-experiment
- Is possible to compare two groups
- Equivalent of experimental and control group
- Participants are unaware that they are taking
part in an investigation - Not as artificial as a laboratory setting
24Scientific Method Infallible?
- Error? - Electrochemically Induced Nuclear Fusion
of Deuterium - Martin Fleischmann, Department of Chemistry, The
Universityof Southampton, Hants. S09
5NH,ENGLANDStanley Pons, Department of
Chemistry, University of Utah, Salt Lake city, UT
84112 USA - Submitted to Journal of Electroanalytical
Chemistry March 11,1989 in final form March 20,
1989 - http//www.totse.com/en/fringe/fringe_science/fusi
on.html - http//encyclopedia.thefreedictionary.com/coldfus
ion - Hoax? - Piltdown Man
- http//www.clarku.edu/piltdown/Prefacegratitude.
html - http//www.talkorigins.org/faqs/piltdown.html
- Deliberate fraud? - molecular-scale transistors
- http//www.socialismtoday.org/70/science.html
- http//seattlepi.nwsource.com/business/88624_bell2
61.shtml
25Scientific Method Protection
- http//www.sfn.org/NL/1997/March-April/ethics_poli
cy.html - http//www.bps.org.uk/publications/journals/princi
ples-of-publishing.cfm - http//www.liv.ac.uk/research/work/conduct.htm
26And Finally
- "There is not the slightest indication that
nuclear energy will ever be obtainable. It would
mean that the atom would have to be shattered at
will." Albert Einstein, 1932
27ReferencesHarvard format
- Wudka, Jose 1998 The Scientific Method
http//phyun5.ucr.edu/wudka/Physics7/Notes_www/n
ode5.htmlSECTION02120000000000000000 accessed
24/02/05 - Holah, M 2005, The Experimental Methods,
http//www.holah.karoo.net/experimental_method.ht
m Accessed 22/02/05 - This is a site aimed at Psychology AS students,
but the material on experimental method is
excellent - Dopohin L 1991 Steps in the scientific method,
http//www.ldolphin.org/SciMeth2.html accessed
24/02/05 - Phillips, E and Pugh, DS, 2000 How to Get a
PhD A Handbook for Students and Their
Supervisors, Open University Press - Kenyon P n.d. The Scientific Method
http//salmon.psy.plym.ac.uk/year1/scimeth3.gif
accessed 24/02/05
28See Also
- Web search on scientific method
- http//en.wikipedia.org/wiki/Scientific_method
- http//koning.ecsu.ctstateu.edu/Plants_Human/scime
th.html - Randi, James n.d. http//www.randi.org/
- But also
- http//www.dharma-haven.org/science/myth-of-scient
ific-method.htm - Note these are very sloppy references
- Do them properly in your dissertation