Research Methods

1
Research Methods Design in Psychology

• Lecture 2
• Survey Design 2
• Lecturer James Neill

2
Overview

• Survey construction - nuts nolts
• Sampling
• Ethics
• Levels of measurement
• Measurement error

3
What is a Survey?

• A standardised stimulus
• A measuring instrument
• A way of converting fuzzy psychological stuff
  into hard data for analysis

4
Survey Construction Nuts Bolts

• Constructing questions
• Modes of response
• Response formats LOM
• Measurement error
• Survey formatting

5
Constructing questions

• Define target constructs
• Check related research questionnaires
• Draft items (aim to have multiple indicators)
• Pre-test revise

6
When drafting questions aim to

• Focus directly on topic/issue
• Be clear
• Be brief
• Avoid big words
• Use simple and correct grammar

7
Bias in questions

• Inapplicable
• Over-demanding
• Ambiguous
• Double negatives
• Double-barrelled
• Leading
• Loaded

8
Bias in responding

• Social desirability
• Acquiescence or Yea- and Nay-saying
• Self-serving bias
• Order effects

9
Modes of Survey Administration

• Interview
• high demand characteristics
• can elicit more information
• Questionnaire
• lower demand characteristics
• information may be less rich

10
Objective vs. subjective

• Objective How times during 2000 did you visit a
  G.P.?
• Subjective Think about the visits you made to a
  G.P. during 2000. How well did you understand
  the medical advice you received?perfectly
  very well reasonably poorly not
  at all

11
Open-ended vs. close-ended

• Open-ended
• rich information can be gathered
• useful for descriptive, exploratory work
• difficult and subjective to analyse,
• time consuming
• Close-ended
• important information may be lost forever
• useful for hypothesis testing
• easy and objective to analyse
• time efficient

12
Open-ended questions - Examples

• What are the main issues you are currently facing
  in your life?
• How many hours did you spend studying this week?
  _________

13
Close-ended questions Example 1

• What are the main issues you are currently facing
  in your life? (please all that apply)
• financial
• physical/health
• academic
• employment/unemployment
• intimate relations
• social relations
• other (please specify)
• ________________________________

14
Close-ended questions Example 2

• How many hours did you spend studying this week?
• less than 5 hours
• 5 to 10 hours
• 10 to 20 hours
• more than 20 hours

15
Close-ended rating scales

• Likert scale
• Graphic rating scale
• Semantic differential scale
• Non-verbal scale
• Frequency scale

16
Likert Scale

• Pick a number from the scale to show how much you
  agree or disagree with each statement
• 1 2 3 4 5
• strongly disagree neutral agree
  strongly
• disagree agree
• 1 2 3 4 5
• strongly agree neutral disagree
  strongly
• agree disagree

17
Graphic Rating Scale

• How would you rate your enjoyment of the movie
  you just saw? Mark with a cross (X)
• not enjoyable very enjoyable

18
Semantic Differential Scale

• What is your view of smoking?
• Tick to show your opinion.
• Bad _____________________
  Good
• Strong _____________________ Weak
• Masculine _____________________
  Feminine
• Unattractive _____________________
  Attractive
• Passive _____________________
  Active

19
Non-verbal Scale

• Point to the face that shows how you feel about
  what happened to the toy.

20
Verbal Frequency Scale

• Over the past month, how often have you argued
  with your intimate partner?
• 1. All the time
• 2. Fairly often
• 3. Occasionally
• 4. Never
• 5. Doesnt apply to me at the moment

21
Sensitivity Reliability

• Scale should be sensitive yet reliable.
• Watch out for too few or too many options

22
Scale of measurement guidelines

• General aim Maximise sensitivity (i.e. more
  options)
• Maximise reliability (i.e. less options)
• How many measurement options?
• Minimum 2
• Average 3 to 7
• Maximum 10?

23

• FEELING ABOUT SOMETHING
• EXTREMELY POSITIVE EXTREMELY
  NEGATIVE
• 2-Categories
• GOOD NOT GOOD
• 3-Categories
• GOOD FAIR POOR
• 4-Categories
• VERY GOOD GOOD FAIR POOR
• 5-Categories
• EXCELLENT VERY GOOD GOOD FAIR
  POOR

24
Watch out for too many or too few responses

• Capital punishment should be reintroduced for
  serious crimes
• 1 Agree 2 Disagree
• 1 Very, Very Strongly Agree 7 Slightly
  Disagree
• 2 Very Strongly Agree 8 Disagree
• 3 Strongly Agree 9 Strongly Disagree
• 4 Agree 10 V. Strongly Disagree
• 5 Slightly Agree 11 V, V Strongly
  Disagree
• 6 Neutral

25
Sampling

• Sampling Terminology
• What is Sampling?
• Sampling Techniques
• Example Shere Hites Sex Survey
• Summary of Sampling Strategy

26
Sampling Terminology

• Population
• Sampling Frame
• Sample
• Representativeness

27
What is sampling?

• Sampling is the process of selecting units
  (e.g., people, organizations) from a population
  of interest so that by studying the sample we may
  fairly generalize our results back to the
  population from which they were chosen.
• - Trochim, 2002

28
Sampling Techniques

• Probability sampling
• Random
• Systematic
• Cluster
• Multi-Stage Cluster
• Non-probability sampling
• Quota
• Convenience
• Snowball

29
Representativeness of sample depends on

• adequacy of sampling frame
• selection strategy
• adequacy of sample size
• response rate both the representativeness
  of people in sample who actually complete survey
• Note It is better to have a small, good sample
  than a large, poor sample.

30
Sampling ExampleShere HiteAmerican Sexology
31
Male-Female Relations

• Shere Hite doyenne of sex polls
• Media furors worldwide attention
• 127-item questionnaire about marriage relations
  between sexes
• 4500 USA women, 14 to 85 years
• Society and men need to change to improve lives
  of women

32
Some of Hites findings....

• 70 married for 5 years having affairs...
• (usually more for emotional closeness than sex)
• 76 did not feel guilty
• 87 had a closer female friend than husband
• 98 wanted basic changes to love relationships
• only 13 married for 2years were still in love
• 84 were emotionally unsatisfied
• 95 reported emotional psychological harassment
  from their men

33
Some of the critical comments....

• She goes in with prejudice comes out with a
  statistic.
• The survey often seems merely to provide an
  occasion for the authors own male-bashing
  diatribes.
• Hite uses statistics to bolster her opinion that
  American women are justifiably fed up with
  American men.

34
Response rate Selection bias - 1

• 100,000 questionnaires
• Sent to a variety of womens groups - feminist
  organisations, church groups, garden clubs, etc.
• 4,500 replied(4.5 return rate)

35
Response rate Selection bias - 2

• We get pretty nervous if respondents in our
  survey go under 70. Respondents to surveys
  differ from nonrespondents in one important way
  they go to the trouble of filling out what in
  this case was a very long, complicated, and
  personal questionnaire.- Regina Herzog,
  University of Michigan Institute for Social
  Research

36
Summary of sampling strategy

• Identify target population and sampling frame
• Selection sampling method
• Calculate power and required sample size
• Maximise return rate

37
Survey Format Checklist

• Introduction/covering letter or verbal
  introducation
• e.g. Who are you? Are you bona fide? Purpose of
  survey? Ethical approval? How results will be
  used? Confidentiality? Further info? Complaints?
• Instructions
• Sets the mind frame, but be aware few people
  will read it without good prompting and being
  easy-to-read
• Group like questions together
• Consider order effects, habituation, fatigue,
  switching between response formats

38
Survey Format

• Font type / size, number of pages, margins,
  double vs. single-siding, colour, etc.
• Demographics - single section, usually at
  beginning or end of questionnaire, only use
  relevant questions
• Space for comments?
• Ending the questionnaire say thanks!
• Pre-test the questionnaire revise/refine

39
Pre-test Revise

• Pre-test items and ask for feedback
• Revise
• items which dont apply to everybody
• redundancy
• skewed response items
• misinterpreted items
• non-completed items
• Reconsider ordering layout

40
Ethical issues How to treat respondents

• Minimise risk/harm to respondents
• Informed consent
• Confidentiality / anonymity
• No coercion
• Minimal deceit
• Fully debrief

41
Other ethical issues

• Honour promises to provide respondents with
  research reports
• Be aware of potential sources of bias/ conflicts
  of interest
• Represent research literature fairly
• Dont search data for pleasing findings
• Acknowledge all sources
• Dont fake (or unfairly manipulate) data
• Honestly report research findings

42
Levels of MeasurementType of Data
Levels of measurement type of data
43
4 levels of measurement

• Nominal/Category
• Ordinal
• Interval
• Ratio

44
Levels of measurement discrete vs. continuous

• Categorical / Nominal (Discrete)
• Ordinal / Rank (Discrete)
• Interval (Discrete?)
• Ratio (Continuous)

45
Each level has the properties of the preceeding
levels, plus something more!
46
Categorical / Nomimal

• Arbitrary assignment of s to categories
• e.g. male 1, female 2
• No useful information, except as labels

47
Ordinal /Ranked Scales

• s convey order, but not distance
• e.g. in a race, 1st, 2nd, 3rd, etc.
• Often must be treated as categorical

48
Interval Scales

• s convey order distance, 0 is arbitrary
• e.g. temperature (degrees C)
• Usually treat as continuous for 5 intervals

49
Ratio Scales

• s convey order distance, meaningful 0
• e.g. height, age
• ratios - e.g. 2 x old, 3 x high

50
Why do levels of measurement matter?

• different analytical proceduresare used for
  different levels of data
• More powerful statistics can be applied to
  higher levels

51
Measurement scales - Analysis

• categorical nominal - non-parametric
• interval ratio- parametric

52
What are parametric stats?

• procedures which estimate PARAMETERS of a
  population, usually based on the normal
  distribution
• any procedure which uses M, SD
• e.g. t-tests, ANOVAs
• any procedure which uses r
• e.g. bivariate correlation, linear regression

53
Practice Exam Question
54
What are non-parametric stats?

• (Distribution-free Tests)
• procedures which do not rely on estimates of
  population PARAMETERS
• any procedure which uses frequency
• e.g. sign test, chi-squared
• any procedure which uses rank order
• e.g. Mann-Whitney U test, Wilcoxon matched-pairs
  signed-ranks test

55
Parametric vs. non-parametric stats?

• parametric statistics are more powerful
• but are also more sensitive to violations of
  assumptions

56
Measurement error

• Observed score
• true score measurement error
• true score systematic error random error
• Measurement error is any deviation from the true
  value.

57
Sources of Error
Non-sampling
Sampling
Paradigm
Personal
58
Sources of measurement error

• Paradigm
• Personal researcher bias
• Sampling
• Non-sampling

59
To minimise measurement error

• Use well designed measures
• Reduce demand effects
• Maximise response rate
• Ensure administrative accuracy

60
Summary

• Survey construction - nuts bolts
• Sampling
• Ethics
• Levels of measurement
• Measurement error

61
Respond to UCs student survey and win an iPod
Video valued at 380

• Features 30GB, music videos, home movies and
  video podcasts, new iPod games, audiobooks, photo
  albums and, of course, an entire library of up to
  20,000 songs.
• Logon to OSIS before 14 March, fill in the
  student survey youll enter the draw!