Online surveys and qualitative analysis: a contradiction in terms?

1
Online surveys and qualitative analysis a
contradiction in terms?

• Martyn Warren

2
Objectives

• Introduce off-the-shelf survey software and its
  capabilities
• Consider the nature of the output from such
  surveys, and the degree to which some of it can
  be open to interpretive approaches
• Stimulate discussion

3
Perseus - a little bit of magic

• Perseus Software Solutions an off-the-shelf
  programme facilitating
• Design of questionnaires
• Administration of surveys by email/WWW
• Collation of responses into MS Access
• Calculation of basic descriptive measures
• Presentation via Powerpoint
• Export into SPSS, Excel, etc

4
Take a look

• Drafting of questionnaire
• Creation of slide presentation
• Note textual responses

5
(No Transcript)
6
(No Transcript)
7
(No Transcript)
8
(No Transcript)
9
(No Transcript)
10
(No Transcript)
11
(No Transcript)
12
How many personal computers (i.e. desktop or
laptop) does your business use?
13
My experience

• Village webmasters (75 response rate)
• Broadband4Devon - small businesses in Devon (38)
• Survey of attitudes of UoP staff to education for
  sustainable development (29)
• Student attitude survey, module GGX2107 (51)
• Fixing dates for meetings, rehearsals, etc

14
Discussion in literature

• Much interest in late 90s in methodological
  issues
• Sampling
• Response rates
• Ethical issues, including privacy,
  confidentiality and informed consent
• Security
• Technology

15
Discussion in literature (2)

• Use of online processes for qualitative
  approaches
• Virtual ethnography (e.g. Hine 2000)
  lurking around chat rooms, etc.
• Online focus groups, Delphi, online interviews
  (real-time and asynchronous), e.g. Jones (1999)
• Content analysis of websites, blogs,
  etc. (e.g. Technorati, Google)

16
What about surveys?

• Is it possible to create circumstances where
  survey software can be used to create rich data,
  susceptible to interpretive analysis?
• Implicit assumption in a random selection of
  methods books that textual data gained through
  surveys is merely an addendum to quantitative
  data, to be coded as nominal data and analysed as
  such.

17
Surveys in social research

• Standardized questions may not get at important
  aspects of a social group, and usually there is
  a limited range of responses a respondent can
  make.
• Surveys collect data on previously identified
  variables limited conceptual flexibility.
• Survey may not succeed for cultural or ethical
  reasons
• (Williams 2003 50)

18
Example village webmasters
Warren and Skerratt 2003
online
offline
19
Example question from Phase 2

• 42. How important do you think the website is to
  the future of the village?
• (5-point scale from essential to irrelevant)
• 43. Can you outline the reasons for your response
  to Question 42?

20
Example output from Phase 2

• It is merely an information tool which is
  available to a minority group of people. I have
  seen the response to parish council meetings
  where there have been 2 people from the public
  attending. If they can not be bothered to attend
  face to face meetings then what chance has a web
  page have of having any important value to the
  village. If this village was less populated (pop.
  4400), then the community might feel a little bit
  closer to one another and the web site
  might have some higher significance!

21
Example output from Phase 2

• The village sometimes seems to be full of
  disinterested sheep-shaggers always has been and
  always will be. Until computer monitors have
  horns and woolly keyboards the Internet will be
  derided as "a load of rubbish for over educated
  wastes of space".

22
Example output from Phase 2

• I derive enormous satisfaction from doing
  something I enjoy (technically and creatively),
  from being able to express the love I have for
  this place to which we moved in 1967 and the
  friendliness we experienced, and most of all the
  almost incessant positive feedback - from people
  casually saying how the maps helped them find
  their way here, to people catching up with
  long-lost friends and relatives, tracing the
  place of their birth, even branches of their
  family they didn't know existed - one woman was
  enabled to put to rest the concern over not
  knowing how her mother had died over 50 years
  ago. Just last week, a man traced a long-lost pal
  because he put his surname into Google, and it
  took him to the Schmoose page to which his pal
  had contributed - and he's in the middle of China
  ! Talking about China, one young man e-mailed
  from an airport in the middle of China to say
  that he couldn't see his parents in my pictures
  of the village fete .. on the day it happened ..
  and! asking were they at home !

23
Questionnaire, Phase 4

• Seven statements posed, arising from Phase 2
  responses, each of the form
• Even if everyone in the village had access to the
  internet, and were able to use it, the website
  would still only represent/serve a part of the
  community.
• (5-point scale, strongly agree to strongly
  disagree)
• Please outline the reasons for your response

24
Example output from Phase 4
25
Example output from Phase 4

• '"Being able to use the Internet" is not the same
  as feeling that it is the easiest and most
  natural way to find information and exchange it
  with banks, government agencies and whatever.
  There must be at least half of our village
  community who will never in their lifetimes use
  the Internet voluntarily for a serious purpose.
  Even the next generation is going to have a good
  share of technophobes. We run a great risk of
  making them a disadvantaged group through no
  fault of theirs whatever.

26
Example output from Phase 4

• I am never very keen on "Portal" systems. As the
  number of possible services and transaction types
  is enormous and likely to grow exponentially,
  putting a portal in the way is going to be hell
  for the portal builder. Not for me thanks! In
  any case, for the village surfer, what is the
  point of NOT going directly to the Inland
  Revenue, the local planning authority, the Dept
  of Social Security or wherever they want to get
  to? The government or agency instructions will
  always say "go to http//junk.agency.gov.uk" and
  not "go to your village web site and see where
  they tell you"! In other words I do feel
  strongly that I could only subtract value from
  such government services, rather than adding it.

27
What have we got?

• Respondents seem more likely to write at length
  online than in conventional surveys (evidence?)
• Textual responses constrained by pre-specified
  questions, but many are rich in content.
• Researcher-respondent interaction limited, but
  speed of electronic communication allows some
  reactive probing.
• Lack of the signals speed, body language,
  facial expression, voice tone that can be
  observed in an interview
• but at the same time responses are unfiltered by
  interviewer/transcriber
• Reflexivity?

28
Qualitative researchers

• use multiple methods to collect rich,
  descriptive, contextually situated data in order
  to seek understanding of human experience or
  relationships within a system or culture.
• Processes of analytical induction from the data
  might then lead to the formulation of simple
  explanatory hypotheses or, using systematic
  approaches such as grounded theory, the
  development of complex theories.
  (Mann Stewart, 2000 2-3)

29
I would not argue

• that textual survey data can fill the place of
  conventional ethnographic techniques
• but I do think there is a case for taking it
  more seriously particularly when arising from
  online processes as one source of qualitative
  data.
• What do you think?

30
Perseus in FSSB

• The Faculty has a copy of Perseus Professional in
  an open access room in Cookworthy
• Contact Jackie Palmer, Learning Technologist,
  SSB Specialist IT Team, Room 311 Cookworthy.
  Phone (3)3277 email J.Palmer_at_plymouth.ac.uk

31
One alternative (of many)
32
(No Transcript)
33
References (1)

• Cho, H., LaRose, R. (1999). Privacy issues in
  internet surveys. Social Science Computer Review,
  17(4), 421-434.
• Crawford, S. D., Couper, M. P., Lamias, M. J.
  (2001). Web surveys - Perceptions of burden.
  Social Science Computer Review, 19(2), 146-162.
• Daley, E. M., McDermott, R. J., Brown, K. R. M.,
  Kittleson, M. J. (2003). Conducting Web-based
  survey research A lesson in Internet designs.
  American Journal of Health Behavior, 27(2),
  116-124.
• Hine, C. (2000). Virtual Ethnography. London
  Sage.
• Jones, S. (Ed.). (1999). Doing internet research
  critical issues and methods for examining the
  Net. Thousand Oaks, CA Sage Publications.
• Kaye, B. K., Johnson, T. J. (1999). Research
  methodology Taming the cyber frontier -
  Techniques for improving online surveys. Social
  Science Computer Review, 17(3), 323-337.
• Knapp, H., Kirk, S. A. (2003). Using pencil and
  paper, Internet and touch-tone phones for
  self-administered surveys does methodology
  matter? Computers in Human Behavior, 19(1),
  117-134.
• Mann, C., Stewart, F. (2000). Internet
  communication and qualitative research a
  handbook for researching online. London Sage.

34
References (2)

• Rhodes, S. D., Bowie, D. A., Hergenrather, K.
  C. (2003). Collecting behavioural data using the
  world wide web considerations for researchers.
  Journal of Epidemiology and Community Health,
  57(1), 68-73.
• Schleyer, T. K. L., Forrest, J. L. (2000).
  Methods for the design and administration of
  Web-based surveys. Journal of the American
  Medical Informatics Association, 7(4), 416-425.
• Sills, S. J., Song, C. Y. (2002). Innovations
  in survey research - An application of Web-based
  surveys. Social Science Computer Review, 20(1),
  22-30.
• Warren, M. F., Skerratt, S. (2003, 17-19
  September 2003). Visiting rural 'virtual
  villages' through online surveys and online
  ethnography. Paper presented at the Association
  of Survey Computing., University of Warwick.
• Wellman, B., Haythornthwaite, C. (Eds.).
  (2002). The internet in everyday life. Oxford
  Blackwell Publishing Ltd.
• Witmer, D., Colman, R., Katzman, S. (1999).
  From paper-and-pencil to screen-and-keyboard
  toward a methodology for survery research on the
  internet. In S. Jones (Ed.), Doing internet
  research critical issues and methods for
  examining the Net (pp. 299). Thousand Oaks, CA
  Sage Publications.
• Yeaworth, R. C. (2001). Use of the Internet in
  survey research. Journal of Professional Nursing,
  17(4), 187-193.
• Williams, M (2003). Making sense of social
  research. London, Sage.