Social Research Methods

1
Social Research Methods

• The Delphi Technique
• Content Analysis

2
The Delphi Technique

• Qualitative research method relies on the
  judgement of a panel of experts
• Name eludes to the Oracle of Ancient Greece
• First used in 1950s by RAND
• Corporation in the USA
• Much admired
• Very flexible
• Good at getting beneath the surface
• Good at addressing difficult or expert
  questions
• More structured than convention interviewing
• But also much criticised
• e.g. dismissed as being discredited
• Reputation tarnished by sloppy execution in early
  days

3
The Delphi Technique

• What is the Delphi Technique?
• a systematic method of collecting opinions from
  a group of experts through a series of
  questionnaires, in which feedback on the groups
  opinion distribution is provided between question
  rounds while preserving the anonymity of the
  respondents responses. (Helmer, 1972, cited in
  Masser and Foley, 1987 217-218)
• Key features
• Questionnaire
• Expert opinions
• Iterative process, i.e. rounds
• Individual opinions mediated by group opinions
• Yet anonymity of individuals

4
The Delphi Technique

• Example applications
• Original application cold war nuclear missile
  proliferation
• Technological forecasting, e.g. medicines in the
  1990s
• Demand forecasting, e.g. Hawaii tourism in the
  Year 2000
• Forecasting changes in hotel management in Hong
  Kong post-1997
• Marketing of the island of Gozo separately from
  Malta
• Environmental Impact Assessment of development of
  Salts Mill
• The setting of priorities in managing heritage
  attractions
• Definitions of ecotourism

5
The Delphi Technique

• Basic Steps
• Choose the members of the coordinating group
• Develop criteria for evaluating potential
  candidates for the expert panel
• Identify potential candidates, perhaps on basis
  of a review of literature/professional
  associations
• Request their participation, perhaps using a
  prestigious person to do the asking
• Finalise panel composition
• Identify issues to be considered and develop the
  initial (scoping) questionnaire
• Send the first questionnaire
• Collate the responses

6
The Delphi Technique

• Develop the second (convergence) questionnaire,
  incorporating all new input, perhaps using a
  numerical scale or ranking system to calibrate
  responses
• Send the second questionnaire
• Collate the responses
• Undertake further iterations as necessary,
  perhaps until an acceptable level of
  convergence is achieved
• Send summary results to all respondents
• Apply the judgement(s) to solve the problem(s)
  being addressed through the Delphi study.

7
The Delphi Technique

• Example Managing Heritage Tourism (Garrod and
• Fyall, 2000)
• Follow-up study to a mail-based survey of 300
  managers of heritage attractions
• Delphi used to explore some intriguing issues in
  greater depth
• Survey asked respondents if they
• were willing to be involved in a
• follow-up study
• Panel members recruited from
• among those who said yes

8
The Delphi Technique

• 17 panel members
• Heritage management consultant 2
• Local authority officer 2
• Heritage organisation officer 2
• Historic property manager 6
• Academic 3
• Museum officer 2
• Three topic areas
• What should be the major priorities in the
  mission of heritage attractions?
• What factors are most likely to influence your
  admission pricing policy over the next decade
• What should be the funding priorities of major
  heritage organisations such as the National Trust
  and English Heritage?

9
The Delphi Technique

• The rounds
• Round 1 establish themes
• Round 2 assess themes and place in rank order
• Round 3 assess ranking and re-rank
• Example Heritage attraction mission
• Second Round Third Round Change
• Conservation 1 1 -
• Accessibility 2 2 -
• Finance 3 4 ?
• Education 4 3 ?
• Quality 5 5 -
• Relevance 6 6 -
• Recreation 7 7 -
• Local Community 8 8 -
• 
  
  Rs 0.9462

10
The Delphi Technique

• Panel membership
• Round 1 2 3
• Heritage management consultant 2 2 1
• Local authority officer 2 2 2
• Heritage organisation officer 2 2 2
• Historic property manager 6 5 5
• Academic 3 2 2
• Museum officer 2 2 2
• Total 17 15 14

11
The Delphi Technique

• Strengths of the Delphi Technique
• The technique is flexible enough to be applied in
  a variety of situations and to a wide range of
  complex problems, for which there is often no
  other suitable means of analysis
• The iterative approach allows experts to
  reconsider their judgements in the light of
  feedback from peers
• The process also gives participants more time to
  think through their ideas before committing
  themselves to them, leading to a better quality
  of response
• The anonymity of the approach enables experts to
  express their opinions freely, without
  institutional loyalties or peer group pressures
  getting in the way

12
The Delphi Technique

• The potential influence of personality is also
  removed in this way
• Redundant noise (issues that tend to side-track
  the debate) can be controlled by the project
  manager
• The process generates a record of the groups
  thoughts, which can be reviewed as required
• The method can be used to evaluate the spread of
  opinion as well as consensus points.

13
The Delphi Technique

• Weaknesses of the Delphi Technique
• Delphi can be extremely sensitive to
• the level of panellists expertise
• the composition of the panel
• the clarity of the questions
• the way in which the project manager reports
  reasons for outliers
• the administration of the questionnaire
• It assumes that experts are willing to allow
  their judgements to be re-formed by the opinions
  of others
• Expert panel is vulnerable to high rates of
  attrition due to
• boredom with the subject matter
• disillusionment with the process, and/or
• lack of time to complete the questionnaire before
  the following round commences

14
The Delphi Technique

• Some Delphi practitioners use monetary payments
  or moral persuasion to encourage panellists to
  stay the course however, this may bias the
  results of the study
• There is also a risk of specious consensus
  being formed, whereby panellists acquiesce and
  conform to the median judgement (group think)
• Where consensus is being sought there is a
  problem in determining what actually constitutes
  consensus
• The technique often requires a substantial period
  of time to complete and can be costly in terms of
  the researchers time

15
The Delphi Technique

• References
• Garrod, B. and Fyall. A. (2005) Revisiting
  Delphi The Delphi Technique in tourism
  research, in in Ritchie, B.W., Burns, P. and
  Palmer, C. (eds) Tourism Research Methods
  Integrating Theory and Practice, CAB
  International, Wallingford.
• Garrod, B. (2003a) Defining marine ecotourism A
  Delphi study. In Garrod, B. and Wilson, J.C.
  (eds) Marine Ecotourism Issues and Experiences.
  Channel View, Clevedon, pp.17-36.
• Garrod, B., and Fyall, A. (2000) Managing
  Heritage Tourism. Annals of Tourism Research.27
  682-708.
• Ilbery, B., Maye, D., Kneafsey, M., Jenkins, T.
  and Walkley, C. (2004) Forecasting food supply
  chain development in lagging rural regions
  Evidence from the UK, Journal of Rural Studies
  20 331-344.

16
The Delphi Technique

• Miller, G. (2001) The development of indicators
  for sustainable tourism Results of a Delphi
  survey of tourism researchers. Tourism
  Management 22 351-361.
• Pan, S.Q., Vega, M., Vella, A.J., Archer, B.H.
  and Parlett, G. (1995) A mini-Delphi approach
  An improvement on single round techniques.
  Progress in Tourism and Hospitality Research 2
  27-39.
• Lloyd, J., La Lopa, J.M. and Braunlich, C.G.
  (2000) Predicting changes in Hong Kongs hotel
  industry given the change in sovereignty from
  Britain to China in 1997. Journal of Travel
  Research 38 405-410.
• Smith, S.L.J. (1995) Tourism Analysis A
  Handbook, 2nd edn. Longman, Harlow.
• Frechtling, D. (1996) Practical Tourism
  Forecasting. Butterworth Heinemann, Oxford.

17
Content Analysis

• What is content analysis?
• (or Hermeneutics)
• Manifest content analysis Semiological
  analysis
• Analysis of what is there Analysis
  of meaning of text
• Attempts to minimise interpretation
  Main purpose is to interpret
• Dealing mainly with manifest content analysis
  here
• Involves the quantification of qualitative
  materials
• Pictures, e.g. photos on the front
• of holiday brochures
• Structured text, e.g. codes of
• conduct for nature tourists
• Writing, e.g. newspaper articles
• A mixture of the above

18
Content Analysis

• Example applications
• Analysis of newspaper reports on a particular
  event, e.g. The Bluestone Project
• Analysis of imagery in brochures promoting
  holidays in Scotland, Ireland and Wales to the
  USA
• Analysis of changing tourist-local social
  relationship as depicted on postcards
• Analysis of childrens drawings of what does
  tourism mean for the place where you live?
• Analysis of ethical codes of conduct, e.g.
  whalewatching
• Analysis of definitions of a new term proposed in
  academic articles, books, etc

19
Content Analysis

• Basic steps
• Identify aims and objectives of project
• Develop hypotheses to be tested with reference to
  current theories and previous research
• Select the sample to be utilised in the research,
  using conventional sampling techniques
• Develop content analysis grid, taking care not to
  overlook previous work or standard
  classifications
• e.g. 1. Date ethical code was introduced
• Before 1980
• 1980-1989
• 1990-1999
• 2000 onwards

20
Content Analysis

• Data reduction may be required, to fit the data
  set to the analysis grid, e.g. discarding some
  data this is no longer relevant
• Undertake the coding up. Good practice is
  normally for two people to do this work, the
  researcher and an independent assistant.
• e.g. 1. Date ethical code was introduced
• Before 1980 35 37
• 1980-1989 28 29
• 1990-1999 27 23
• 2000 onwards 10 11
• Determine inter-coder reliability figure, e.g. 92

21
Content Analysis

• If inter-coder reliability figure is
  unacceptable, return to development of coding
  grid and repeat process
• Analyse results to test hypotheses, e.g.
  cross-tabulating pairs of variables and testing
  for strength of association using chi-square (X
  2), e.g.
• Active Passive
• Men 70 30
• Women 50 50
• X 2 8.33 with 1 degree of freedom, so p
  0.01
• Report findings

22
Content Analysis

• Example An Analysis of Whalewatching Codes of
• Conduct (Garrod and Fennell, 2004)
• Sample was a collection of 58 codes of conduct
  for whalewatching from around the world (Carlson,
  2001)
• Common themes extracted
• Approach data, e.g. minimum aircraft approach
  distance, minimum boat approach distance, maximum
  number of boats at a time
• Interaction data, e.g. swimming, touching,
  feeding
• Management orientation data, e.g. permit
  requirements, restrictions on viewing pods with
  calves, control of pollution
• General information, e.g. region, date put in use
• 93 inter-coder reliability achieved

23
Content Analysis

• Example finding
• Region of code
• North America (incl. Mexico) 17
• South and Central America (incl. Caribbean) 14
• Europe 14
• Asia 5
• Africa 4
• Australia, New Zealand, Micronesia 3
• Antarctic and Arctic -
• Global 1
• Specific Basis of Rule
• Voluntary 36
• Regulatory (statutory or non-voluntary) 22
• Chi-square for first three regions vs. specific
  basis of rule 8.00 with 2
• d.f., p 0.018 (North America operating
  significantly more non-
• voluntary than voluntary codes Europe the
  opposite)

24
Content Analysis

• Strengths of (manifest) content analysis
• Enables rigorous analysis of essentially
  qualitative data
• Relies on standard sampling techniques to acquire
  data set issues of reliability and validity can
  be addressed
• Conventional statistics, e.g. chi-square tests,
  can be used to test hypotheses
• Inter-coder reliability figure enables the
  adequacy of the coding-up process to be measured

25
Content Analysis

• Weaknesses of (manifest) content analysis
• Development of analysis grid can be a highly
  subjective process
• Reduction of data set to fit analysis grid can
  introduce further subjectivity
• How high an inter-coder reliability figure is
  required before the iterative process can be
  ended?
• Coding process can be very time-consuming
• Essentially the quantification of qualitative
  issues would qualitative techniques such as
  hermeneutics, focus groups or semi-structured
  interviews not be better suited?

26
Content Analysis

• References
• Garrod, B. and Fennell, D. (2004) An analysis of
  whalewatching codes of conduct, Annals of
  Tourism Research 31 334-352.
• Fennell, D. (2001) A content analysis of
  ecotourism definitions. Current Issues in
  Tourism 4 403-421.
• Malloy, D. and Fennell, D. (1998) Codes of
  ethics and tourism An exploratory content
  analysis, Tourism Management 19 453-461.
• Finn, M., Elliott-White, M. and Walton, M. (2000)
  Tourism Leisure Research Methods Data
  Collection, Analysis and Interpretation. London
  Pearson Education.