The Delphi Technique

1
The Delphi Technique

• Brian Garrod
• Institute of Rural Sciences
• University of Wales Aberystwyth

2
What is the Delphi Technique?

• Qualitative research technique
• but with quantitative elements
• Relies on the judgement of a panel of experts
• Iterative process, taking place over a number of
  rounds
• Much admired
• Very flexible
• Good at getting beneath the surface of issues
• Good at addressing difficult or expert
  questions
• More structured than conventional interviewing
• But also much criticised
• Often dismissed as being discredited
• Accused of being anti-democratic/anti-participator
  y
• Reputation tarnished by sloppy execution in early
  days

3
The Delphi Technique

• Name eludes to the Oracle of Ancient Greece
• First used in 1950s by RAND Corporation in the
  USA
• Cold war application
• Panel of experts formed
• Question related to likely
• number of USSR nuclear
• missiles
• Mean and standard deviation
• Feed back between rounds
• Repeated until consensus approached

4
The Delphi Technique

• What is the Delphi Technique?
• a systematic method of collecting opinions from
  a group of experts through a series of
  questionnaires, in which feedback on the groups
  opinion distribution is provided between question
  rounds while preserving the anonymity of the
  respondents responses. (Helmer, 1972, cited in
  Masser and Foley, 1987 217-218)
• Key features
• Systematic
• Questionnaire
• Expert opinions
• Iterative process, i.e. rounds
• Feedback - individual opinions mediated by group
• Yet anonymity of individuals

5
The Delphi Technique

• Example applications
• Technological forecasting, e.g. medicines in the
  1990s
• Demand forecasting, e.g. Hawaii tourism in the
  Year 2000
• Forecasting changes in hotel management in Hong
  Kong post-1997
• Marketing of the island of Gozo separately from
  Malta
• Environmental Impact Assessment of development of
  Salts Mill
• The setting of priorities in managing heritage
  attractions
• Definitions of ecotourism

6
The Delphi Technique

• Example Managing Heritage Tourism (Garrod and
• Fyall, 2000)
• Follow-up study to a mail-based survey of 300
  managers of heritage attractions
• Delphi used to explore some intriguing issues in
  greater depth
• Survey asked respondents if they
• were willing to be involved in a
• follow-up study
• Panel members recruited from
• among those who said yes

7
The Delphi Technique

• 17 panel members
• Heritage management consultant 2
• Local authority officer 2
• Heritage organisation officer 2
• Historic property manager 6
• Academic 3
• Museum officer 2
• Three topic areas
• What should be the major priorities in the
  mission of heritage attractions?
• What factors are most likely to influence your
  admission pricing policy over the next decade
• What should be the funding priorities of major
  heritage organisations such as the National Trust
  and English Heritage?

8
The Delphi Technique

• The rounds
• Round 1 establish themes
• Round 2 assess themes and place in rank order
• Round 3 assess ranking and re-rank
• Example Heritage attraction mission
• Second Round Third Round Change
• Conservation 1 1 -
• Accessibility 2 2 -
• Finance 3 4 ?
• Education 4 3 ?
• Quality 5 5 -
• Relevance 6 6 -
• Recreation 7 7 -
• Local Community 8 8 -
• 
  
  Rs 0.9462

9
The Delphi Technique

• Panel membership
• Round 1 2 3
• Heritage management consultant 2 2 1
• Local authority officer 2 2 2
• Heritage organisation officer 2 2 2
• Historic property manager 6 5 5
• Academic 3 2 2
• Museum officer 2 2 2
• Total 17 15 14

10
Strengths of the Delphi Technique

• The technique is flexible enough to be applied in
  a variety of situations and to a wide range of
  complex problems, for which there is often no
  other suitable means of analysis
• The iterative approach allows experts to
  reconsider their judgements in the light of
  feedback from peers
• The process also gives participants more time to
  think through their ideas before committing
  themselves to them, leading to a better quality
  of response
• The anonymity of the approach enables experts to
  express their opinions freely, without
  institutional loyalties or peer group pressures
  getting in the way

11
Strenghts of the Delphi Technique

• The potential influence of personality is also
  removed in this way
• Redundant noise (issues that tend to side-track
  the debate) can be controlled by the project
  manager
• The process generates a record of the groups
  thoughts, which can be reviewed as required
• The method can be used to evaluate the spread of
  opinion as well as consensus points.

12
Weaknesses of the Delphi Technique

• Delphi can be extremely sensitive to
• the level of panellists expertise
• the composition of the panel
• the clarity of the questions
• the way in which the project manager reports
  reasons for outliers
• the administration of the questionnaire
• It assumes that experts are willing to allow
  their judgements to be re-formed by the opinions
  of others
• Expert panel is vulnerable to high rates of
  attrition due to
• boredom with the subject matter
• disillusionment with the process, and/or
• lack of time to complete the questionnaire before
  the following round commences

13
Weaknesses of the Delphi Technique

• Some Delphi practitioners use monetary payments
  or moral persuasion to encourage panellists to
  stay the course however, this may bias the
  results of the study
• There is also a risk of specious consensus
  being formed, whereby panellists acquiesce and
  conform to the median judgement (group think)
• Where consensus is being sought there is a
  problem in determining what actually constitutes
  consensus
• The technique often requires a substantial period
  of time to complete and can be costly in terms of
  the researchers time

14
Proposed Best-Practice Guidelines

• The Delphi technique should not be seen as a main
  tool of investigation but a means of
  supporting/extending studies which better
  established and more reliable methods of
  investigation.
• The topic must be appropriate, for example there
  must be no widely-perceived correct answers to
  the questions posed.
• Questions must be pilot-tested to avoid ambiguity
• Panellists should be recognised experts in their
  field (a self-assessment selection procedure may
  be useful in this respect).
• The panel should comprise a good balance of
  different disciplines and areas of expertise.
• Adequate time must be given to experts to think
  deeply about the questions at hand.

15
Proposed Best-Practice Guidelines

• Once a subsequent round has commenced, those
  completing the previous round late should
  nevertheless be excluded from continuing.
• Criteria for panel balance should be set in
  advance. Should these no longer be met the study
  should be terminated.
• Attrition of the panel may be minimised by
  selecting experts who already have a strong
  interest in the outcome of the project.
• This is preferable both to using monetary payment
  and moral persuasion as a means of ensuring that
  experts remain committed to the project.
• Experts must also believe that the Delphi
  technique is a valid way of going about the task
  at hand.

16
Proposed Best-Practice Guidelines

• Full anonymity must be preserved at all times
  between the panellists (but not necessarily
  between the panellists and the coordinating
  researchers).
• The coordination group should make themselves
  available as a resource for locating further
  information on specific subjects or clarifying
  the questions.
• The coordination group should intervene in the
  process as little as possible.
• The panellists must do the initial scoping
  themselves, the coordination group should not set
  the agenda for discussion (although they will
  have to determine the research questions that
  will need to be answered through this process).
• Where consensus is being sought, the coordination
  group should determine the criteria for bringing
  the consensus rounds to a close before the
  project begins.

17
Conclusions

• Quick and dirty?
• Delphi a warning from history!
• Need to establish best practice