A Structured Conceptualization Approach to Survey Questionnaire Development

1
A Structured Conceptualization Approach to Survey
Questionnaire Development

• David Filiberto
• Cornell University

American Evaluation Association 2005
2
Project Background

• New York State Energy Research and Development
  Authority (NYSERDA) need to evaluate home energy
  use in New Yorks Southern Tier.
• Primary data collection? Survey of homeowners?
• Produce instrument with diverse stakeholder
  buy-in while managing complexity of survey
  content and design.

3
What is concept mapping?

• Concept mapping is a method that can be used to
  help an individual or group describe their ideas
  about a topic in a pictorial form.
• Useful when researchers want to involve relevant
  stakeholder groups in the creation of a research
  project.
• Practical applications include strategic
  planning, product development, market analysis,
  decision making, and measurement development
  among others.

4
From Idea to Map in Six Steps

• Develop a Focus
• Brainstorm Ideas
• Sort and Rate Statements
• Compute the Map
• Interpret the Map
• Utilize the Map

Source Trochim, W. 1999
5
Preparing the Project Develop a Focus, Engage
Diverse Groups

• Focus Prompt
• Please generate short phrases or statements to
  finish the following sentence
• Specific factors and concerns that influence my
  energy consumption and energy conservation are.

6
Generate Ideas

• Participants generate brainstormed statements to
  the focus prompt.

• The fact that the landlord pays for it. (52)
• My children don't understand or care how the
  bills get paid, so they take no care to conserve.
  (55)
• Length of life of the practice or technology.
  (35)
• Every time I think I'm doing something to save
  energy, my bill does not go down. It is very
  discouraging. (87)
• Don't want to go into debt to pay for
  conservation measures. (2)
• The steps that would help the most in terms of
  saving money on energy -- such as hiring a
  contractor to insulate my old house -- are also
  the hardest to afford. (10)
• Hard to find time to research options thoroughly.
  (95)

7
Participants Build the Conceptual Framework
Sort the Statements

• Group the statements according to how similar in
  meaning they are to one another.
• Record the results. Give a short phrase or title
  that describes the contents of the pile.

My children don't understand or care how the
bills get paid, so they take no care to conserve.
(55)
Length of life of the practice or technology. (35)
The fact that the landlord pays for it. (52)
Pile Title or Main Topic Lack of
Knowledge Record here the identifying number
of each item in this pile, separating the ID
numbers with commas. 28, 8, 86, 66, 23, 13, 85,
5
8
Compute the Map
9
Interpret Maps

• Choose more appropriate Cluster Names.
• Move statements with high bridging values that
  appear misplaced to clusters where they are more
  suited.
• Look for directional patterns from clusters-
  North/South, East/West.

10
Technical / Analytical
Social / Psychological
7 Social and Environmental Awareness
55
72
11
21
74
88
71
69
70
15
73
78
80
79
14
89
75
66
23
13
82
12
39
85
86
6 Time Knowledge Considerations
16
61
46
57
81
19
6
42
84
8
10
33
41
9
95
37
64
18
96
36
24
5 Influence/ Control
5
30
3
20
25
98
32
44
47
50
2 Hassle Factors
22
93
35
29
28
7
26
56
31
58
97
91
59
34
17
51
4
38
94
60
83
92
1 Financial Considerations
4 Fear and Anxiety
53
40
87
2
1
48
100
67
54
43
49
76
63
65
99
52
3 Program and Subsidy Issues
68
90
77
27
62
45
contains all the details and provides a
conceptual framework
11
Utilize the Map

• Best Practice Steps for a Successful Survey
• (Salant and Dillman 1994)
• 1. Understand and avoid error.
• 2. Be specific about what new. information you
  need and why.
• 3. Choose the survey method that works best for
  the project.
• 4. Decide how to sample.
• 5. Write good questions.
• 6. Design and test questionnaire.
• 7. Put together the people and equipment that
  will be able to carry out the survey in the
  necessary time frame.
• 8. Code, computerize and analyze the data.
• 9. Present results in a way that is informative
  to the target audience.

• Remember the Map
• A Structured Conceptualization from the prompt
• Specific factors and concerns that influence my
  energy consumption and energy conservation are
• Statements and Clusters
• Questions and Sections

12
From Map to Questionnaire

• A Collaborative Process
• Outline and order the questionnaire sections to
  correspond to Map Clusters.
• Develop survey questions from statements within
  each Cluster section.

13
(No Transcript)
14
Results

• A 47 question survey instrument developed to
  evaluate Home Energy Use in Central New York.
• Pretest required the rewrite of only one
  question.
• Drop off/pickup survey implemented in Ithaca,
  Binghamton, Owego and Trumansburg New York.
• response rate over 95

15
Conclusion

• Concept Mapping provides an alternative approach
  to develop a survey instrument with greater
  collaboration and buy-in than traditional best
  practices of survey design.
• The clustering of statements gave order to the
  survey sections and provided pertinent material
  for question generation.