___________________________ ARL Workshop on User Surveys

1
___________________________ARL Workshop on
UserSurveys

• Measurement Scales and Types of Questions
• Brinley Franklin

2
____________________________Assumptions For This
Section

• The organization knows the information sought
• The sampling population has been evaluated
• A sampling plan and type of survey have been
  chosen
• The organization is ready to begin preparing the
  survey instrument

__________________________________________________
__
ARL Workshop on User Surveys

Slide 1
3
___________________________Importance Of This
Section

• Reliable and valid survey questions are a
  cornerstone of effective survey research
• Done correctly, questions can produce valuable
  information
• Done incorrectly, questions can gather
  inappropriate information, introduce bias, or
  even invalidate the user survey

__________________________________________________
___
ARL Workshop on User Surveys

Slide 2
4
___________________________Topics To Be Covered

• The effects of attitudes, behavior, knowledge,
  and demographics on question construction
• Understanding the surveys context
• Guidelines for asking survey questions
• Open or closed questions?
• Response choices/measurement scales

__________________________________________________
_____
ARL Workshop on User Surveys

Slide 3
5
___________________________The Effects of
Attitudes, Behavior, Knowledge and Demographics
onQuestion Construction

• Attitudes
• Behavior
• Knowledge Questions
• Demographic Considerations

__________________________________________________
____
ARL Workshop on User Surveys

Slide 4
6
__________________________Attitudes

• Attitudes are complex and are difficult to define
  and measure
• Attitude questions often have two components
• how respondents feel and
• how strongly they feel

__________________________________________________
____
ARL Workshop on User Surveys

Slide 5
7
___________________________Behavior

• Questions refer to what respondents actually do
• Generally concerned with identification,
  quantity, time, duration, or frequency (what, how
  many, when, how long, how often)
• Asking specific questions about a long period of
  time yields imprecise answers
• Lists help respondents recall things they might
  have done

__________________________________________________
___
ARL Workshop on User Surveys

Slide 6
8
___________________________Knowledge Questions

• Can determine if respondents know enough about
  a topic to warrant asking their opinion
• May identify gaps in knowledge that need to be
  addressed
• Sometimes help to explain attitudes and
  behavior

__________________________________________________
____________________ ARL Workshop on User
Surveys Slide 7
9
___________________________Demographic
Considerations

• Knowing something about the surveys targeted
  respondents can help frame the questions and
  determine an appropriate level of specificity
• If specific responses are sought and the range of
  answers is large, ask for the exact information
  dont present a multiple choice question
• Use current words and terms
• Decide if comparability is sought. If so,
  standardize questions

__________________________________________________
_______________________ ARL Workshop on User
Surveys Slide 8
10
___________________________Understanding the
Surveys Context

• Focus the questions on the surveys specific
  purpose(s) or information sought
• Agree upon the precise data being sought in
  advance
• Clarify or define all potentially imprecise or
  ambiguous terms
• Terminology, language level, and time frame
  should be appropriate for respondents
• Stick with the least number of questions needed
  for adequate measurement

__________________________________________________
_________________________ ARL Workshop on User
Surveys Slide 9
11
___________________________Understanding the
Surveys Context (Continued)

• Standardize survey(or)s
• Use anonymous surveys to ask sensitive questions
  about personal behaviors or attitudes

__________________________________________________
______________________ ARL Workshop on User
Surveys Slide 10
12
___________________________Guidelines For Asking
Survey Questions

• Ask concrete questions that are precise and
  unambiguous
• use time periods that are related to the
  importance of the question
• Use conventional grammar, punctuation, and
  spelling
• Use the proper context when constructing survey
  questions

__________________________________________________
_____________________ ARL Workshop on User
Surveys Slide 11
13
___________________________Using Conventional
Grammar, Punctuation, and Spelling

• Use complete sentences
• Avoid abbreviations, slang and colloquialisms
• Avoid jargon and technical expressions
• Have the questions reviewed by experts in survey
  question writing
• Have the questions reviewed by potential
  respondents
• Adopt or adapt questions used successfully in
  other surveys

__________________________________________________
______________________ ARL Workshop on User
Surveys Slide 12
14
___________________________Using Proper Context

• Shorter questions take less time and often lead
  to shorter answers
• Use longer questions with lead-ins to provide
  background information and collect data on
  sensitive concerns (e.g., journal cancellations)
• Use loaded questions cautiously
• Avoid biasing words or phrases
• Avoid two-edged questions (avoid and)
• Avoid negative questions or ones that require an
  exercise in logic

__________________________________________________
____________________________ ARL Workshop on User
Surveys Slide 13
15
___________________________Open Or Closed
Questions?

• Open questions are used to solicit answers in the
  respondents own words
• Closed questions, where responses are
  pre-selected, are considered more efficient and
  reliable
• Use a checklist to decide between open and
  closed questions

__________________________________________________
______________________ ARL Workshop on User
Surveys Slide 14
16
___________________________Open Questions

• Are useful in getting unanticipated answers
• Describe the respondents, not the surveyors
  world
• Provide quotable material
• Are primarily used to make decisions about
  individuals, not groups
• may be difficult to compile, compare and
  interpret

__________________________________________________
____________________________ ARL Workshop on User
Surveys Slide 15
17
___________________________Closed Questions

• Provide standardized data that can be analyzed
  statistically
• Answers have a better chance of relevance and
  consistency over time
• Are more difficult to write and validate
• May overlook things respondents might volunteer
  in open questions

__________________________________________________
_____________________ ARL Workshop on User
Surveys Slide 16
18
___________________________Checklist For
Deciding Between Open And Closed Questions
Open Questions Closed Questions
__________________________________________________
________ ARL Workshop on User Surveys

Slide 17
19
___________________________Response
Choice/Measurement Scales

• Nominal/Categorical response choices
• Ordinal response choices
• Numerical response choices

__________________________________________________
____________________________ ARL Workshop on User
Surveys Slide 18
20
___________________________Nominal/Categorical
ResponseChoices

• Are not numerical or preferential values
• Often used to categorize respondents
  characteristics
• Findings are usually reported as percentage
  groupings

__________________________________________________
____________________________ ARL Workshops on
User Surveys Slide 19
21
___________________________Suggestions For
Constructing Nominal/Categorical Responses

• Generally try to be exhaustive
• Try to make choices mutually exclusive
• Try to use categories that are meaningful to
  respondents and will make sense when results are
  reported

__________________________________________________
_________________________ ARL Workshop on User
Surveys Slide 20
22
___________________________Ordinal Response
Choices

• Respondents are asked to rate or rank choices
• Are often used more than the other types of
  questions
• Are typically used to measure satisfaction
• Survey findings are usually reported as an
  average score or percentage

__________________________________________________
_______________________ ARL Workshop on User
Surveys Slide 21
23
___________________________Suggestions For
Constructing Ordinal Responses

• Consider five types of response options
  Endorsement/Frequency/Intensity/Influence/
  Comparison
• Determine through pre-testing if a neutral
  response is called for
• Use a 5point to 7point scale (either odd or even)
• Know when to put the negative end of the scale
  first

__________________________________________________
____________________________ ARL Workshop on User
Surveys Slide 22
24
__________________________Numerical Response
Choices

• Ask respondents to quantify a response
• Provide completely objective response
• Allow many statistical analyses to be
• performed and reported

__________________________________________________
________ ARL Workshop on User Surveys

Slide 23
25
___________________________Suggestions For
Constructing Numerical Responses

• Decide if discrete (i.e. exact) numbers
• work best
• Consider using intervals if precision is
• not needed
• Decide if it is most appropriate to offer
• numerical responses along a continuum

__________________________________________________
______ ARL Workshop on User Surveys

Slide 24
26
___________________________Conclusion - You
Should Now Have a Better Understanding Of

• Why survey questions are important
• Characteristics of the user population that
  influence the question
• asked
• The context for framing survey questions
• Guidelines for asking survey questions
• The difference between open and closed
  questions, and when to
• use each type
• The three types of response choices
  nominal/categorical
• ordinal and numerical

__________________________________________________
________ ARL Workshop on User Surveys

Slide 25
27
___________________________Related Readings

• Babbie, Earl R. Survey research methods.
  Belmont, CA Wadsworth Press, 1990.
• Bradburn, Norman M. and Sudman, Seymour. The
  current status of questionnaire design. In Paul
  P.
• Biemer et al, Measurement errors in surveys.
  New York John Wiley, 1992.
• Converse, Jean M. Survey questions
  handcrafting the standardized questionnaire.
  Beverly Hills, CA Sage Publications, 1986.
• Fink, Arlene and Kosecoff, Jacqueline. How to
  ask survey questions. Volume 2 of The survey
  kit. Thousand Oaks, CA Sage Publications,
  1985.
• Fowler, Floyd J. Survey research methods.
  Newbury Park, CA Sage Publications, 1993.
• Kosecoff, Jacqueline and Fink, Arlene.
  Evaluation basics a practitioners manual.
  Beverly Hills, CA Sage Publications, 1982.
• Schuman, Howard and Presser, Stanley. Question
  and answers in attitude surveys. New York
  Academic Press, 1981.
• Sudman, Seymour and Bradburn, Norman M. Asking
  questions. San Francisco, CA Jossey Bass, 1982.

__________________________________________________
______ ARL Workshop on User Surveys

Slide 26