Asking Users and Experts

1
Asking Users and Experts

• by Xianghua Ding
• Hoang Minh Ho Dac

2
Outline

• Asking Users
• Interviews
• Questionnaires
• Asking Experts
• Heuristic evaluation
• Walkthroughs
• Cognitive
• Pluralistic

3
Asking Users

• Interviews
• Developing questions
• Planning interviews
• 4 Types of interviews
• Data analysis and interpretation
• Questionnaires

4
Developing Questions

• Avoid long questions
• Avoid compound sentences
• How do you like this cell phone compared with
  previous ones that you have owned?
• How do you like this cell phone? Have you owned
  other cell phones? If so, how do you like it?
• Avoid using jargon
• Avoid leading questions
• Why do you like this style of cell phone?
• Be alert to unconscious bias

5
Planning an Interview - Steps

• Introduction
• To introduce himself, explain the purpose and
  get consent
• Warmup session
• Using easy, non-threatening questions
• Main session
• To ask prepared questions from easy to difficult.
• Cool-off period
• a few easy questions
• Closing session
• To thank the interviewee and to clear up the
  scene
• Make the interview as pleasant as possible
• The golden rule is to be professional

6
Planning an Interview - Advice

• Dressing
• In a similar way to the interviewee if possible
• Prepare a consent form and ask the interviewee to
  sign in
• Make the equipment work
• Make sure your recorder works and know how to use
  it in advance.
• Record answers exactly

7
4 Types of Interviews

• Unstructured interviews
• Structured interviews
• Semi-structured interviews
• Group interviews

8
Unstructured Interviews

• Both interviewer and interviewee have control and
  can steer the topic
• More like a conversation
• Focus on particular topic but go into depth
• Questions are open
• No predetermined content and format
• Interviewee is free to answer questions as fully
  or as briefly as she wished
• Need a plan to make sure the main things to be
  covered

9
Unstructured Interviews- Advantage
Disadvantage

• Disadvantages
• Time consuming
• Ethical issues
• Impossible to replicate the process
• Difficult to analyze all data

• Advantages
• Generate rich data

10
Structured Interviews

• The interviewer has the most control
• Pose predetermined , closed questions
• The study is standardized
• The same questions are used with each
  participants
• Useful when study goals are clear and specific
  questions can be identified

11
Semi-structured Interviews

• Combines features of structured and unstructured
  interview
• Use both closed and open questions
• Has a basic script for guidance
• so the same topics are covered
• Probe are device to get more information
• Do you want to tell me anything else?
• Be aware not to preempt an answer
• You seem to like this color
• Accommodate silence
• Prompt the person to help her along

12
Semi-structured Interviews -An example

• Which websites do you visit most frequently?
• ltAnswer several but stresses that se prefers
  uci.comgt
• And why do you like it?
• ltAnswergt
• Tell me more about X?
• ltAnswergt
• Anything else?
• ltAnswergt
• Thanks. Are there any reasons that you havent
  mentioned?

13
Group Interviews

• Involve a small group guided by a interviewer to
  facilitate discussion
• Focus group
• Normally 3-10 people are involved
• Participants are representative of a certain type
  of users they normally share certain kind of
  characteristics
• Allows diverse and sensitive issues to be raised

14
Group Interviews Advantage Disadvantage

• Disadvantages
• Facilitators need to be skillful
• Difficult to get people together in a suitable
  location and time

• Advantages
• Method is readily understood
• Findings appear believable
• Low-cost
• Quick results
• Easily be scaled

15
Data Analysis Interpretation

• Quantitatively
• For structured interviews
• Qualitatively
• For unstructured Interviews
• A coding form may be developed
• Comments may be clustered along themes and
  anonymous quotes used to illustrate points of
  interest
• Tools such as NUDIST

16
Asking Users Questionnaires

• An alternative technique for getting users
  opinions
• Can have closed and open questions
• Strengths
• Distributed to a large number of people
• Provide evidence of a wide general opinion

17
Guidelines for Designing Questionnaires

• Make questions and instructions clear
• If possible, ask closed questions and offer a
  range of answers
• Include a no-opinion for questions that seek
  opinions
• General questions should precede specific ones
• Group related items
• Specify age as a range
• Different versions for different population
• Balance between white space and compactness
• If scales are used, the range should not overlap
• The ordering of scales should be intuitive and
  consistent

18
Example of Poorly Designed Questionnaires
1. State your age in years 2. How long have you
worked here? (check one only) 3. How long
have you use the Internet? (Check one only)
4. Do you use the Web to 5. How useful
is the Internet to you?
Purchase goods Send email Visit chatrooms Find
information
1 yr 2 yrs 3 yrs gt 3 yrs
lt 1 yr 1-3 yrs 3-5 yrs gt 5 yrs
19
Question and Response Format

• Checkbox
• Used for demographic or background data
• Respondents check an appropriate box or circle a
  response
• Ranges
• used for getting opinions
• 2 types
• 1. Likert Scales
• 2. Semantic Differential Scales

20
Example of Likert Scales

• Instruction
• In the following questions, 1 represents
  strongly agree and 5 represents strongly
  disagree. Please check only one.
• The company website is helpful
• 1 2 3 4 5
• The website color is annoying

1 2 3 4 5
? Should we mix positive questions with negative
questions? ? What is the best rating scale ?
(e.g. odd like 1-3, 1-5, or even)
21
Example of Semantic Differential Scales

• Instruction for each pair of adjectives, place
  a cross at the point between them that reflects
  the extent to which you believe the adjectives
  describe the home page. You should place only one
  cross between the marks on each line.
• Attractive _______ Ugly
• Clear _______ Confusing
• Helpful _______ Unhelpful

? How to calculate the total score?
22
Administering Questionnaires

• Two important issues
• 1. How to reach a representative sample of
  people
• 2. How to ensure a reasonable response rate
• With small number fewer than 20, 100 is often
  achieved
• With larger populations, 40 return is generally
  acceptable

• Solutions
• Tell people it is OK to complete just a part
• Include stamped, self-addressed envelope
• Explained why you need the questionnaires
• Assure anonymity
• Contact users
• Offer incentives

23
Online questionnaires

• Email
• Can be targeted to specific users
• Quick response
• Limited to text
• Web-based
• Flexible graphical design
• Errors could be corrected easily
• Immediate data validation
• Less time for data analysis
• Low cost for copying and postage
• Have random samples of respondents
• Response rate may be lower than paper form

24
Steps to develop web-based questionnaires

• Devise the questionnaire on paper, following the
  guidelines
• Identify a random sample of population. Avoid
  biased or convenience sampling
• Turning the paper questionnaire into a web-based
  version
• error-free interactive
• Accessible and readable from all online users
• Identification information handled confidentially
• User-test before distributing

25
Examples of Web-based Questionnaires

• What do you think about the questionnaires at
  this website? Are they good or bad? Why?
• http//www.perseusdevelopment.com/surveytips/sampl
  esurveys.html

26
Analyzing Questionnaire Data

• Display data graphically (e.g. bar charts)
• Often simple statistics are needed (number of
  participants, percentage of responses)
• Identify any trends, patterns or relationship
  between responses

27
Example Statistics Table
28
Example Pie Chart
29
Example (cont.)
30
Asking Expert

• Heuristic evaluation
• Introduction
• Core heuristics
• Doing heuristic evaluation
• Heuristic evaluation for web sites
• An example
• Guidelines of website
• Walk through
• Cognitive Walkthrough
• Pluralistic walkthrough

31
Heuristic Evaluation Introduction

• An inspection technique in which experts evaluate
  whether user-interface elements conform to a set
  of heuristics
• Closely related to design guidelines
• Different sets of heuristics for different
  products

32
Core Heuristics

• Nielsens
• Visibility of system status
• Match between the system and the real world
• User control and freedon
• Consistency and standards
• Help user recognized, diagnose, and recover from
  errors
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency to use
• Aesthetic and minimalist design
• Help and documentation

33
Doing Heuristic Evaluation

• Briefing session
• Experts are told what to do
• A prepared script is useful as a guide
• Ensure each person receives the same briefing
• Evaluation Period
• Experts independently inspecting the product,
  using heuristics for guidance.
• At lease 2 passes
• Give a feel of the flow of the interaction and
  the product scope
• Focus on specific interface elements
• Debriefing session
• Discuss their findings, prioritize the problems,
  and suggest the solutions

34
Heuristic Evaluation for Websites an Example

• http//www.lib.uci.edu/

35
Heuristic Evaluation for Websites an Example

• Heuristics tailored from Nielsens original set
• Internal consistency
• Is the logo, format, text , font or usage of
  terms consistent?
• Minimizing the users memory load
• Layout
• Is it compact? Is the page layout meaningful? Is
  there too much text on the page?
• ..

36
Heuristic Evaluation for Websites an Example

• Findings about the websites
• The formatting of pages and presentation of logos
  are consistent of website
• Some forms require users to recall instead of
  recognition
• http//www.lib.uci.edu/services/workshops/isform.h
  tml
• The layout is kind of complicated, kind of too
  much text
• http//www.lib.uci.edu/online/ebooks.html

37
Heuristic Evaluation for Websites - Guidelines

• Navigation
• Avoid orphan pages that are not connected to the
  home page, which lead users into dead ends
• Avoid long pages with excessive white space that
  force scrolling
• Provide navigation support
• Avoid non-standard link colors
• Provide consistent look and feel
• Access
• Avoid complex URLs
• Avoid long download time that annoys users
• Information design

38
Walkthroughs

• An alternative to heuristic evaluation
• To predict users problems without doing user
  testing
• How ? - Walk through a task with the system and
  recognize usability problems
• 2 types
• 1. Cognitive walkthroughs
• No user participation
• 2. Pluralistic walkthroughs
• Users, experts, specialists, designers,
  developers are involved

39
Steps in Cognitive Walkthroughs

• Userscharacteristics are identified
• The walkthrough team come together for task
  analysis
• The walkthrough team go through each task,
  answering the three questions
• Will users know what to do?
• Will users see how to do it?
• Will users understand from the feedback whether
  their action are correct or not?
• 4. Record what cause problems, why and how
  serious they are to users
• 5. Revise the design to fix the problems

40
An Example of Cognitive Walkthroughs

• What to Evaluate Java Home Page
• Task to find the tutorial for Java 3D
• Typical users students, developers
• Go through each step to complete the task
• At Java Home Page
• Will users know what to do ? yes, to search the
  Tutorial Section
• Will users see how to do it ? - yes, click on
  the left submenu Tutorial
• Will users understand the feedback ? yes, it
  leads to the List of Tutorials
• At The List of Tutorials

41
Steps in Pluralistic Walkthroughs

• Choose a task to evaluate.
• Take a series of screenshots to complete that
  task.
• 2. Each member of the team looks at the screen
  pictures and writes down the sequence of actions
  they would take to move from one screen to
  another
• 3. First, users present their suggest of actions
• Next, experts present their findings
• Last, developers comment
• 4. Go back to step 1 with another task

42
An Example of Pluralistic Walkthroughs

• Suppose you are a member in the
  walkthrough team.
• Purpose to evaluate the web site of
  Sea World
• Step 1
• Chosen task to find the location and open hours
• Scenarios
• There are 2 screens in the path to find Sea
  Worlds location and open hours

43
1
2

• Step 2
• You are shown the above 2 screens in the path to
  complete the task
• Can you say what action will lead you from the
  first screen to the second screen? (dont consult
  with other members in the team)
• Step 3
• Do other members in the team agree with you?
• If yes, what do you think about the interface?
  If no?

44
Cognitive vs. Pluralistic
Pluralistic
Cognitive

• Strong focus on users tasks
• Multidisciplinary evaluators with user
  participation in evaluation
• Difficult to arrange time and location for the
  walkthrough team
• Time-consuming
• Explore only a limited number of tasks

• Focus on users problems in detail
• No user involved in evaluation.
• Do not need a working prototype
• Time-consuming
• Has narrow focus, only useful for certain types
  of systems

45
Thank you!