Conducting User Studies

1
Conducting User Studies

• Vesa Purho
• Research Analyst
• Nokia Networks

2
User Study Process
7. Share results
6. Analyse results
5. Conduct study
4. Recruit participants
3. Choose users
2. Choose method
1. Define need
3
1. Define need

• What information is needed and why
• Determining the scope of the study
• Checking that the information is not already
  available within the company

Outputs

• List of informationneeds

4
2. Choose method

• The method depends on the information need
• Timetable for the information is also important
• Different methods take different amount of
  resources
• Creating preliminary budget (work hours and
  direct costs)
• Budget places some restrictions

Outputs

• List of methods to be used
• Preliminary budget

5
Questionnaire

• A questionnaire is a set of questions sent to the
  representatives of the target audience which they
  fill in and return. Format can be paper or
  electronic.
• Questionnaires are typically used in getting
  information that can be statistically analysed
  satisfaction rates, demographic data. Therefore
  the questions typically are closed questions
  (yes/no, scale).
• Pros and cons of using questionnaires
• can be sent to a large number of people
• can be filled anonymously
• conducting the study and data analysis can be
  outsourced
• if they contain open questions, they are hard to
  interpret
• no possibility to follow-up the answers
• not all people answer questionnaires (usually
  those who have strong positive or negative
  feelings)

6
Survey

• A survey is a questionnaire that is completed in
  an interview (phone or face-to-face).
• Surveys are typically used to get information on
  why users are satisfied or dissatisfied with the
  documentation or to find out areas of development
  in the documentation.
• Pros and cons of using surveys
• can contain open questions
• possibility to follow-up on answers
• provide information that was not anticipated
• don't provide information on what the users
  actually do
• more expensive and harder to arrange than
  questionnaires
• may be hard to interpret the reasons behind
  individual answers

7
Focus Group

• A focus group is a meeting with user
  representatives (8 - 12) from different
  customers. The meeting has a clear focus, an
  issue to be discussed, but also some freedom in
  discussion is allowed.
• Focus groups are typically used to gather
  improvement suggestions or requirements for
  documentation, or getting users' opinions on new
  ideas and prototypes.
• Pros and cons of using focus groups
• the participants can discuss on different
  alternatives and make a consensus decision
• the group can prioritise development areas for
  you
• provide information that was not anticipated
• don't provide information on what the users
  actually do
• one dominant participant may affect others
• if the facilitator for the discussion is not
  skilled, the discussion may go away from the
  actual subject

8
Customer Partnering

• Customer partnering is a series (4 - 6) of focus
  groups with participants from one customer.
• Customer partnering is typically used to involve
  customer in designing the documentation of a new
  product.
• Pros and cons of using customer partnering
• the participants can be given home work to be
  done between meetings (to keep log books, for
  example)
• the participants can ask their co-workers opinion
  on matters between meetings
• the people involved in customer partnering are
  motivated to participate in the work
• you only meet representatives from one customer
• it may be hard to find out motivated people to
  participate so that they are willing to do their
  homework
• you don't observe what the users actually do

9
Contextual Inquiry

• Contextual inquiry means visiting users at their
  workplace and observing their daily work (seeing
  the context of their work environment)
• Contextual inquiry is typically used to find out
  what users actually do and how they use
  documentation in real life.
• Pros and cons of using contextual inquiry
• you can see how they actually do their work and
  what tasks they have
• you can see how they search for information
• you can ask them why they are doing a particular
  task
• contextual inquiries take a lot of resources and
  time to conduct and the analysis of the data is
  also time consuming
• it may be difficult to get the permission to
  visit a user site because of confidentiality
  issues
• you only see the tasks the users happen to do
  during the visits

10
3. Choose users

• Identify user groups
• various experience levels and documentation
  needs
• Identify customers (internal and/or external)
• amount depends on the method and budget
• Create final budget

Outputs

• List of user groupsand customers
• Final budget

11
4. Recruit participants

• Decide who on your side is going to take part in
  the study
• People responsible for customer relationships
  should be involved in recruiting
• Phone or mail the selected persons asking their
  approval and appropriate time (if not a mass
  questionnaire)
• Be prepared for rejections

Outputs

• List of participants

12
5. Conduct study

• Collect background information on the customers
  if you are going to meet with them
• Be prepared to answer questions concerning your
  product itself
• Be aware what you can say and promise to the
  customers. Don't promise something you yourself
  can't do.

Outputs

• Results of thestudy

13
6. Analyse results

• Compare the results against the needs defined in
  phase 1
• Analyse statistical information, if any
• Prioritize and plan corrective actions
• Corrections can be done either as a part of a
  normal development project or in a separate
  project

Outputs

• Analysis of theresults
• Plan for changes

14
7. Share results

• Share the results with other groups (RD,
  Marketing etc.)
• Share with other documentation groups
• Report back to the customer on the results and
  what actions are initiated
• Different communication sets for different
  purposes

Outputs

• Study report
• Slides

15
Literature

• Hackos JoAnn, Redish Janice User and Task
  Analysis for Interface Design.
• Hackos JoAnn et.al Customer Partnering Data
  gathering for complex on-line documentation. IEEE
  Transactions on Professional Communication, vol.
  40 number 2
• Hom James The Usability Methods Toolbox
  (http//www.best.com/jthom/usability/)
• Porter Lynnette et.al Research Strategies in
  Technical Communication
• Templeton Jane The Focus Group