HCI Evaluation

1
Session 5

• HCI Evaluation

2
Outline

• Explain key evaluation concepts terms.
• Describe the evaluation methods and techniques
• Planning evaluations.

3
Four types of evaluation (Preece et al (2002)

• quick and dirty
• usability testing
• field studies
• predictive evaluation

4
Types of evaluation

• quick dirty evaluation describes the common
  practice in which designers informally get
  feedback from users or consultants to confirm
  that their ideas are in-line with users needs
  and are liked.
• done any time.
• Emphasises fast input to the design process
  rather than carefully documented

5
Field evaluation

• Usability testing in the field involves
  recording typical users performance on typical
  tasks in controlled settings
• The users are watched recorded on video their
  key presses are logged. Eye movements can be
  recorded using an eye-tracker.
• This data can be used to measure time on
  task,identify errors help explain why the users
  did what they did.
• User satisfaction questionnaires interviews are
  used to gauge users opinions.

6
Predictive evaluation

• Experts predict usability problems.
• A key feature of predictive evaluation is that
  users need not be present
• Relatively quick inexpensive

7
Determine the goals (Preece et. Al),,2002)

• What are the high-level goals of the evaluation?
  
• Who wants it and why?
• The goals influence the paradigm for the study
• Some examples of goals
• Identify the best metaphor on which to base the
  design.
• Check to ensure that the final interface is
  consistent.
• Investigate how technology affects working
  practices.
• Improve the usability of an existing product .

8
Evaluating and interpreting data

• Preece et al give the following list for issues
  to be considered
• - Reliability can the study be replicated?-
  Validity is it measuring what you thought?-
  Biases is the process creating biases?- Scope
  can the findings be generalized?- Ecological
  validity is the environment of the study
  influencing it - e.g. Hawthorn effect

9
And similarly on ethical issues

• Develop an informed consent form
• Participants have a right to- know the goals of
  the study- what will happen to the findings-
  privacy of personal information- not to be
  quoted without their agreement - leave when they
  wish - be treated politely

10
Select your test subjects with care

• More attention needs to the impact of individual
  differences in HCI (Dillon and Watson, 1996, User
  analysis in HCI-the historical lessons from
  individual differences research. International
  Journal of Human-Computer Studies 45 619-637
• Research in the Digital Media network at the
  University of Sunderland (Mark Hindmarch) has
  highlighted the need for care in choosing
  subjects
• Individual differences in subjects may affect the
  evaluation

11
HCI Evaluation of Handheld Software

• An interesting discussion on the design and
  evaluation of hand-held devices such as palm
  pilots can be found at
• www.ehards.com/nova/papers/hci.pdf 2001

12
Evaluation activity

• You can use the evaluation form from this web
  site on your evaluations
• http//www.acm.org/7Eperlman/question.cgi?formQU
  IS
• At the beginning of your tutorial spend 20 30
  minutes assessing the HCI design of 2 software
  applications that you use
• The applications should be 1 small and 1 large
• It would be better if the application should not
  be very well-established and highly tested
• The representative task or tasks that you choose
  should take between 10 and 15 minutes in order to
  give you enough but not too much to look at.