PS50118

1
PS50118 Interacting with Technology

• Laboratory vs. Field
• Usability Evaluation
• Jason Cooper

2
Outline

• What is Usability?
• Carrying out Usability Testing
• Usability Evaluation of Mobile Devices
• The Comparison
• Conclusion

3
What is Usability?

• Commonly considered to be a way of ensuring that
  systems that promote interactivity with a user
  are easy to learn, effective to use, and
  enjoyable from the users perspective (Preece et
  al 2002)
• ISO 9241-11 states that usability refers to the
  extent to which a product can be used by
  specified users to achieve specified goals with
  effectiveness, efficiency and satisfaction in a
  specified context of user

4
Usability Goals (Preece et al 2002)

• Effectiveness This goal refers to how good the
  system itself is at doing what it is supposed to
  do.
• Efficiency Refers to how the system supports
  the users in carrying out their activities and
  whether they are able to use the system
  productively once they have gained enough
  experience.
• Safety Is concerned with the way in which the
  user is protected from dangerous conditions and
  undesirable situations
• Utility Refers to the way that the system is
  capable of providing the correct kind of
  functionality to the user at the right time, such
  that they are able to do what they need or want
  to.
• Learnability Considers how easy it is for a user
  to become competent in the use of a system.
• Memorability Refers to how straightforward it
  is for the user of the system to remember how to
  use it once it is initially learned.

5
Usability Testing The Lab 1

• Traditionally usability testing is carried out in
  a controlled environment where the product is
  tested to determine whether it can be considered
  usable.
• The controlled environment usually consists of a
  laboratory where a set of pre-planned activities
  or scenarios can be run and repeatedly measured.
  
• Its goal is to assess whether the product will
  do what it is intended to do.

6
Usability Testing The Lab 2

• Data collected includes opinions of users of the
  system and performance on the set of activities.
• Quantitative performance measures are gathered
  which all for the following types of data to be
  produced
• Time taken by the user to complete a specific
  activity.
• Time to complete an activity after a specified
  time away from the product
• Number and type of errors per activity
• Number of errors per specified unit of time
• Number of times the user had to navigate to the
  online help or manuals
• Number of users making a particular error
• Number of users completing a specific activity
  successfully

7
Usability Testing The Field

• Usually conducted to determine how a product or
  prototype is adopted and use by people in their
  working and everyday lives.
• Length of time that testing can taken within the
  field can vary from just a few minutes to months
  or in some cases even years depending on what the
  product that that is being tested.
• Provides predominantly qualitative data such
  descriptions of peoples behaviours and
  activities.
• Collected by observing and interviewing users,
  collecting video, audio and field notes that
  attempt to detail what has occurred in the
  environment during the testing,

8
Usability Evaluation of a Mobile Device 1

• Shift in focus from the lab to the field.
• Nielsen et al (2002) asserted that all mobile
  devices should always been evaluated with a
  realistic and natural setting.
• Initial reason for this were that it was thought
  lab testing was unlikely to be able to find all
  problems that occur in real mobile usage (Johnson
  P. 1998)
• There also appears to be an implicit assumption
  that usability of a mobile device could only be
  properly evaluated in the field. (Gregory et al
  2000 Brewster 2002)

9
Usability Evaluation of a Mobile Device 2

• However there still remained a significant
  preference for lab based evaluation with 71
  being undertaken in the lab and 19 being
  conduced within the field. (Kjeldskov 2005)
• Reasons for this included
• Field was considered to be time consuming in
  terms of organisation and collection of data.
• Complication of data recording
• Lack of control
• Hard to know whether all was evaluated that
  should of been evaluated.
• Whereas laboratory evaluations
• Controlled conditions
• Clear set tasks
• Peaceful space that enabled concentration
• Control over activities and monitoring
• Special equipment

10
The Comparison

• Different empirical studies were found that
  attempt to compare usability evaluation of mobile
  systems in different settings, however they
  provide different results, but all attempt to
  focus on the number, type and severity of mobile
  device usability problems that are found in the
  relative settings.
• Kjeldskov et al (2004) Is it Worth the Hassle?
  Exploring the Added Value of Evaluating the
  Usability of Context-Aware Mobile Systems in the
  Field.
• Kaikkonen et al (2005) Usability testing of
  mobile applications A comparison between
  laboratory and field testing
• Nielsen et al (2006). It's worth the hassle! the
  added value of evaluating the usability of mobile
  systems in the field.

11
Field, its not worth the hassle! 1

• Laboratory evaluation discovered the exact same
  number of usability problems as was discovered in
  the field.
• There was a lack of control that was experienced
  within the field.
• Both the field and laboratory evaluations were
  able to deliver context-aware related problems,
  which contradict with some literature that
  suggest context-aware problems are better
  acquired in a field setting.

12
Field, its not worth the hassle! 2

• In conclusion the two studies reported that
• Realistic aspects were not a problem.
• Possibility of Lab problems being false
  positives.
• Proposal of use of field studies in other areas
  of the development lifecycle, providing a better
  insight into what was needed from the system in
  the first place.

13
Field, its worth the hassle! 1

• Nielsen et al (2006) proposed that the
  contradictory results in Kjeldskov et al (2004)
  and Kaikkonen et al (2005) reports were possibly
  due to
• Low number of test subjects.
• Same data collection techniques were not employed
  in the field as it was in the lab.
• Conflicting procedures.

14
Field, its worth the hassle! 2

• Nielsen et al (2006) carried out a comparison
  study of field and evaluation usability
  evaluation of mobile device using the similar
  conditions and same data collection equipment it
  showed
• Usability problems categorised as relating to
  either cognitive load or interaction style were
  identified only in the field evaluation
• Reason given this was that field enabled
  realistic setting which in turn meant the user
  become frustrated easier.
• The nature of laboratory setting was also said to
  increase the mental demands and frustration level
  of the participants significantly.
• When both the evaluations were conducted in the
  same way, field was more successful at
  identifying the more significant usability
  problems
• In conclusion although the cost, complexity and
  amount of time it takes to carry out a field
  evaluation is a down side, Nielsen et al consider
  the added value gained in terms of the capability
  of field evaluation to provide usability issues
  not detected in the laboratory setting makes
  field evaluation worthwhile.

15
Conclusion 1

• So is it worth it?
• Yes If we are able to detect usability problems
  in the field that are not detected in the lab
  then we must undertake a field study.
• But why are there so little people doing it then?
• This is for all the reasons outlined in the
  presentation.
• Costly in terms of money and time
• Little Control
• People wonder still is it worth while
• Complicated
• We must also remember that mobile systems are
  relatively new, so people are still use to doing
  it in the lab, but as more take place within the
  field I believe it is likely more advantages will
  emerge.

16
Conclusion 2

• However alternatives do exist, whereby the field
  is simulated in the lab. (D. Svanæs ???).
• Hybrid Research Strategy with a full-scale
  simulated ward environment created with the help
  of health workers.
• With the aid of video recording they were able to
  observe details in patient-doctor interaction and
  in technology that were overlooked in the field
  study. However the field study gave a much
  richer picture.
• Conclusion is that both the lab and field
  supplement each other and it is the combination
  that provide valuable insights that can not be
  gained from one method alone.

17
References

• Preece, J., Rogers, Y., Sharp, H., (2002)
  Interaction design beyond human-computer
  interaction. West Sussex Wiley
• Nielsen, C. M., Overgaard, M., Pedersen, M. B.,
  Stage, J., and Stenild, S. 2006. It's worth the
  hassle! the added value of evaluating the
  usability of mobile systems in the field. In
  Proceedings of the 4th Nordic Conference on
  Human-Computer interaction Changing Roles (Oslo,
  Norway, October 14 - 18, 2006). A. Mørch, K.
  Morgan, T. Bratteteig, G. Ghosh, and D. Svanaes,
  Eds. NordiCHI '06, vol. 189. ACM, New York, NY,
  272-280. DOI http//doi.acm.org/10.1145/1182475.1
  182504
• Kjeldskov, J., Skov, M. B., Als, B. S. and Høegh,
  R. T. (2004) Is it Worth the Hassle? Exploring
  the Added Value of Evaluating the Usability of
  Context-Aware Mobile Systems in the Field. In
  Proceedings of the 6th International Mobile HCI
  2004 conference. LNCS, Springer-Verlag.
• Kaikkonen, A., Kallio, T., Kekäläinen, A.,
  Kankainen, A. and Cankar, M. (2005) Usability
  testing of mobile applications A comparison
  between laboratory and field testing. Journal of
  Usability Studies, 1(1)4--16.
• Baillie, L. (2003) Future Telecommunication
  Exploring actual use, In Proceedings of IFIP TC13
  International Conference on Human-Computer
  Interaction, (INTERACT '03). IOS Press
• Abowd G, D,. Mynatt, E,D Charting past, present,
  and future research in ubiquitous computing, ACM
  Transactions on Computer-Human Interaction
  (TOCHI), v.7 n.1, p.29-58, March 2000
   doigt10.1145/344949.344988