Usability Evaluation Issues in Commercial and Research Systems

1
Usability Evaluation Issues in Commercial and
Research Systems

• Laila Dybkjær, Niels Ole BernsenNISLab,
  University of Southern Denmark
• Hans Dybkjær SpeechLogic, Prolog Development
  Center A/S
• ASIDE 2005-11-10COST Workshop, Ålborg University
• Slides available at www spokendialogue.dk/publicat
  ions/2005k/ASIDE-2005-11-10.ppt

2
Its all about design usable design ...
Users, prompts, modalities, media, ...
Usability!
3
Usability in academia and industry

• Academia New and challenging?
• Focus on advanced systems and new knowledge
• Industry Cost? ROI? Market? Customers?
• Focus on state-of-the-art and functionality
• But they have much to learn from each other
• Lots of research results to streamline for
  industry
• Lots of simple systems questions open to
  research
• Gap EU research visions vs. industrial reality

What can they learn from each other?
4
Three examples
System Traffic FAQ NICE HCA
Task / domain Road traffic information Holiday allowance information H. C. Andersens life and fairytales edutainment
Purpose Commercial CommercialGov. support Research
I/O Speech Speech Speech, gesture, 3D graphics
Language Da 50 words Da 500 words En 2000 words
Target Car drivers All employees Children 10-18
Who built it PDC PDC, NISLab NISLab and 4 other EU partners
Wide range in purpose and complexity
5
Cost and complexity
0
2
9
23
13
36
Months Traffic400 hoursSimple FAQ4000
hoursComplex NICE HCA40000 hoursVery complex
F 2005
F1
F2 2002
P1
P22005
Academic focus Prototypes Industry focus
Final systems
6
Usability evaluation criteria

• Difficult to select right criteria for given
  system
• Is purpose to compare, to investigate, or to
  define contract?
• To make proper selection, one must know the range
  and properties of criteria available
• Many usability criteria vaguely defined
• E.g. adequacy or sufficiency of
• Quantifiability often missing
• Subjective or qualitative evaluation
• New system types require new criteria
• Must be clearly defined and operationalised

Standards may emerge, but new needs keep coming
7
Core usability evaluation criteria
System Criteria Criteria
Traffic Interaction problemsCorrectness Task and domain completeness
FAQ Interaction problemsCorrectnessTransaction success Task and domain completeness
NICE HCA Conversation success NaturalnessReasoning capabilities Ease of useError handling Scope of user modellingEntertainment and education valueUser satisfaction
Clearly different focus in academia and industry
8
Usability evaluation methods

• Which one to choose depends e.g. on
• Evaluation purpose
• Resources (who, time, money)
• Stage of development process
• Examples of methods
• Walkthrough (early)
• Focus groups (early, but ok any time)
• Wizard-of-Oz (early-middle)
• Field test (late)
• Heuristic evaluation (best early but also ok
  later)
• User interviews and questionnaires (any time)

Many current practice methods!
9
Usability evaluation methods
System Methods
Traffic Walkthrough (using DialogDesigner)Semi-formal WOZ (using DialogDesigner)In-house scenario-based testExpert evaluation of domain information
FAQ Walkthrough (manually) In-house and external scenario-based testQuestionnaire on webMonitored scenario-based lab testsField data analysisExpert evaluation of domain information
NICE HCA WOZ in schools and in museumLab-test of first and second prototypePost-lab-test interviews
Industrial systems need broad range of methods
10
Data and analysis
System Data Analysis
Traffic Logfiles Problem identification via observation and feedbackLog-based analysis of problems
FAQ Logfiles trans-criptions trans-action annotations questionnaires Problem identification via ob-servation, feedback from users and domain experts, and ana-lysis of transcribed dialogues
NICE HCA Logfiles trans-criptions topic annotation Eng-lish evaluation interviews Analysis of WOZ and lab test data for design input analysis of lab tests and interviews to get users opinion and develop new criteria
Research ? more data and analysis needed
11
Electronic model ? IT tools possible

• Sketch, prompt design and recording,
  walk-through, WOZ, document, test, formal
  properties (coherence, well-formedness, ...),
  ..., you name it

A lot of knowledge and theory can be made
operational
12
Academia and industry do meet ...

• Industrial actions and challenges
• Optimise existing processes
• Automation (transcription support, annotation,
  ...)
• Use known results and theories
• Unknown effects of new technology
• Academic challenges
• Highly sophisticated technology
• New factors to analyse, define, and measure
• On-line adaptivity to users skills, expertise,
• Investigate troubles with simple systems

... even though they are also different beasts