Interpretive Evaluation

1
Interpretive Evaluation

• be the ball
• get in the zone

2
Todays Agenda

• SHW2 Post-Mortem
• Fitts Law a loose end from last time
• Simpler (higher-level) user modeling
• Expert system case study
• Interpretive Evaluation
• Ethnography (mainly)
• Cooperative Evaluation (a little)

3
Low Level Models

• Fitts Law
• Models movement times for selection (reaching)
  tasks in one dimension
• Basic idea Movement time for a well-rehearsed
  selection task
• Increases as distance to target increases
• Decreases as size of target increases

4
Original Experiment

• 1-D

A
W
5
Components

• ID - Index of difficulty
• ID is information theoretic quantity
• Based on work of Shannon consider less
  information (less uncertainty) with a larger
  target
• Fitts original ID log2(2A/W) nobody sure why

ID log2 (A/W 1.0)
width tolerance of target
bits result
distance to move
6
Components

• MT - Movement time
• MT is a linear function of ID
• a and b are experimental constants

MT a bID
7
Exact Equation

• Run empirical tests to determine a and b in MT
  a bID
• Will get different ones for different input
  devices and device uses

MT
ID
8
Questions

• What do you do in 2D?
• WxL rect one way is ID log2(A/min(W,L) 1)
• If we have Fitts law, then why in the GOMS-KLM
  problem did we say that the P operator has time
  1.10 s?
• Where can this be applied in user interface
  design?

9
ATOMIKFitts Law in Modern Practice

• Alphabetically Tuned and Optimized Mobile
  Interface Keyboard
• Zhai, Hunter, Smith IBM 2001
• Model keys as atoms in a molecule, use Fitts law
  to model attractive forces
• Run a Monte Carlo simulation to derive
  force-optimized layout

10
Simpler User Modeling

• How do attributes of users influence the design
  of user interfaces?
• Are there some design guidelines that we can
  derive from different attributes?

11
User Profiles

• Attributes
• attitude, motivation, reading level, typing
  skill, education, system experience, task
  experience, computer literacy, frequency of use,
  training, color-blindness, handedness, gender,
• Novice, intermediate, expert

12
Motivation

• User
• Low motivation, discretionary use
• Low motivation, mandatory
• High motivation, due to fear
• High motivation, due to interest

• Design goal
• Ease of learning
• Control, power
• Ease of learning, robustness, control
• Power, ease of use

13
Knowledge Experience

• Experience
• task system
• low low
• high high
• low high
• high low

• Design goals
• Many syntactic and semantic prompts
• Efficient commands, concise syntax
• Semantic help facilities
• Lots of syntactic prompting

14
Job Task Implications

• Frequency of use
• High - Ease of use
• Low - Ease of learning remembering
• Task implications
• High - Ease of use
• Low - Ease of learning
• System use
• Mandatory - Ease of using
• Discretionary - Ease of learning

15
DiscussionCONFIG Expert System

• The specific UI problems are not discussed in
  great detail, but it is clear that fixing the UI
  problems did not raise system use. Why?
• Does this case study support the generalizations
  about motivation and experience that we just
  made? Why (not)?
• What were the major process flaws that you
  noticed, and how could they be improved?
• Do you agree with the authors recommendations?

16
Evaluation

• Predictive
• Discount Usability (last Thurs)
• Cognitive Modeling
• GOMS family (last Tues)
• contextual (next Tues)
• Interpretive (today)
• User testing (much later)

17
Approaches

• Experimental (Lab studies, quantitative)
• Typically in a closed, lab setting
  Manipulate independent variables to see effect on
  dependent variables
• Naturalistic (Field studies, qualitative)
• Observation occurs in real life setting Watch
  process over time
  Ecologically valid

18
Interpretive Evaluation

• Experimental Formal and objective
• Interpretive More subjective
• Concerned with humans, so there is no such thing
  as an objective reality
• Do you agree?
• Sociological, anthropological approach
• Users involved, as opposed to predictive
  approaches

19
IE Beliefs

• Sees limitations in scientific hypothesis testing
  in closed environment
• Lab is not real world
• Cant control all variables
• Context is neglected
• Artificial, short tasks

20
IE Methods

• A number of different methods or techniques in
  this area exist
• Ethnography
• Our main focus
• Contextual inquiry
• More specific form of ethnography with a focus on
  asking questions
• Field study
• Common notion, often equated with ethnography
• Observational study
• Much more informal, just watching users

21
Focus Ethnography

• Deeply contextual study
• Immerse oneself in situation you want to learn
  about (anthropological and sociological roots)
• Observing people in their cultural context
• Interpretation of data is primary
• Behavior is meaningful only in context

22
Ethnographic Philosophy

• Argues that formal environment of controlled
  study is artificial --- Experimenter wields
  power over subject
• So get into working environment of user and
  learn from the user
• Interpretation is primary, not data

23
Ethnographic Objectives

• Understanding the user
• Understand goals and values
• Understand individuals or groups interactions
  within a culture
• Try to make tacit domain knowledge explicit in an
  unbiased fashion
• For UI designers Improve system by finding
  problems in way it is currently being used

24
Field Tools and Techniques

• In person observation
• Audio/video recording
• Interviews
• Wallow in the data

25
Observation is Key

• Carefully observe everything about users and
  their environment
• Think of describing it to someone who has never
  seen this activity before
• What users say is important, but also non-verbal
  details

26
Observations

• Things of interest to evaluator
• Structure and language used in work
• Individual and group actions
• Culture affecting work
• Explicit and implicit aspects of work
• Example Office work environment
• Business practices, rooms, artifacts, work
  standards, relationships between workers,
  managers,

27
Interviews Important

• Have a question plan, but keep interview open to
  different directions
• Be specific
• Create interpretations with users
• Be sure to use their terminology
• At end, query What should I have asked?
• Record interviews

28
Ethnography Steps

• 1. Preparation
• Understand organization policies and work culture
• Familiarize yourself with system and its history
• Set initial goals and prepare questions
• Gain access and permission to observe interview
• 2. Field study
• Establish rapport with users
• Observe/interview users in workplace and collect
  all different forms of data
• Follow any leads that emerge from visits
• Record the visits

Rose et al 95
29
Ethnography Steps

• 3. Analysis
• Compile collected data in numerical, textual and
  multimedia databases
• Quantify data and compile statistics
• Reduce and interpret data
• Refine goals and process used
• 4. Reporting
• Consider multiple audiences and goals
• Prepare a report and present findings

30
Ethnography AnalysisAffinity Diagram

• Write down each quote/observation on a slip of
  paper
• Put up on board
• Coalesce items that have affinity
• If they are saying similar things about an issue
• Give names to different groups (colors too)
• Continue grouping subgroups
• A hierarchy will be formed

31
Why is Ethnography Useful?

• Can help designer gain a rich and true assessment
  of user needs
• Help to define requirements
• Uncovers true nature of users job
• Discovers things that are outside of job
  description or documentation
• Allows you to play role of end-user better
• Can sit in when real users not available
• Open-ended and unbiased nature promotes discovery
• Empirical study and task analysis are more formal
  ethnography may yield more unexpected
  revelations

32
Types of Findings

• Can be both
• Qualitative
• Observe trends, habits, patterns,
• Quantitative
• How often was something done, what per cent of
  the time did something occur, how many different
  

33
Drawbacks ofEthnographic Methods

• Time required
• Can take weeks or months
• Scale
• Most use small numbers of participants just to
  keep somewhat manageable
• Type of results
• Highly qualitative, may be difficult to
  present/use
• Acquired skill learn by doing
• Identifying and extracting interesting things
  is challenging

34
Ethnomethodology

• Concurrent/informed ethnography
• Study is being done in conjunction with a system
  being developed
• Helps keep focus on user throughout design
• - Requires lots of time and coordination
• Contrast with Participatory (Scandinavian) Design
  (well discuss this a bit later)
• bring users from workplace to be designers
• basically the philosophical polar opposite of
  ethnography, but for similar ends

35
Cooperative Evaluation

• User is viewed as collaborator in evaluation, not
  a subject
• Friendly approach
• Relaxed version of think-aloud
• Evaluator and participant can ask each other
  questions

36
CE Methods

• Seeks to detect errors early in a prototype
• Experimenter uses tasks, also talks to
  participant throughout, asks questions
• Have debriefing session at end

37
Some Words on P1Understanding the Problem

• Reports due 2/6 post to project workspace
• Format reqt printable (Word, PDF, HTML)
• Besides points listed, special emphasis on
• User group analysis and implications for design
• Convincing argument that you are addressing a
  real problem or deficiency (can be speculative)
• Dont inflict technology on users just because
  its cool
• Do surveys, interviews, focus groups do
  SOMETHING that gives you the needed
  justification!
• Careful (sub)task analysis (HTA, flowchart)

38
Next on the Menu

• Cognitive modeling
• Contextual methods
• Design of Everyday Things
• Design