Evaluation in ExperienceFocused HCI: Exploring the Virtual Intimate Object

1
Evaluation in Experience-Focused HCI Exploring
the Virtual Intimate Object

• Future Applications Lab
• Viktoria Institute
• Gothenburg, Sweden
• 13 January 2006
• Joseph Jofish Kaye
• Culturally Embedded Computing
• Cornell Information Science
• jofish_at_cornell.edu

2
Evaluation in Experience-Focused HCI

• Evaluation in experience-focused HCI is the poor
  cousin of design. As a field (if we are one!) we
  have excellent theoretical foundations (Dourish,
  Wright McCarthy), a wide repertoire of design
  techniques (Equator Project, IDEO) and extensive
  tools for building (CS, EE, PD, ID). But
  traditional HCI evaluation techniques (GOMS, etc)
  are inadequate for ready-at-hand technologies
  used in ways unreplicable in the lab.

3
Virtual Intimate Object (VIO)

• A small red circle in Windows taskbar, or in a
  Mac window.
• When circle is clicked, partners circle glows
  bright red, then fades over time.
• http//io.infosci.cornell.edu

• Kaye, Levitt, Nevins, Golden Schmidt.
  Communicating Intimacy One Bit at a Time. Ext.
  Abs. CHI 2005, ACM Press.
• Kaye. I just clicked to say I love you.
  alt.chi, Ext. Abs. CHI 2006, ACM Press.

4
Intimacy in HCI

• History of elegantly designed but ad-hoc
  interfaces for communicating intimacy in HCI
  hard to evaluate
• Fields Thresholds (Dunne Raby DoP2 '94)
• Feather, Scent Shaker (Strong Gaver CSCW'96)
• The Bed (Dodge CHI'97)
• UbiComp'03 Workshop on Intimacy
  www.intimateornot.org
• Digital Family Portrait (Mynatt et. al. CHI'01)
• inTouch (Brave, Ishii, Dahley CSCW'98)
• inStink, Honey Im Home (Kaye '02, interactions
  '04)

5
How much does Ariel love Eric? Count the hearts!
Disney (1997) The Special Edition Little
Mermaid Coloring Book. Golden Books Publishing.
Answer 19
6
Two VIO Studies

• Study I
• 10 couples in existing long-distance
  relationships (n20)
• 5 couples assigned to VIO (n10)
• 5 couples assigned to MinIO (n10)
• Paper logbooks pre, post-tests, daily questions
• Study II
• 80 subjects recruited through long distance
  relationship communities on LiveJournal
• All subjects took pre-survey all were asked to
  take post-survey
• Four groups VIO, VIO daily survey, daily
  survey alone, nothing
• All surveys online - Currently being analyzed

7
Experience surveys starting points

• Gaver et. al Cultural Probes
• Geertz Thick description
• Dourish, Wright McCarthy, etc experience-based
  HCI
• Iterative design Design, Build, Evaluate..

8
Thick Description

• Winking isnt just something your eye does
• Its culturally embedded
• Its only the thick description of the context
  and culture that lets us understand the role of
  the wink in sharing a conspiracy, or even
  parodying another sharing conspiracy.
• Geertz, C. (1973) Interpretation of Cultures Ch.
  1

9
Cultural Probes

• Gives context around a situation
• Originally for inspiring the design part of the
  design ? build ? evaluate ? iterative design
  cycle
• Repurposed here for inspiring the evaluation part
  of the cycle
• ltprovocative statementgtWere changing a key,
  generally unspoken assumption by proposing that
  evaluation is creative, arbitrary, and
  complex.lt/provocativegt
• Gaver et. al. Cultural Probes, interactions 6(1)
  1999

10
Three topics

• Questions/tasks about the technology being
  evaluated.
• Questions/tasks about the situation.
• Questions/tasks about instrument being used to
  gather information

11
And three kinds of questions(for organization,
not canonical)

• Context Where? When? Take a photo of How
  long? How often?
• Metaphor What TV show? What song? What colour?
  What metaphor?
• Valence Whats the best? Whats the worse?
  Whats your favorite? What should we change?
  What shouldnt we change? Draw the best, the
  worst.

12
Question Design chart
13
Getting beyond usability

• Evaluating in context requires working in context
• VIO becomes ready-at-hand, not present-at-hand,
  and so evaluation must be in context.
• Users must have significant, continuous use, and
  you must have solved the usability problems
  otherwise most of what you get is usability
  issues i.e. Mobile Probes paper
• Hulkko et. al. Mobile Probes Proc. NordCHI 04

14
Discussion

• Unpacking meanings of evaluate
• tell me if people like it
• tell me if a variable that I can track has
  changed
• make a comparison between my stated (unstated?)
  goals/values/principles and the built technology
• to tell me what to do next
• When? (Formative, iterative, final, discrete,
  continuous)

15

• Thanks to Maria Håkansson, Lars Erik Holmquist,
  Phoebe Sengers, the Culturally Embedded Computing
  Group, my subjects and my coauthors.
• Joseph Jofish Kaye
• Culturally Embedded Computing
• Cornell Information Science
• jofish_at_cornell.edu
• This talk at jofish.com/talks. VIO at
  io.infosci.cornell.edu