CS260: Research Topics in HCI Fall 2006

1
CS260 Research Topics in HCI Fall 2006

• John Canny
• UCB EECS

2
The Course

• CS260 this semester is a focused overview of
  current research in HCI. The focus is set by the
  students taking it.
• There will be at least one class presentation by
  each student, based on their area of interest.
• There is also a semester-long class project,
  either individual or pair.

3
Sample ProjectsTactile Feedback in Music
Applications
Kerry Kimes
4
Sample ProjectsUsability analysis of UC-WISE
5
MISS - Multiple-Interface Scheduling System

• CS260, Spring 2002
• Juha Johansson
• February 28

6
User Study for Designers OutpostKatie Everitt
7
Sample Projects
Privacy Management in Ubiquitous Computing

• Xiaodong Jiang
• Scott Lederer
• February 28, 2002

8
CS260 Project Checkpoint

• Livenotes
• Second Iteration of UI

Matthew Kam Orna Tarshish
9
ObjDraw A tool for use in CS61a at Berkeley

• Project Checkpoint
• Ryan Stejskal
• Feb. 28, 2002

Create a new tool that allows students to explore
one of the hardest big ideas object-oriented
programming outside of class time Similar to an
existing tool, envdraw A graphical interface to
object-oriented programming as implemented in 61a
10
Sound Visualization for the Hearing-Impaired

• Wai-ling Ho Ching
• Convenient, low cost home audiometry is desirable
  and implementable with the power of current home
  computers
• Pure tones of varying frequencies and degrees of
  loudness are played to each ear to determine
  range and degree of hearing loss

11
Source Code Visualization

• Zachary Weinberg
• Decades of history, thousands of modifications
• No easy way to find the changes you care about
• Poor collective memories

12
HCI the state of the union

• Ideas appeared in special issue of Queue Magazine
  on HCI (JFC guest editor).
• Outline brief recent history of modern HCI
• Why things are different
• What it means for interactive system design

13
Modern HCI

• We can define modern HCI as the iterative,
  user-centered design of systems.
• There are two key data points in this evolution,
  the Xerox Alto and the Xerox Star.
• They represent two very different approaches to
  design, and two very different outcomes.

14
Xerox Alto

• Began in1970, soon after PARC formed. Design
  team Alan Kay, Chuck Thacker, Butler Lampson
• Real target was a laptop (dynabook), but a
  personal computer was the closest you could
  achieve in the 1970s.
• Features
• Mouse
• Overlapping windows
• Ethernet
• But
• Still mostly text UI
• Lacked a killer app

15
Xerox Star

• Very different pedigree. Not a research
  prototype.
• Created by Xeroxs product division (Don
  Masarro).
• Goal was to support generic office work.
• Project leader David Liddle tappedexperts from
  PARC to help with Starsdesign process.
• Started with a best practice designdocument
• Scenarios
• Task analysis
• Conceptual modeling
• Rapid prototyping

16
Xerox Star

• The Stars design process is completely modern
  its a perfectly good example of best practice
  today.
• The result was a completely modern UI design (a
  WIMP interface).
• Liddle stated that the Star was a big
  improvement on its successors
• Its a fair statement. Stars UI wascopied in
  Macs and later PCs.
• Its object-oriented design wascleaner than the
  leading OSes decades later.

17
Design process

• Neither machine succeeded in the marketplace,
  although the Stars design is arguably the most
  influential of any machine on todays UIs.
• They represent two different design processes
• Visionary, techno-centric (and theory-centric)
  design, embodied in the Alto.
• User-centric, market-driven, evolutionary design,
  embodied in the Star.

18
Where we are

• Desktop PCs are primarily office (knowledge work)
  machines.
• The IT market is much broader now. Intels
  Taxonomy
• Office
• Mobile
• Health
• Home ( emerging regions?)
• The last 3 are different environments,
  top-to-bottom. The hardware is different too In
  mobile we have cell phones, PDAs, cameras, GPSes,
  portable games, iPods etc. The Health and Home
  markets seem to be still in flux.

19
Where we are

• Looking again at the new markets
• Mobile
• Health
• Home
• There are many differences in how people access
  information
• Many more short episodes
• More proactivity (reminders, alerts, automation
  etc.)
• More interaction between device and world
• Scarcity of a desktop with keyboard, screen,
  mouse
• Increased role of context where, when, who,
  history,

20
Where we are

• HCI has mostly followed an evolutionary strategy
  its target (the knowledge work environment) has
  stayed the same for most of its history.
• Were currently in a disruptive period where
  evolution gives way to revolution.

21
Technology themes
Perceptual interfaces (vision, speech, sensing)
Context modeling
22
Technology themes

• Perceptual Interfaces Translate sensed data into
  relevant (system) actions for the users
  task/activity. Sensing is everywhere in new IT
  domains (cameraphones, occupancy sensors).
• Context Is what is understood by humans when
  they interact with each other, and makes
  efficient communication possible. Very important
  for emerging IT domains like mobile and home.
• Each supports the other.

23
Context

• Traditional HCI definition (Abowd et al.)
• Place and time
• User preferences
• User activity (task)
• Most of the papers on context focus on what data
  to use, not on what to do with it.
• Alternative notion Context comprises
• User activity (what is it, and users role?)
• Situation (social convention, what would a
  stranger do?)
• This allows us to assign meaning to observations
• What are the anticipated consequences of the
  observations?

24
Perceptual Interfaces

• Perceptual Interfaces (like computer vision and
  speech) have fared poorly in knowledge work
  environments. Why?
• They are doing much better in the new domains.
• E.g. the largest market for speech software now
  appears to be health care.
• But new environments are physically challenging
  noisy, erratic lighting etc.
• But these environments are also context-rich (not
  true for the office).

25
Todays mobile phone

• This years Smartphone (free with service
  contract)
• 150-200 MHz ARM processor
• 32 MB ram
• 2 GB flash (not included)
• Windows-98 PC that boots quickly!
• Plus
• Camera
• AGPS (Qualcomm/Snaptrack)
• DSP cores, OpenGL GPU
• EV-DO (300 kb/s), Bluetooth

26
Context-Aware Face Recognition
27
Perceptual Interfaces - Vision

• Cameraphones are capable serious computer vision
  now. Right now, the vision algorithms available
  include
• Motion
• Barcodes
• OCR text (business cards etc.)
• Technically feasible
• Face recognition
• Building or streetscape recognition

28
Context-Aware Face Recognition

• Face recognition alone - 43
  accurate(state of the art computer vision)
• Context analysis alone - 50
  accurate(Face prediction from contextual data
  on the phone)
• ContextContent analysis - 60 accurate

Figure 1. (Top) Subjects with frontal pose,
(Bottom) Same
29
Context-Aware Place Recognition

• Image analysis alone - 30
  accurate
• Context analysis alone - 55
  accurate
• ContextContent analysis - 67 accurate

30
MMM2 Context to Community
31
Photo Share Guesser
32
Perceptual Interfaces - Vision

• TinyMotion is a software mouse for cameraphones.
• By moving the camera against any background,
  real-time image motion estimation provides mouse
  coordinates.Also great for games

33
Perceptual Interfaces - Speech

• Speech recognition technology has improved
  steadily in the last ten years, particularly in
  noisy environments.
• Speech was never a good match for office
  environments.
• But the mobile playing field is completely
  different.
• Mobile users often need their eyes and hands
  free, and the phone will always have a voice
  channel for telephony.

34
Speech on Mobile Phones

• Restricted speech recognition is available on
  many phones.
• Large-vocabulary recognition just appeared on
  cell phones last year (Samsung P207). Its a huge
  step. It enables the next generation of mobile
  speech-based apps
• Message dictation
• Web search
• Address/business lookup
• Natural command forms(no need to learn them)
• Most of this technology was developed in the US
  by VoiceSignal Technologies.

35
Speech for Developing Regions

• Speech is an even more important tool in
  developing regions.
• Literacy is low, and iconic (GUI) interfaces can
  be hard to use.
• Unfortunately, IT cannot help most of these
  people because they lack even more basic skills
  fluency in a widely-spoken language like English
  or Mandarin.
• This project focuses on teaching English in an
  ecologically appropriate way.
• Speech-based phones are ideal for this.

36
Speech for Developing Regions

• Speech (with headset) allows students to learn
  while working.
• It leaves their eyes and hands free, and engages
  their minds during tedious, manual work.
• Some game motifs
• Safari hear sound say the name in English
• Karoake in English
• Listen and summarize BBC, cricket etc.
• Treasure hunt leave LB clues in English
• Adventure games dialog-driven scenarios

37
Context-Awareness

• Context-awareness is the holy grail for next
  generation mobile applications
• Location (e.g., video store, kitchen) heavily
  shapesthe users likely actions. So does
  time,place, identity of friends, etc.
• These data are often inferred byperceptual
  systems.
• But when people say context theymean much
  more con-text is literallywith-the-text its
  all the other informationneeded to make sense of
  a text (or a user interface action).

38
Making more of context

• There is an enormous amount of social sciences
  that points to two sources of high-level context
• Activity based on subjects personal history and
  what they are engaged in.
• Situation a set of socially-understood normed
  environments and behaviors within them.
• In both cases there is a structure of
• People (and their roles)
• Objects
• Actions or scripts

39
Making more of context

• There is an enormous amount of social sciences
  that points to two sources of high-level context
• Activity based on subjects personal history and
  what they are engaged in.
• Situation a set of socially-understood normed
  environments and behaviors within them.
• In both cases there is a structure of
• People (and their roles)
• Objects (and their genres)
• Actions or scripts

40
Making more of context

• Dealing with context is a great challenge because
  it covers so many fields, from sociology and
  linguistics to machine learning.
• But the potential payoffs are great as well
• E.g. JCR Lickliders OLIVER (Online Vicarious
  Expediter and Responder)

41
Summary

• We are at a fork in the road in HCI. We face a
  new playing field (app. Domains) with new
  demands.
• We have an opportunity (and probably a necessity)
  to build systems that are much more
  context-aware.
• Future systems can also leverage machine
  perception to build and exploit context.
• The net result is systems that understand (and
  respond to) us much better.

42
Next time

• I propose to start on learning systems for
  Wednesday. I can cover the lecture.
• Id like a volunteer for Mondays lecture.