Using%20Hand-Held%20Computers%20and%20PCs%20Together:%20The%20Pebbles%20Project

1
Using Hand-Held Computersand PCs TogetherThe
Pebbles Project
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. Myers
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

2
Multiple Devices

• Most of our time spent in places where there is
  embedded technology
• Offices, meeting rooms, classrooms, homes
• Often multiple devices will be available
• Mobile phone and PDA and a PC
• Multiple peoples PDAs
• PDAs in a Smart Room

3
Handhelds will be communicating

• 802.11
• BlueTooth
• Cell-phone network
• (Infrared)

4
Premises of our Research

• With the coming wireless technologies,
  connecting the PCs and PDAs together will no
  longer be an occasional event for
  synchronization. Instead, the devices will
  frequently be inclose, interactive
  communication.

• Brad Myers, Using Hand-Held Devices andPCs
  Together, Comm. ACM,Vol. 44, No. 11. Nov.,
  2001. pp. 34 - 41.

5
Research Agenda

• How can multiple devicesbe used effectively
  together,at the same time?
• How can the user interface and functionality be
  spread across multiple devices?

6
Research Agenda

• How can multiple devicesbe used effectively
  together,at the same time?
• How can the user interface and functionality be
  spread across multiple devices?

Multi-MachineUser Interfaces
7
Example Power Point Control

• Use PC to give the presentation
• Use hand-held to control the PC
• Two-way communication
• Hand-held shows picture of slide,notes, list of
  titles, timer, etc.

8
Pebbles is
DAs for ntry of oth ytes and ocations
from xternal ources.

• P
• E
• B
• B
• L
• E
• S

http//www.pebbles.hcii.cmu.edu/
9
Handhelds in an Office

• Augment desktop applications
• Use multiple devices at the same time

10
Handhelds in Meetings

• Attendees use handhelds to interact andannotate
  presentation
• Augment collaboration
• Take notes

11
Handhelds in Classrooms

• Data projectors for instructors slides
• Students could have computers for
• Notetaking linked to instructors slides
• In-class testing
• Running simulations

12
Handhelds in Homes

• Interact with embedded computation
• Smart homes
• Not just speech and vision as interfaces
• Interact with appliances, lights, etc.
• Personal Universal Controller (PUC)

13
Office Use (For Individuals)

• How can handhelds augment desktop applications?

14
Applications for Individuals

• Extra input and output devices have been shown to
  be useful
• But can be expensive and hard to configure
• People have PDAs and are attached to PC
• For example, cradles for recharging
• Customizable, extensible
• Extend desktop applications

15
Scrolling with the PDA

• For scrolling using the non-dominant hand
• Studies showed parallel and efficient uses of
  both hands together
• Generates Windows scrolling events

16
Results of Study of Scrolling with PDA

• Scrolling with buttons on PDA was fastest
• PDA scrollers similar to mouse speed
• Using 2 hands is effective!

17
Remote Clipboard

• Transfer information between PDA and PC
• Connects their clipboards together
• Transfer content or reference
• Works with all applications
• Also between multiplecomputers

18
Shortcutter

• User-created panels of controls
• Create custom interfaces and extensions to PC
  applications
• And then take them with you
• Direct manipulationfor edit, then setproperties
• Palm or PocketPC

19
Shortcutter Widgets

• Buttons
• Sliders
• Knobs
• Mouse pad
• Graffiti Pad (Palm)
• Gesture panel

20
Shortcutter Actions

• Send any keyboard key, mouse button, scrolling
  action or string to PC
• Open a file or URL
• Run an application
• Invoke any PC menu or button
• Windows message
• Recorded
• Switch to a different Shortcutter panel
• Control the Mouse

21
Shortcutter Actions, cont.

• Control external devicesthrough PCs serial port
• Directly (e.g., projectors)
• X-10 for electrical devices
• Macro
• Can be multi-application
• Application-specific
• Same button, different messages
• Useful for application setsbrowsers, compilers

22
More Scenarios of Use

• Lean-back mail reading
• Controlling WinAmp
• and many others

23
Study of Individual Use

• Time to tap on button depended on size
• Few errors
• People often didnt look at PDA

24
Study of Individual Use

• Moving hands to both PDA and mouse only about 15
  slower than just moving to the mouse

msec
1H Keyboard-gtMouse
728
1H Keyboard-gtPDA
744
1H Mouse-gtKeyboard
701
slower
1H PDA-gtKeyboard
639
Keyboard -gt MousePDA
838
15.1
MousePDA -gt Keyboard
791
12.8
25
Use in Meetings

• Enhance groups collaboration and control

26
Original ApplicationRemote Commander

• Allow PDAs to control a PC
• Can be used with any application
• Uses the standard (single) cursor
• Dont have to jump up and grabmouse
• Perform all mouse andkeyboard functions
• Use PDA like touchpad
• Graffiti or our ownpop-up keyboard

27
PocketPC version

• Get PCs screen onto PocketPC
• Full view, or one-to-one zooming
• Scroll with iPaqs buttons

28
Scribble

• Multiple people draw on top of whatever on PC
  screen, not just PowerPoint
• Each user has own cursorand color
• Save by PrintScreen
• Erase by refresh

29
SlideShow Commander

• For PowerPoint
• PC Only Windows 95, 98, NT, 2000, ME, XP
• Use PC to give the presentation
• Full features of PowerPoint
• Use hand-held as remote control for PC
• Can wander away from the keyboard
• Two-way communication
• View, Navigate, Highlight, Time

30
View on Hand-Held slide

• See thumb-nail of current slide
• Black and white or color

31
View on Hand-Held notes

• See the notes of the current slide

32
View on Hand-Held titles

• See the list of titles

33
View on handheld Time

• Multi-function timer
• View large
• And in corner

34
Control Demonstrations on Palm

• Control PC applications and external devices
• Uses Shortcutter
• Easy demo and resume show

35
Control Demonstrations on PocketPC

• Task list of PC applications
• Tap brings one to front
• PowerPoint continuesbehind easy to resume

36
Commercialized

• SlideShow Commandercommercialized by
• Available at Office Depot,CompUSA, etc.
• Palm and PocketPC

Synergy Solutions, Inc. http//www.synsolutions.co
m/
http//www.slideshowcommander.com/
37
Studies of Presentations

• Summer study of 2 HCI Masters students
• Contextual Inquiry of 9 presentations
• Audience 10 to hundreds
• 7 used PowerPoint, 4 used NetMeeting
• Found 220 breakdowns
• Most were minor problems
• Averaged 8.7 lost minutes per talk (14.5)
• Designed SlideShow Commander to eliminate some of
  these problems

38
MultiCursor

• For special applications that are aware of
  multiple inputs
• E.g. Shared Whiteboard
• Single Display Groupware
• Multiple people, one display
• A number of interestingissues
• Palettes, widgets
• Section handles
• Undo

39
Chat

• Communicate to another PDA user through the PC
• PC serves as a conduit
• For side notes and messages
• For example, in negotiationmeetings
• Send to all or to a specificperson

40
Command Post of the Future

• Large DARPA funded project
• Make commanders more effective

41
Study of Laser Pointers

• Studied properties of laserpointer interaction
  techniques
• Hand-wiggle ?8 pixels
• Delay until moving average stable ?1.5 sec
• Delay until target acquired ?1.5 sec
• Different devices and poses do not help much

42
Tap Speed Study

• Tapping directly onSmartBoard is fastest
• Semantic Snarfing next
• But high error rate
• Then regular mouse
• Laser pointing slowest
• Used separate physicalbutton on handheld

43
Semantic Snarfing

• Interacting at a distance
• Grab contents to handheld
• Picture
• Menus
• Text
• Re-visualization
• Magic Lenses

44
Classroom Use

• Using handheld computers in classrooms
• For testing
• Improve large lectures
• Provide immediate feedback to instructor

45
Hardware

• HP donated 110 Jornada 680 and 100 720 computers
• Windows CE
• Lucent donated Wavelan wireless cards

46
Context

• Collaborator Prof. David Yaron of Chemistry
• Chemistry 09-106 Modern Chemistry II
• Spring, 2000 and Spring, 2001
• About 90 students
• Mostly freshmen
• Loaned one Jornada and Wavelan card to each
  student for the whole semester
• So would get used to using it
• Offset technical difficulties with benefits to
  student

47
Wireless Andrew

• CMU has almost complete coverage of 802.11b
  support
• Wavelan
• Donations from Lucent, etc.
• Funding from state of Pennsylvania, etc.
• Can use computers wirelessly in all classrooms
  and offices
• Most lawns, dorm rooms, etc.
• Windows, Mac, Linux, Windows CE support

48
Pebbles Tests

• Create through html pages
• Arbitrary html in questions and answers
• Pictures, formatting, links
• Use FrontPage, etc. to author
• Embed tags to show question and answers

49
Concept Test

• Students allowed to answer multiple times
• Server stores all answers
• Prof. Yaron displayedquestions and answers
  onthe board
• Evidence that significantlycontributes to
  learning
• Mazur, E. (1997).Peer InstructionA User's
  Manual, Prentice Hall.

50
Instructors View

• In class, instructor sees a chart of answers
• Optionally refreshes every 3 seconds
• May be projected forwhole class tosee
• Top displaysinstructions forstudents

51
Results

• Unfortunately, no data (yet) about effects on
  learning
• Students thought concept tests were a valuable
  part of the class
• Students significantly preferred using handhelds
  for concept tests.

Raise Hands Hold up cards Handhelds
25 4 63
52
Home Use

• Personal Universal Controller
• Help for people with muscular disabilities

53
Personal Universal Controller

• Two-way communication
• Appliances describe their functions
• Handheld PUC creates interface based on
  descriptions
• Handheld PUC controls the appliance
• Appliance sends back status as feedback

Specifications
Control
Feedback of Status
54
Initial Experiments

• Use real PocketPC
• Pretend that controls devices

55
PocketPC results

• For both appliances, users of actual interfaces
• Took about twice as long
• Made at least twice as many mistakes as users of
  the handheld interfaces
• Needed external help five times more often

56
Current Steps
Specification Language lt?xml version"1.0"
encoding"UTF-8"?gt ltspec xmlns"puc.xsd"
xmlnsxsi"http//www.w3.org/2001/XMLSchema-instan
ce" xmlnsxsd"http//www.w3.org/2001/XMLSch
ema" name"Audiophase 5 CD Stereo"gt
ltgroupingsgt ltstate name"PowerState"gt
lttype name"OnOffTypegt ltvalueSpacegt
ltboolean/gt lt/valueSpacegt
ltvalueLabelsgt ltmap value"false"gt
ltlabelgtOfflt/labelgt lt/mapgt
ltmap value"true"gt
ltlabelgtOnlt/labelgt lt/mapgt lt/valueLabelsgt
lt/typegt ltlabelsgt ltlabelgtStereo
Powerlt/labelgt ltlabelgtPowerlt/labelgt ltlabel
gtPowrlt/labelgt ltlabelgtPwrlt/labelgt
lt/labelsgt ltprioritygt10lt/prioritygt
lt/stategt

• XML specification language and protocol for
  describing appliances
• Create panels automatically from the
  specification
• Find realappliances whichwe can control

57
Current Work Handhelds forPeople with Muscular
Disabilities

• Using handhelds as interface to PCs
• People with Muscular Dystrophyhave fine-motor
  control but losegross motor control
• Difficulties with mouse and keyboard,but stylus
  OK
• Handhelds as interface to other devices
• Control room lights, telephone, wheelchair, etc.

58
Remote Commander Adaptations

• Faster Acceleration
• More flexible tapping
• Turn off key repeat
• Multiple Keyboards

59
General Architecture
On the PC
Any PC app.
BradsPalm Pilot
RemoteCmd
Windowseventstream
PebblesPC
RobsPocket PC
MultiCursor
PebblesDraw
Serial,IR orsockets
BensWindows CE
Direct connection or sockets
SlideshowCommander
PowerPoint
OLEAutomation
Various PDA apps
Various dlls
60
Protocols

• Can use Pebbles protocols to develop your own
  application
• Libraries for Palm, Windows CE, PC
• Independence from communication medium
• Use by
• PalmAmp from IronCreek Software
• Intel research
• others
• Windows messages or sockets

61
Downloads

• Most of this software is available for free
  downloading
• First release in Feb, 1998
• Just released version 5 (Feb, 2002)
• Downloaded over 30,000 times
• About 200 times a week

http//www.pebbles.hcii.cmu.edu/
62
Conclusions

• Handhelds are becoming ubiquitous
• Connecting technologies will improve
• Important to study how can be used when connected
  to computers and each other
• Multi-Machine User Interfaces will be
  increasingly important
• People will want to use the most convenient
  device for their information and control

63
Thanks to Our Sponsors!

• Supported by grants from
• DARPA
• Microsoft
• Pittsburgh Digital Greenhouse
• NSF
• And equipment grants from
• Hewlett Packard
• Lucent Technologies
• Palm Computing
• Symbol Technologies

• IBM
• SMART Technologies, Inc.
• Synergy Solutions, Inc.
• Handango

64
Using Hand-Held Computersand PCs TogetherThe
Pebbles Project
Thank you!
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. Myers
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles