The Pebbles Project General Overview: Using HandHeld Computers and PCs Together

1
The Pebbles ProjectGeneral OverviewUsing
Hand-Held Computers and PCs Together
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University

• Brad A. MyersCarnegie Mellon University
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

SUNY Technology Conference 2002
2
Multiple Devices

• Most of our time spent in places wherethere is
  embedded technology
• Offices, meeting rooms, classrooms, homes
• People have handheld devices
• Palm, PocketPC, cell phone
• Often multiple devices will be available
• Mobile phone and PDA and a PC
• Multiple peoples PDAs
• Smart Classrooms
• PDAs in a Smart Room

3
Handhelds will be communicating

• 802.11
• Wi-Fi
• BlueTooth
• Cell-phone network
• 3G
• (Infrared)

4
Premises of our Research

• With the coming wireless technologies,
  connecting the PCs and PDAs together will no
  longer be an occasional event for
  synchronization. Instead, the devices will
  frequently be inclose, interactive
  communication.

• Brad Myers, Using Hand-Held Devices andPCs
  Together, Comm. ACM,Vol. 44, No. 11. Nov.,
  2001. pp. 34 - 41.

5
Research Agenda

• How can multiple devicesbe used effectively
  together,at the same time?
• How can the user interface and functionality be
  spread across multiple devices?

6
Research Agenda

• How can multiple devicesbe used effectively
  together,at the same time?
• How can the user interface and functionality be
  spread across multiple devices?

Multi-MachineUser Interfaces
7
Example Power Point Control

• Use PC to give the presentation
• Use hand-held to control the PC
• Two-way communication
• Hand-held shows picture of slide,notes, list of
  titles, timer, etc.
• Easy to switch to demo
• Easy to click on movies and sounds

8
Pebbles is
DAs for ntry of oth ytes and ocations
from xternal ources.

• P
• E
• B
• B
• L
• E
• S

http//www.pebbles.hcii.cmu.edu/
9
Handhelds in an Office

• Augment desktop applications
• Use multiple devices at the same time

10
Handhelds in Meetings

• Attendees use handhelds to interact andannotate
  presentation
• Augment collaboration
• Take notes

11
Handhelds in Classrooms

• Data projectors for instructors slides
• Students could have computers for
• Notetaking linked to instructors slides
• In-class testing
• Running simulations

12
Handhelds in Homes

• Interact with embedded computation
• Smart homes
• Not just speech and vision as interfaces
• Interact with appliances, lights, etc.
• Personal Universal Controller (PUC)

13
Office Use (For Individuals)

• How can handhelds augment desktop applications?

14
Applications for Individuals

• Extra input and output devices have been shown to
  be useful
• But can be expensive and hard to configure
• People have PDAs and are attached to PC
• For example, cradles for recharging
• Customizable, extensible
• Extend desktop applications

15
Scrolling with the PDA

• For scrolling using the non-dominant hand
• Studies showed parallel and efficient uses of
  both hands together
• Generates Windows scrolling events

16
Results of Study of Scrolling with PDA

• Scrolling with buttons on PDA was fastest
• PDA scrollers similar to mouse speed
• Using 2 hands is effective!

17
Remote Clipboard

• Transfer information between PDA and PC
• Connects their clipboards together
• Transfer content or reference
• Works with all applications
• Also between multiplecomputers

18
Shortcutter

• User-created panels of controls
• Create custom interfaces and extensions to PC
  applications
• And then take them with you
• Direct manipulationfor edit, then setproperties
• Palm or PocketPC

19
Shortcutter Widgets

• Buttons
• Sliders
• Knobs
• Mouse pad
• Graffiti Pad (Palm)
• Gesture panel

20
Shortcutter Actions

• Send any keyboard key, mouse button, scrolling
  action or string to PC
• Open a file or URL
• Run an application
• Invoke any PC menu or button
• Windows message
• Recorded
• Switch to a different Shortcutter panel
• Control the Mouse

21
Shortcutter Actions, cont.

• Control external devicesthrough PCs serial port
• Directly (e.g., projectors)
• X-10 for electrical devices
• Macro
• Can be multi-application
• Application-specific
• Same button, different messages
• Useful for application setsbrowsers, compilers

22
More Scenarios of Use

• Lean-back mail reading
• Controlling WinAmp
• and many others

23
Use in Meetings

• Enhance groups collaboration and control

24
Original ApplicationRemote Commander

• Allow PDAs to control a PC
• Can be used with any application
• Uses the standard (single) cursor
• Dont have to jump up and grabmouse
• Perform all mouse andkeyboard functions
• Use PDA like touchpad
• Graffiti or our ownpop-up keyboard

25
PocketPC version

• Get PCs screen onto PocketPC
• Full view, or one-to-one zooming
• Scroll with iPaqs buttons

26
Scribble

• Multiple people draw on top of whatever on PC
  screen, not just PowerPoint
• Each user has own cursorand color
• Save by PrintScreen
• Erase by refresh

27
MultiCursor

• For special applications that are aware of
  multiple inputs
• E.g. Shared Whiteboard
• Single Display Groupware
• Multiple people, one display
• A number of interestingissues
• Palettes, widgets
• Section handles
• Undo

28
Chat

• Communicate to another PDA user through the PC
• PC serves as a conduit
• For side notes and messages
• For example, in negotiationmeetings
• Send to all or to a specificperson

29
Semantic Snarfing

• Interacting at a distance
• Grab contents to handheld
• Picture
• Menus
• Text
• Re-visualization
• Private Drill-downof public information

30
Study of Laser Pointers

• Studied properties of laserpointer interaction
  techniques
• Hand-wiggle /-8 pixels
• Delay until moving average stable /-1.5 sec
• Delay until target acquired /-1.5 sec
• Different devices and poses do not help much

31
Home Use

• Personal Universal Controller

32
Personal Universal Controller

• Two-way communication
• Appliances describe their functions
• Handheld PUC creates interface based on
  descriptions
• Handheld PUC controls the appliance
• Appliance sends back status as feedback

Specifications
Control
Feedback of Status
33
Initial Experiments

• Use real PocketPC
• Pretend that controls devices

34
Initial Study results

• For both appliances, users of actual interfaces
• Took about twice as long
• Made at least twice as many mistakes as users of
  the handheld interfaces
• Needed external help five times more often

35
Current Steps
Specification Language encoding"UTF-8"? xmlnsxsi"http//www.w3.org/2001/XMLSchema-instan
ce" xmlnsxsd"http//www.w3.org/2001/XMLSch
ema" name"Audiophase 5 CD Stereo"




Off

On
Stereo
Power Power Powr Pwr
10

• XML specification language and protocol for
  describing appliances
• Create panels automatically from the
  specification
• Finding realappliances whichwe can control

36
General Architecture
On the PC
Any PC app.
BradsPalm Pilot
RemoteCmd
Windowseventstream
PebblesPC
RobsPocket PC
MultiCursor
PebblesDraw
Serial,IR orsockets
BensWindows CE
Direct connection or sockets
SlideshowCommander
PowerPoint
OLEAutomation
Various PDA apps
Various dlls
37
Protocols

• Can use Pebbles protocols to develop your own
  application
• Libraries for Palm, Windows CE, PC
• Independence from communication medium
• Use by
• PalmAmp from IronCreek Software
• Intel research
• others
• Windows messages or sockets

38
Downloads

• Most of this software is available for free
  downloading
• Also, about 20 published papers
• First release in Feb, 1998
• Recently released version 5 (Feb, 2002)
• Downloaded over 30,000 times
• About 200 times a week

http//www.pebbles.hcii.cmu.edu/
39
More in Next Session!

• Next session 1120 - 1205
• Details of use in classrooms and for the
  handicapped
• SlideShow Commander!
• Concept Tests
• Handhelds as Assistive Technology

40
Conclusions

• Handhelds are becoming ubiquitous
• Connecting technologies will improve
• Important to study how can be used when connected
  to computers and each other
• Multi-Machine User Interfaces will be
  increasingly important
• People will want to use the most convenient
  device for their information and control

41
Thanks to Our Sponsors!

• Supported by grants from
• DARPA
• Microsoft
• Pittsburgh Digital Greenhouse
• NSF
• General Motors
• And equipment grants from
• Hewlett Packard
• Lucent Technologies
• Palm Computing
• Symbol Technologies

• IBM
• SMART Technologies, Inc.
• Synergy Solutions, Inc.
• Handango

42
The Pebbles ProjectGeneral OverviewUsing
Hand-Held Computers and PCs Together
Human Computer Interaction Institute School of
Computer Science Carnegie Mellon University
Thank you!

• Brad A. MyersCarnegie Mellon University
• bam_at_cs.cmu.edu
• http//www.cs.cmu.edu/pebbles

SUNY Technology Conference 2002