Toolkits for Physical Objects TUIs

1
Toolkits for Physical Objects (TUIs)

• Jack Li
• Advanced User Interface Software
• November 30
• Fall 2004

2
Outline

• Motivation for TUIs
• Input Technologies
• TUI Toolkits

3
Current Limitations of UIs
MOTIVATION FOR TUIS

• Desktops and laptops suck in our entire attention
• As a result we live in two worlds the real
  physical world and the virtual cyber world
• Constantly wired in order to be in both worlds
• Loss of rich interactions found in everyday
  things (particularly haptic)

4
Motivation
MOTIVATION FOR TUIS

• Driven by ubiquitous computing
• Both people and technology moving away from the
  desktop into the environment
• Input different ways of interacting
• Output different ways of displaying

5
Tangible User Interfaces
MOTIVATION FOR TUIS

• TUIs augment the physical world by integrating
  digital information with everyday physical
  objects
• Goal to bridge the gap between bits and atoms
• 1. Interactive surfaces
• 2. Coupling information with physical objects
• 3. Ambient media for background awareness

6
Outline

• Motivation for TUIs
• Input Technologies
• TUI Toolkits

7
Physical Objects as Input
INPUT TECHNOLOGIES

• Pretty much anything walls, papers, utensils,
  clothes hangers!
• Inspiration
• sci-fi novels, own imagination
• everyday life, current practices

8
Devices
INPUT TECHNOLOGIES

• Cameras
• Electronic tags (RFID technology)
• Sensors
• Light
• Motion
• Heat
• Force
• Motors
• Orbs

9
Framework for the GUI
INPUT TECHNOLOGIES

• (a)model-view-controller

10
Framework for the TUI
INPUT TECHNOLOGIES

• (b)MCRpd model-control-representation(physical
  and digital)

11
Outline

• Motivation for TUIs
• Input Technologies
• TUI Toolkits
• Phidgets (2001)
• Calder (2004)
• Papier-Mâché (2004)

12
Toolkit Goals

• Same as when GUIs were built
• Make easy for non-specialized experts to create
  (should not need a vision or hardware guru on the
  team)
• Hide dirty implementation details
• Abstract proper functionality with well-defined
  API

13
Overview
PHIDGETS

• First toolkit for physical objects
• Reusable physical devices with proper abstraction
  for creating a wide variety of applications
• An API that had the right amount of abstraction

14
GUI Differences
PHIDGETS

• Connection manager
• Way of detecting when devices go on/off
• Identification
• Every physical object should have a virtual
  counterpart
• Simulation
• A way to test code even if phidget not present

15
Architecture
PHIDGETS
16
The Physical Device
PHIDGETS

• Primitive device components
• Sensors
• Motors
• Switches
• Circuit board
• CY7C63000 USB micro-controller
• Communications layer
• USB communication standard

17
Wire Protocol
PHIDGETS

• Low-level software in Windows and microcontroller
  for communication thats hidden from end
  programmers
• Commands from phidgets
• Unique Id numbers
• Device type (servo, powerbar)
• Specific events (tag for RFID reader)
• Commands from Windows
• Set device state (output like servo)

18
Computer Software
PHIDGETS

• PhidgetManager
• Connection manager
• Event onAttach()
• Event onDetach()
• Int count
• Enum item
• Phidget-specific devices
• All have
• String deviceType
• Boolean isAttached
• Long serialNumber
• RFID
• onTag()
• Servo
• onPositionChanged(), motorPosition(index)

19
Simple Application
PHIDGETS

• Phidget Manager

• Phidget Servo

20
Outline

• Motivation for TUIs
• Input Technologies
• TUI Toolkits
• Phidgets (2001)
• Calder (2004)
• Papier-Mâché (2004)

21
Main goals
CALDER

• Primarily geared towards interaction/product
  designers
• Support for more refined products
• Support on the hardware side

22
Supported Devices
CALDER
Small size is big win!
23
Fluid prototyping
CALDER
Objects easily embedded!
24
Communication
CALDER

• Wired
• PIC16C745 ?C with USB interface engine
• Same as Phidgets (and iStuff)
• Wireless
• Tradeoff between size and power (perhaps Moores
  law will remedy)
• Small batteries, antenna, and pulse-width
  modulation

25
Software
CALDER

• Surrogates (GUI widgets) to manipulate physical
  object output

26
Outline

• Motivation for TUIs
• Input Technologies
• TUI Toolkits
• Phidgets (2001)
• Calder (2004)
• Papier-Mâché (2004)

27
Overview
PAPIER-MACHE

• Emphasis on software side side
• Input model that provides common programming
  abstractions
• Focus on objects being lightweight

28
Software Abstractions
PAPIER-MACHE

• Phobencapsulates information related to this
  physical object
• Allsource producer, time
• RFIDPhobtag
• VisionPhobmean color, bounding box, rotations
• PhobListenerlistens for Phobs produced
• phobAdded(), phobUpdated(), phobRemoved()
• PhobProduceractual devices that generates events
• PhobManagermaintains PhobProducers

29
A Common Event Model
PAPIER-MACHE

• PhobListener is the interface among all devices
• Methods are the same
• phobAdded(), phobUpdated(), phobRemoved()
• Vision
• phobAdded() when new object detected,
  phobUpdated() when old object changed (size,
  color), phobRemoved() when old object disappears
• RFID (different from what the paper says)
• phobAdded() when tag detected, no phobUpdated(),
  phobRemoved() when tag disappears
• Facilitates easier retargetting and substitutions
  (allows for more exploration and faster,
  prototyping)

30
Marble Answering Example
PAPIER-MACHE

• 1. Get the manager that keeps track of all input
  devices (PhobProducers)
• RFIDManager manager RFIDManager.getManager()
• 2. Set up a map associating all objects with an
  audio clip
• AssociationFactory factory new
  AudioClipFactory() AssociationTreeTable assocMap
  new
• AssociationTreeTable(factory)
• 3. Attach this map so new objects are assigned
  new clips and old ones get played
  manager.addPhobListenerForAllProducers(assocMap)

31
A Behavior Associations
PAPIER-MACHE

• A common behavior we noticed
• New objects get stored/assigned, old objects get
  retrieved a database of sorts
• An extension of PhobListener

32
A Behavior Associations
PAPIER-MACHE

• AssociationTreeTable
• addClassifierFactoryPair(Classifier, Factory)
• ObjectClassifier
• isMemberOfClass(Phob phobToCompare)
• AssociationFactory
• createAssociationEltForPhob(Phob phob)

33
Monitoring Window
PAPIER-MACHE
34
Device stateconnection mgr
PAPIER-MACHE
35
Camera Feedback
PAPIER-MACHE
36
Association Map
PAPIER-MACHE
37
Extras
PAPIER-MACHE
38
Summary
39
Major Toolkit differences

• Phidgets
• Complete, balanced, varied assortment of physical
  objects easy to program with
• Calder
• Small size allows for more fluid prototyping at
  the PHYSICAL level
• Papier-Mâché
• Unified event model allows easier development and
  retargetting at the SOFTWARE level

40
Conclusion

• TUIs seek to augment the physical world with
  digital information
• Toolkit support allows non-experts to develop TUI
  application, like GUI toolkits
• Successful toolkits must manage physical
  connections, gather/alter relevant physical state
  information, and provide good feedback for these
  added complexities

41
Future work

• Input adaptation
• M to N mapping of input devices
• Input models
• What other kinds of behaviors can be abstracted
• Will be more clear with the development of more
  applications

42
References

• Hiroshi Ishii , Brygg Ullmer, Tangible bits
  towards seamless interfaces between people, bits
  and atoms, Proceedings of the SIGCHI conference
  on Human factors in computing systems, p.234-241,
  March 22-27, 1997, Atlanta, Georgia, United
  States
• Ullmer, B. and H. Ishii, Emerging Frameworks for
  Tangible User Interfaces, in Human-Computer
  Interaction in the New Millennium,
  Addison-Wesley. pp. 579 -- 601, 2001.
• Saul Greenberg, Chester Fitchett. "Phidgets easy
  development of physical interfaces through
  physical widgets," Proceedings of the 14th annual
  ACM symposium on User interface software and
  technology, November 11-14, 2001, Orlando,
  Florida
• Scott R. Klemmer, Jack Li, James Lin, James A.
  Landay. "Papier-Mache toolkit support for
  tangible input," Proceedings of the 2004
  conference on Human factors in computing systems,
  p.399-406, April 24-29, 2004, Vienna, Austria
• Lee, J.C. Avrahami, D. Hudson, S.E. Forlizzi,
  J. Dietz, P.H. Leigh, D.L., "The Calder
  Toolkit Wired and Wireless Components for
  Rapidly Prototyping Interactive Devices",
  Designing Interactive Systems (DIS), ISBN
  1-58113-787-7, pp. 167-175, August 2004