Lecture 17: Development of and for Devices

1
Lecture 17Development of and for Devices

• Jake Wobbrock
• 05-830Advanced User Interface Software

2
Not a Proper Toolkit Lecture

• This lecture might have been about UI toolkits
  for devices
• Theres just one problem
• There really arent any
• At least, not like we think of toolkits for
  desktop software
• Instead we will talk about things that help you
  develop for devices.
• And other interesting device issues

3
Overview

• Introduce work in multiple semi-related areas of
  software development of and for devices
• Palm OS
• Pocket PC / Windows CE
• .NET Compact Framework
• J2ME
• Waba
• BREW
• WAP and WML
• DirectX

4
Overview cont.

• Highlight HCI research projects with and for
  devices
• Phidgets
• Rapid prototyping for physical devices (physical
  widgets)
• Pebbles Project
• Interoperation of handhelds and PCs
• Handheld Web browsing (various efforts)
• WebThumb, Power Browser, The Gateway, M-Links

5
Motivation Handhelds

• Cell phones and PDAs have seen huge growth in use
  over recent years.
• Last year 2002 there was one cell phone sold
  for every 15 human beings on the planet
  Worldwide mobile phone unit sales totaled 423.4M
  in 2002, a 6 increase from 2001.
• From 1999 to 2000 the number of PDAs sold nearly
  doubled, from some 3.6M units to 6.9M By 2004,
  the number of units is expected to top 33.5M.

• Source Biz Journals

6
Motivation cont. Gaming

• Nintendo GameBoy
• NTT DoCoMo Inc. handhelds running Java
• Others by Qualcomm, L.M. Ericsson, Motorola,
  Siemens
• Convergence Games for phones and PDAs, not only
  custom handhelds

• Source CNN

7
The Punch Line

• PDAs, phones, other handhelds experiencing great
  growth in use
• Development community needs to have good tools
  and toolkits
• What is available today?
• Sadly, not much (at least compared to whats
  available for the desktop)

8
Palm OS
Visit http//www.palmsource.com/products/

• Palm OS by PalmSource
• Acer
• AlphaSmart
• Fossil
• Garmin
• Handspring
• HandEra
• Kyocera
• Palm
• Samsung
• Sony
• Symbol

9
Palm OS Facts

• As of January 2002
• 16M Palm-powered PDAs shipped (to date)
• 7000 applications
• 150,000 registered developers
• Growing at 1000 per month

• Source Palm OS Programming, OReilly

10
Palm OS Development

• Two main ways
• CodeWarrior for Palm OS
• PRC-Tools (GCC)
• Other ways
• Sun KVM and J2ME
• IBM VisualAge Micro Edition (Windows or Linux)
• Jump (use Java, compiles to Motorola 68K)
• Waba (subset of Java, stripped-down VM)
• PocketStudio (Pascal-based, aimed at Delphi
  users)
• Pocket C (develop on the handheld for the
  handheld)
• Assembler SDK (ASDK) write directly for
  Motorola 68K
• Pocket Smalltalk (includes garbage-collecting VM)

11
CodeWarrior for Palm OS

• Special line of Metrowerks CodeWarrior releases
  for Palm OS development
• Three key components
• Editor
• Constructor
• Palm OS Emulator (POSE)
• Write code in C/C
• Downloaded to Palm via HotSync

12
Editor
13
Constructor
14
Emulator (POSE)

• Runs on the desktop
• Emulates from a very low level
• Designed to simulate speed despite running on a
  desktop
• Useful for debugging
• Not useful for interactions requiring
  pen-dexterity

15
Technical Details

• Palm OS apps center around event cascade
• Catch events by defining handlers and handling
  them
• But not nearly as much framework as Visual Basic
  or MFC more like Win32
• Often involves large switch statements
• Single-threaded, single-process

16
The Event Cascade
17
CodeWarrior Summary

• Generally the best option
• Originally for Macintosh, then for Windows!
• BUT
• Constructor is often not as WYSIWYG or Direct
  Manipulation as youd like (e.g., text fields,
  scroll bars)
• It lacks options for all programmatically-settable
  options that affect visual appearance of
  controls (e.g., button borders)
• Emulator doesnt work exactly like the device
• Sometimes this matters, often this doesnt

18
PRC-Tools (GCC)

• Alternate means for Palm OS development
• Developed before CodeWarrior ran on Windows by
  the Free Software Foundation
• Collection of tools for Unix or Windows
• Gnu C Compiler (GCC) for Motorola 68K
• Build-PRC (combine binaries into .prc file)
• Gnu Debugger (GDB)
• PilRC (compiles resource files)
• Falch IDE combines these with a UI

19
Pocket PC? Windows CE?

• Pocket PC
• Not considered (strictly-speaking) a PDA
• Or at least not marketed that way
• Meant to be a full-fledged PC but pocket-sized
• Refers to a device (platform)
• Also to a set of standard applications common on
  all Pocket PC devices, esp. Pocket PC Shell
• Windows CE (WinCE)
• Refers to just the OS that runs on Pocket PCs
• Has large number of capabilities (components) for
  the Pocket PC developer to choose from
• Current version is WinCE .NET

20
Pocket PCs vs. Palm OS Devices

• Basic PDAs allow you to store and retrieve
  addresses and phone numbers, maintain a calendar,
  and create to-do lists and notes. More
  sophisticated PDAs can run word processing,
  spreadsheet, money manager, games and electronic
  book reading programs and also provide email and
  Internet access.

• Source Microsoft

21
Pocket PCs

• Toshiba
• Compaq iPaq
• HP Jornada
• Audiovox
• T-Mobile
• Dell Axim
• ViewSonic
• Razor Zayo
• Cassio Cassiopeia
• NEC MobilePro

22
Pocket PC Facts

• Projected 2003 sales between 14M 17M devices
• Compare to PC sales of 135M 138M
• Pocket PC projected to catch up by 2008!
• Sales hit 10M after just 10 months on the market
• Introduced April 2000
• In May 2001, held 26 of market. Rest held by
  Palm OS devices.

• Source CNet

23
The Shrunken Desktop

• Pocket Word
• Pocket Excel
• Pocket Outlook
• Pocket IE
• Pocket Windows Media Player
• ??

24
Pocket PC Development

• eMbedded Visual Tools 3.0 (VC, VB)
• Largely just like Windows programming
• Can even use MFC for VC
• Compiles for many different processors

25
Challenge

• Problem Portability
• Not all Pocket PCs utilize the same WinCE
  components
• So not all Pocket PC apps will run on all Pocket
  PCs
• Despite same underlying OS!
• Example The menu bar at the bottom is specific
  to Pocket PCs, not WinCE
• Created and passed to the Pocket PC shell
• Developers often have to change much UI code for
  different Pocket PC platforms
• Example Handheld PC (H/PC) is wide and short, so
  tall dialogs have to be re-laid-out

26
Resource Editor
27
Pocket PC Summary

• Powerful palmtop computers
• More complicated than Palm OS Devices (e.g., can
  be multi-threaded)
• Less integrated vertically than Palm OS devices
• Complicates development (e.g., UI components not
  often supported on all Pocket PC devices)
• With more power and capability comes more
  complexity and complication
• Affects the toolkits for development on such
  devices

28
.NET Compact Framework

• Subset of .NET framework for desktops
• Not PDA-specific but targeted to any smart
  devices PDAs, mobile phones, set-top boxes,
  automobiles, etc.
• Delivered as extensions to Visual Studio .NET
  called Smart Device Programmability
• Tools and programming model are same as for
  desktop .NET platform
• Easy transition from .NET desktop development to
  .NET compact framework development

29
.NET Compact Framework cont.

• Argument is that there are currently too many
  mobile CPUs and OSes, so development is difficult
• True but we know the Microsoft answer
• .NET would allow integration of devices across
  multiple platforms, networks, and programming
  languages
• A vision behind .NET in general
• Compact Framework is a subset of all .NET
  capabilities
• Reduces footprint on resource-strained devices

30
J2ME Java 2 Micro Edition

• Uses subset of Java Virtual Machine for smaller
  footprint (i.e., KVM)
• Highly optimized runtime environment
• Not PDA-specific smart cards, pagers, mobile
  phones, set-top boxes, vehicle telematics systems
  (e.g., OnStar system)
• Set of APIs defined by the Java Community Process
  Program
• Includes user interface, security, networking
  protocols, and more

31
J2ME Architecture

• J2ME Architecture defines
• Configurations A virtual machine and a minimal
  set of class libraries
• For Palm OS, configuration is Connected Limited
  Device Configuration (CLDC)
• Profiles Higher-level APIs that further define
  the application, user interface, and
  device-specific properties
• For Palm OS, profiles are PDA and Mobile
  Information Device Profile (MIDP)

32
J2ME Architecture cont.
From http//java.sun.com/j2me/j2me-ds.pdf
33
Waba

• Subset of Java
• Stripped-down VM small and fairly fast
• No longs, doubles, exceptions, or threads
• Open source http//www.wabasoft.com/
• Waba defines
• A language (strict subset of Java)
• A virtual machine (Waba VM)
• A class file format (strict subset of Java
  bytecode)
• A set of foundation classes

34
BREW

• Qualcomms Binary Runtime Environment for
  Wireless (BREW)
• Like J2ME, provides application execution
  environment
• Goes farther! Provides business model for
  certifying, downloading, and charging premium
  content
• With J2ME, this is left entirely to the marketers
  and developers to figure out for themselves
• Based on C (so it is fast, and small binaries)

35
WAP and WML

• WAP (Wireless Application Protocol)
• Provides flat Web content on handhelds
• Requires constant connectivity by the device
• Lacks a rich interactive experience for end-users
• WML (Wireless Markup Language)
• Specifies content and user interface for
  WAP-delivered information
• Based on XML so describes data, not just
  presentation of data
• WMLScript
• WAPWMLWMLScript HTTPHTMLJavaScript
• Provides only minimal interactivity

36
WAP and WML cont.

• Four things specified by WML
• Text and image presentation and layout, including
  a variety of formatting commands
• Deck/card organizational metaphors for
  information (similar to HyperCard)
• Inter-card navigation and linking
• Card parameterization and state management
• The fate of WAP
• Hasnt been the success its advocates hoped
• New push to make possible highly interactive
  content
• J2ME, BREW, .netCF, Waba, but these all send a VM
  and increase footprint

37
Game Development DirectX

• DirectX Advanced suite of multimedia APIs
• DirectDraw, Direct3D, DirectSound, DirectMusic,
  DirectInput, DirectPlay, DirectShow
• Allows accessing of special hardware capabilities
  (e.g., graphics and sound cards) without
  requiring hardware-specific code
• DirectX exists for Pocket PC
• Handheld gaming projected to be a big new market
  opportunity
• Again, similar to programming for desktop

38
Transition from (Mainly) Industrial Tools to
Research Initiatives for and with Devices
39
Phidgets (Greenberg et al.)

• Phidgets are physical widgets
• Play the same role in physical UIs as widgets do
  in graphical UIs
• Package IO
• Hide implementation details, provide abstraction
• Have additional characteristics
• Require a connection manager
• A link between the software and hardware parts
• A simulation mode in software only

40
Phidget Architecture

• Physical Device
• Packaged physical unit to be used by the
  designer. Most phidgets are built around a
  circuit board.
• Wire Protocol
• Communication protocol between device and host
  computer. Not visible to end-programmers.
• Phidget Manager
• COM object that has event-based API for
  end-programmers for connection management.
• Phidget-specific COM objects
• Created by the Phidget Manager whenever a device
  is seen. They correspond 11 to physical devices.
• IGlabPhidget interface
• Generic interface which all phidgets have.
  End-programmers use it for basic identification
  and capabilities.
• Phidget-specific interface
• Capabilities specific to a particular phidget.
  Extends the basic IGlabPhidget interface.
• Phidget ActiveX controls
• Wraps up the phidget code and provides an
  on-screen user interface and the ability to
  simulate the phidget as an control.

41
Pebbles (Myers et al.)

• Research project exploring the interoperation of
  handhelds and PCs
• Handhelds can be an aspect of the user, PCs can
  be a fixture of the environment
• Users may move from environment to environment,
  PCs may stay behind
• Handhelds can control other devices too
• Personal Universal Controller
• Handhelds can be used to control a PC by people
  with motor impairments

42
A Current Research Issue

• Handheld User Interfaces for Web Browsing
• Challenges
• Constrained screen-space
• No mouse cursor (e.g., no link roll-overs)
• Pages designed on a desktop for display on a
  desktop
• Scrolling more difficult
• Graphic refresh can be slow, scrolling blurry

43
Handheld Web Browsing

• The Gateway (MacKay)
• Pages displayed as thumbnails expand as certain
  parts of them are interacted with (CHI 2003).
• WebThumb (Wobbrock et al.)
• Exploration of interaction techniques for better
  handheld browsing (UIST 2002).
• M-Links (Trevor et al.)
• Uses modes to separate action from navigation on
  mobile phone Web pages (UIST 2001).
• Power Browser (Buyukkokten et al.)
• Generates summary views of handheld web content
  for display (CHI 2000).

44
Conclusions

• Toolkits for development for devices are far from
  Amulet-level sophistication
• Many processors and devices exist complicates
  development and toolkit design
• Numerous standards and niche technologies
• Minimizing footprint on resource-constrained
  devices is a key constraint
• HCI research can tackle issues beyond the
  specifics of device hardware and software
• e.g., interaction techniques for PC/PDA
  interoperability
• e.g., interaction techniques for browsing