The Endeavour Expedition: 21st Century Computing to the eXtreme

1
The Endeavour Expedition21st Century Computing
to the eXtreme

• Randy H. Katz, Principal Investigator
• EECS Department
• University of California, Berkeley
• Berkeley, CA 94720-1776

2
The Endeavour Expedition21st Century Computing
to the eXtreme

• New Ideas
• Systems Architecture for Vastly Diverse
  Computing Devices (MEMS, cameras, displays)
• Wide-area Oceanic Data Information Utility
• Sensor-Centric Data Management for Capture
  and Reuse (MEMS networked storage)
• Negotiation Architecture for Cooperating
  Components (Composable system architecture)
• Tacit Knowledge Infrastructure to support
  High-Speed Decision-Making
• Information Management for Intelligent
  Classroom Environments
• Scalable Safe Component-based Design and UI
  Design Tools

R. H. Katz, Principal Investigator, University
of California, Berkeley

• Impact
• Enhancing human understanding by making it
  dramatically more convenient for people to
  interact with information, devices, and other
  people
• Supported by a planetary-scale Information
  Utility, stress tested by applications in
  decision making and learning, achieved thru
  new methodologies for design, construction,
  and administration of systems of
  unprecedented scale and complexity

Schedule
Usability Studies Early Tool Design
Implementation of UI Sys Design Tools
Tools Release Final Evaluations
Design Methodologies
Initial Application Implementation Evaluation
Refined Implementation Final Evaluation
Information Applications
Initial Architectural Design Testbeds
Initial Evaluation 2nd Gen Redesign
Final Deployment Evaluation
Information Utility
Initial Architectural Design Document
Initial Experiments Revised Design Doc
Final Experiments Architecture Docs
Jun 99 Start
Jun 00
Jun 01
May 02 End
3
Expedition Goals

• Enhancing understanding
• Dramatically more convenient for people to
  interact with information, devices, and other
  people
• Supported by a planetary-scale Information
  Utility
• Stress tested by challenging applications in
  decision making and learning
• New methodologies for design, construction, and
  administration of systems of unprecedented scale
  and complexity
• Figure of merit how effectively we amplify and
  leverage human intellect
• A pervasive Information Utility, based on fluid
  systems to enable new approaches for problem
  solving learning

4
Why Endeavour?

• Endeavour to strive or reach a serious
  determined effort (Websters 7th New Collegiate
  Dictionary) British spelling
• Captain Cooks ship from his first voyage of
  exploration of the great unknown of his day the
  southern Pacific Ocean (1768-1771)
• Brought more land and wealth to the British
  Empire than any military campaign
• Cooks lasting contribution comprehensive
  knowledge of the people, customs, and ideas that
  lay across the sea
• He left nothing to his successors other than to
  marvel at the completeness of his work.

5
Expedition Assumptions

• Human time and attention, not processing or
  storage, are the limiting factors
• Givens
• Vast diversity of computing devices (PDAs,
  cameras, displays, sensors, actuators, mobile
  robots, vehicles) No such thing as an average
  device
• Unlimited storage everything that can be
  captured, digitized, and stored, will be
• Every computing device is connected in proportion
  to its capacity
• Devices are predominately compatible rather than
  incompatible (plug-and-play enabled by on-the-fly
  translation/adaptation)

6
Expedition Challenges

• Personal Information Mgmt is the Killer App
• Not corporate processing but management,
  analysis, aggregation, dissemination, filtering
  for the individual
• People Create Knowledge, not Data
• Not management/retrieval of explicitly entered
  information, but automated extraction and
  organization of daily activities
• Information Technology as a Utility
• Continuous service delivery, on a
  planetary-scale, on top of a highly dynamic
  information base
• Beyond the Desktop
• Community computing infer relationships among
  information, delegate control, establish
  authority

7
Driving Factors

• Technology Push
• Accelerating developments at the eXtremes
• Cluster-based compute/storage servers
• MEMS sensor/actuators, CCD cameras, LCD displays,
  
• User Pull
• More effective community leverage the next power
  tool
• Desire
• Enhanced interaction, ease of use
• Easier configuration, plug and play
• Less fragile tools, always there utility
  functionality

8
Computing Revolution Devices in the eXtreme
9
Expedition Approach

• Information Devices
• Beyond desktop computers to MEMS-sensors/actuators
  with capture/display to yield enhanced activity
  spaces
• InformationUtility
• InformationApplications
• High Speed/Collaborative Decision Making and
  Learning
• Augmented Smart Spaces Rooms and Vehicles
• Design Methodology
• User-centric Design withHW/SW Co-design
• Formal methods for safe and trustworthy
  decomposable and reusable components

• Fluid, Network-Centric System Software
• Partitioning and management of state between soft
  and persistent state
• Data processing placement and movement
• Component discovery and negotiation
• Flexible capture, self-organization, and re-use
  of information

10
Interdisciplinary, Technology-Centered Expedition
Team

• Alex Aiken, PL
• Eric Brewer, OS
• John Canny, AI
• David Culler, OS/Arch
• Joseph Hellerstein, DB
• Michael Jordan, Learning
• Anthony Joseph, OS
• Randy Katz, Nets
• John Kubiatowicz, Arch
• James Landay, UI

• Jitendra Malik, Vision
• George Necula, PL
• Christos Papadimitriou, Theory
• David Patterson, Arch
• Kris Pister, Mems
• Larry Rowe, MM
• Alberto Sangiovanni-Vincentelli, CAD
• Doug Tygar, Security
• Robert Wilensky, DL/AI

11
Organization The Expedition Cube
12
Base Program Leader Katz

• Broad but necessarily shallow investigation into
  all technologies/applications of interest
• Primary focus on Information Utility
• No new HW design commercially available
  information devices
• Only small-scale testbed in Soda Hall
• Fundamental enabling technologies for Fluid
  Software
• Partitioning and management of state between soft
  and persistent state
• Data and processing placement and movement
• Component discovery and negotiation
• Flexible capture, self-organization, info re-use
• Limited Applications
• Methodology Formal Methods User-Centered Design

13
System Architecture for Vastly Diverse
DevicesLeader Culler

• Design Issues for Small Device OS
• Current managing address spaces,thread
  scheduling, IP stack, windowing system, device
  drivers, file system, APIs, power management
• How can OSs for tiny devices be made radically
  simpler, manageable, and automatically
  composable?
• Devices of Interest Dust Motes

14
Communication-Centric Architecture

• Base Scalable Infrastructure
• highly available
• persistent state (safe)
• databases, agents
• service programming environment

• Active Proxies
• connected to the infrastructure
• soft-state, bootstrap protocol
• transcoding,

• Ubiquitous Devices
• billions
• sensors / actuators
• PDAs / smartphones / PCs
• heterogeneous

• Service Paths
• aggregate flows (rivers)
• transcoding operators

15
The Large Service-Centric Platform Arch
Infrastructure Services

• Enable distributed creation/deployment of
  scalable, available services
• Service registry, aggregate execution env.,
  transparency
• Persistent distributed data structures
• Massive fluid storage (Oceanic Storage)
• Adaptive high-bandwidth flows (rivers)
• Build infrastructure via composition of services

16
The Small Radically Simple OS for Management
Composition

• Basic Assumptions
• Communication is fundamental
• Direct user interface is the exception not the
  norm
• Critical resource is scheduling data movements,
  not arbitrary threads of computation
• Tiny OS Little more than an FSM
• Commands event stream merged with
  sensor/actuator events
• General thread compiled to sequence of bounded
  atomic xacts
• Constant self-checking and telemetry
• Rely on the infrastructure for complex processing
  
• Correctness-by-construction techniques for
  cooperating FSMs (tie in to HW/SW co-design)

17
Implementation Deployment of Oceanic Data Info
UtilityLeader Kubiatowicz

• Ubiquitous devices require ubiquitous storage
• Consumers of data move, change access devices,
  work in many different physical places, etc.
• Needed properties
• Strong Security data must be encrypted whenever
  it is in the infrastructure
• Coherence too much data for naïve users to keep
  coherent by hand
• Automatic replica management and optimization
  huge quantities of data cannot be managed
  manually
• Simple and automatic recovery from disasters
  probability of failure increases with size of
  system
• Utility model world-scale system requires
  cooperation across administrative boundaries

18
Utility-Based Infrastructure
Canadian OceanStore
Sprint
ATT
IBM
Pac Bell
IBM

• Confederations of (Mutually Suspicious) Utilities
• Settlement system among service providers
• Buy and sell capacity as needed

19
OceanStore Architecture/Technology

• Name and Data Location
• Issue Find nearby data without global
  communication
• Approach Data location is aform of
  gradient-search of local pools of data (use of
  attenuated Bloom-filters)
• High Availability and Disaster Recovery
• Issue Eliminate backup as independent/fallible
  technology
• Approach Erasure-codes/mobile replicas provide
  stable storage for archival copies and snapshots
  of live data
• Introspective Monitoring and Optimization
• Issue Optimize performance on a global scale
• Approach Monitoring and analysis of access/usage
  relationships
• Rapid Update in Untrusted Infrastructure
• Issue Updates should not reveal info to
  untrusted servers
• Approach Incremental cryptographic
  techniques/oblivious function techniques to
  perform update

20
Applications that Enhance Human Activity

• Tacit Information Mining exploit info flows
  relationships to improve collaborative work
• 3D activity spaces for representing
  decision-making activities, people, information
  sources
• Visual cues to denote strength of ties between
  agents, awareness levels, activity tracking,
  attention span
• Smart Spaces
• Electronic collaborative problem-based learning
• Physical and Virtual Learning Spaces
• Enabled by information appliances
• UI design/exploitation of tacit information

21
Experimental Testbeds
Soda Hall
IBM WorkPad
Smart Dust
Velo
Nino
LCD Displays
MC-16
Motorola Pagewriter 2000
CF788
Pager
WLAN / Bluetooth
Smart Classrooms Audio/Video Capture
Rooms Pervasive Computing Lab CoLab
H.323 GW
GSM BTS
Wearable Displays
TCI _at_Home Adaptive Broadband LMDS
Millennium Cluster
CalRen/Internet2/NGI
Millennium Cluster
22
Summary Putting It All Together

• 1. eXtreme Devices
• 2. Data Utility
• 3. Capture/Reuse
• 4. Negotiation
• 5. Tacit Knowledge
• 6. Classroom
• 7. Design Methods
• 8. Scale-up

Devices Utility Applications
Component Discovery Negotiation
Fluid Software
Info Extract/Re-use
Self-Organization
Decision Making Group Learning
23
Base Program Schedule
Year 1
Year 2
Year 3
Design Methodology
Refined Tools Flow
Information Utility
Information Applications
24
Year 1 Milestones

• Design/initial deployment smart space testbed
  Initial usability evaluation/refinement
• Initial design, prototype, and early evaluation
  of fluid software run-time environ
• Initial design component advertisement protocols
  i/f negotiation spec language
• Initial prototype/refinement of component
  advertisement protocols interface negotiation
  specification language
• Initiate prototype refinement of distributed,
  persistent storage system

• Initial design of sensor-centric/stream-capture
  oriented data mgmt system
• Initiate prototype refinement of sensor-centric
  data mgmt system
• Design of distributed, persistent storage system
• Initial design of tool flow for
  infrastructure-embedded software functionality
• Initiate implementation of system design tools
  for early testing
• Completion of initial system architecture design
  document and early system evaluation

25
Problem
Technical Approaches
Coherently managing billions of devices where
none are average Information on demand,
available wherever needed, on a global scale,
in an untrusted infrastructure Pervasive
management of massive stream-oriented information
collection/inference in the wide-area
Data movement transformation Paths, not
threads Persistent state/soft state
partitioning Non-blocking RMI for remote
functionality Support for MEMS devices,
cameras, displays, etc. Serverless/homeless/freely
flowing data Opportunistic distribution,
promiscuous caching, without administrative
boundaries High availability/disaster recovery,
application-specific data consistency,
securityOverlapping, partially consistent
indices Data freedom of movement Expanding
search parties to find data, using
application-specific hints Extract, manage,
analyze streams of sensor data Path-based
processing integrated with storage Data
reduction via filtering/aggregation Distributed
collection processing Evidence accumulation
from inherently noisy sensors
26
Problem
Technical Approaches

Overwhelming config-uration complexity of large
heterogeneous systems Ineffectiveness of
technology-mediated collaborative workBetter
support for rapid decision making Enabling
Problem-based Learning in Enhanced Physical
Virtual Spaces Correctness by Construction Safe
Component Design
Dynamic self-configuration advertise provided
services, discover components providing required
services, negotiate interface contracts, monitor
compliance, eliminate non-performing
confederates Infer communications flow, indirect
relationships, availability, participation to
enhance awareness support opportunistic
decision making New collaborative applications
3D activity spaces for representing
decision-making activities, people, info
sources Visual cues weighting relationships
among agents, awareness levels, activity tracking
attention span Device/net-independent
people-to-people comms via pervasive
translation/adaptation Information
dissemination technologies Wide-area
information mgmt/access Formal specifications
and methods Safety enforcement,
design/development methods Proof carrying
code/secure protocol verification
27
Summary and Conclusions

• 21st Century Computing
• Making peoples exploitation of information more
  effective
• Encompassing eXtreme diversity, distribution, and
  scale
• Computing you can depend on
• Key Support Technologies
• Fluid software computational paradigms
• System and UI support for eXtreme devices
• Pervasive, planetary-scale system utility
  functionality
• Active, adaptive, safe and trusted components
• New power tool applications that leverage
  community activity

28
Industrial Collaborators
SRI