Telegraph: A Universal System for Information

1
Telegraph A Universal System for Information
2
Telegraph History Plans

• Initial Vision
• Carey, Hellerstein, Stonebraker
• Regres, B-1
• Sweat, ideas and further vision
• 4 of my grads committed
• Brewer 2 grads committed
• Franklin will play
• obvious tie-ins with other projects

3
Telegraph Architecture
synergies!
Control, DigLib
Query/Browse/Mine
Mariposa, Millenium, Control
Global Agoric Federation
Continuously Reoptimizing Query Processor
Adaptive Data Placement
River, Ninja, Aetherstore, Control,STIX
Storage Manager (FS, DB, Web)
Ninja, GiST, IStore
4
Storage Manager

• Historic chance to start over!
• new hardware realities
• variable-length segments, not blocks
• big main memories
• extra CPUs at the devices (IStore)
• revisit and clean up infrastructure for
  transactions
• clean API supporting both log-based
  version-based schemes version-based runs today!
• big SW Eng. challenge
• unify DB/FS/Web server!
• Clients Ninjas persistent hash table, query
  processing, web server, Linux (NT?) filesystem.
• (Mohan Lakhamraju, Rob von Behren, Steve Gribble)

5
Query Engine

• Shared-nothing (cluster)
• all data flow (no blocking ops)
• auto load-balance to micro/macro changes in
  environment
• adaptivity more important than raw performance!!
• CONTROL! ripple join, online reordering
• (Shankar Raman)
• continuously reoptimizing query plans
• tie-ins with STIX (Christos/Sinclair/Russell/Helle
  rstein)
• (Ron Avnur)
• first steps in handling streaming sources

6
Cluster Data Layout

• issues fragmentation, placement, replication on
  106 disks. For DB/FS/Web.
• goals availability, efficiency, consistency,
  manageability.
• Adaptivity cooperative vs. competitive ()
  techniques?
• (Mehul Shah)

7
Global Federation

• Global distribution
• federated DBMS layer a la Mariposa/Cohera
• address all the hard stuff they dropped!
• Global data placement
• as in cluster, but must be competitive. (Mehul
  Shah)
• Global query processing (Amol Deshpande)
• Agoric query optimization
• distributed query processing
• Global metadata
• yellow pages both for services datasets
• Millenium/Ninja tie-ins?

8
Applications

• Really finding stuff in all the worlds data?
• UI meets AI meets Logic (browse/mine/query)
• CONTROL is key seamless, non-blocking
  interaction
• multi-res output and feedback during browse/query
• hints, wizards, training (AI mining, user in the
  loop)
• build on existing scalable spreadsheet/xform
  tools (Shankar Raman)