V LDB

1
V LDB
2
Boris Gelman Vice President Architecture Informati
on Services VISA bgelman_at_visa.com
2
V LDB The Concept
2
2

• V LDB Very Very Large Database
• New concept or change to VLDB concept ?
• Data Structure
• Petabyte tables with 100s billions of rows
• Complex table structures
• Non-uniform physical data representation of
  petabyte tables
• Query
• Well-defined subsets (index and/or partition) on
  tables small (10,000) -gt medium (300,000) -gt
  large (1,000,000)
• Undefined subsets very large (1,000,000,000)
  -gt very very large (100,000,000,000)
• Complex joins
• Complex group bys and sorts
• Workload
• Multiple categories of queries running
  concurrently (transaction research, analytics,
  data mining)
• Inserts and selects concurrently against the
  same tables
• 24 7 operation with very limited maintenance
  windows
• SLAs are very strict

3
V LDB Problems
2

• Data Partitioning
• Smart partitioning hash, expression, -gt
  hybrid multi-level partitioning
• Smart partition manipulation detach / attach
  partition online
• Query Execution
• Hash join on petabyte tables ?
• Performance Tuning does not work
• Adaptive and buffer-pool aware query
  optimization ?
• System-category aware query optimization ?
• Optimizer efficiency ?
• Backup/Restore does not work
• Data replication is not a substitute for backup
  data corruption, application errors, human errors
• Smart backup/restore related to smart data
  partitioning !

4
V LDB Problems
2

• Database Federation
• Single database system cannot hold a combination
  of ODS (gt 1 PB) and cross-functional
  multi-subject DW (gt 200 TB) - it is impractical
• Data Abstraction Layer federated tables
  partitioned across multiple database systems!
• Federated Database is easier to maintain and
  backup, and availability is higher!
• Federated Database Performance Single Database
  System Performance !!!