A Redundant Global Storage Architecture

1
A Redundant Global Storage Architecture

• Jim Leek, David Schultz, Ben Schwarz
• jrleek,dschultz,bschwarz_at_CS

2
Storage as a Service

• Many companies already provide access to massive
  data sets as a service (e.g. Google)
• Provide access to raw storage as a service
• Advantages
• Already know how to manage storage clusters
• More reliable than personal storage
• Available anywhere
• Disadvantages
• Security?

3
Storage as a Service

• Clients allocate large chunks of storage
• 64 MB for now
• Cost based on bandwidth and total space
• Bandwidth is most important
• Disk BW increasing at 40/yr, capacity at 60/yr
• Space costs money too
• Making data always available consumes power
• Backups
• More to recover when a disk is lost

4
Network Architecture

• Storage network is a collection of chunk servers
• Space allocation and deallocation is centralized
  at the master
• Stateless design simple recovery

5
Network/Translation Layer
Master Server (lookups) Chunk Server
(storage)

• Responsible for replicating
• Fault tolerant
• Higher bandwidth requests
• Buffers requests

• Central authority
• Small number
• Fault tolerant
• Low bandwidth
• Distributes load
• Block location cache

Forward bytestream
read write openIOGroup closeIOGroup
whoIsMaster
allocate free locateBlock
6
Filesystem Design

• Encrypted on the client side
• Always consistent
• Structured as a Merkle tree
• Copy on write
• Inexpensive snapshots reduce the impact of user
  error
• Block allocation/cleaning similar to LFS
• Local WAL to provide fast synchronous writes when
  required

7
Updating a Block
8
Updating a Block
9
Updating a Block
10
Updating a Block
11
Updating a Block
12
I/O Model

• Servers provide no transaction support
• Servers only see encrypted data anyway
• xFS designers showed how to do isolation entirely
  on the client side
• Server respects WAW dependencies
• Clients attach each write to an I/O group
• Groups are committed in order

13
Security
Thread Model
CS
Attacks
CS

• MIM
• Deduce data loc.
• Timing attacks
• Statistical attacks
• Compromised CS?

Alice
CS
e.g. file size
Bob
CS
Improvement over existing architectures!
Even chunkservers are not in-the-know
14
Security II

• Filesystem - Encrypt data prior to storing.
  Decrypt on retrieval. Not a fundamental
  limitation of OceanStore.
• How do we handle data that should be read by a
  set of people (e.g. group privileges)? One public
  key. Distribute multiple private keys. Use GPG!

DATA Privi(Pub(DATA)) Privk(Pub(DATA))

• Multiple private keys for member identification
• Or use a single private key for anonymity

15
Security Guarantees

• MACs guarantee that the server cant change the
  data
• Hypothetical attack server selectively ignores
  writes, inconsistent versions trick the client
• Merkle tree structure provides fork consistency
• Informally, if the server prevents clients A and
  B from seeing each others updates, the clients
  will either detect the problem or live in
  parallel universes from then on
• Freshness?

16
Conclusion

• Improvement in security and privacy over existing
  architectures
• Simple stateless server design makes recovery
  easier
• AFS and xFS have problems with recovery
• Virtually unconstrained write ordering should
  provide good performance

17
Questions
18
Related Work

• OceanStore
• Distributed object location
• Google Filesystem
• Master/Chunkserver architecture
• LFS
• Large sequential writes
• WAFL
• WAL for synchronous semantics
• Tree of blocks for consistency