iSAN

1
iSAN

• Ganesh Narayan
• CASL, IISc

2
Agenda

• Motivation
• Design Criteria
• Architecture
• Implementation
• Case Studies
• concurrency ctrl, security, stable storage
• Conclusions

3
MOTIVATION

• Storage Systems Primer, NAS SAN
• Problems

4
Motivation

• Storage Service
• Data centric
• Web, E-Commerce, Storage Service Providers
• Increasing Storage Demand
• Moores Law Aint Enough! (but could help!)
• Storage Architectures
• DAS
• NAS
• SAN

5
Motivation

• Direct Attached Storage
• directly connected to the storage-consumer

6
Motivation

• Network Attached Storage
• rich, typed, variable length, hierarchical file
  abstraction of storage

NAS
7
Motivation

• Storage Area Network
• untyped, fixed size, block abstraction of storage

SAN
8
Motivation

• SANs - throughput scalability
• Technologies Fibre Channel iSCSI
• However . . .
• Concurrency Control
• SAN Security
• SAN Services I/O Fencing, Stable Storage
• Storage Consumer Awareness
• Intelligence in SAN helps!

9
DESIGN

• Criteria, State of the Art FC iSCSI
• Problems

10
Design Criteria

• Interoperability
• Permutations and Combinations
• FC zoning, FSPF, private/public loop
• Throughput
• iSCSI zerocopy, hw chksum, seq no wrapping
• FC flow ctrl iSCSI cong ctrl, failure
  detection
• Availability
• Network Storage Consumer Availability

11
Design Criteria

• Security
• Migrating to SAN, on-store/in-wire Security
• iSCSI IPSec, IKE chksum(s), mcast sec
• Storage Sharing
• Fail-over, Scalability, Flexible Load Balancing
• I/O Fencing, Concurrency Control
• Intelligence
• QoS, VPN, Multicast KISS may not work!
• Different Consistency/Security guarantees
• Storage Consumer Awareness

12
ARCHITECTURE

• Proposed Architecture, Literature
  Corroboration

13
Architecture

• Storage Consumer Groups
• Logical Link Control
• Virtual LAN
• Group Communication System
• Strong Virtual Synchrony
• Protocol Composition
• Target Emulation
• Let the Sanlet be!
• resides in the edge switches emulates scsi
  target forms a vsync group spanned by vlan

14
Architecture

• iSAN - Schematic

15
Architecture

• iSAN Data Flow

PS Proto Stack, SC Storage Consumer, T
Target, S - Storage
16
Architecture

• iSAN Protocol Composition

PS
T
T
LLC / VLAN
17
Architecture

• Corroboration
• Interoperability - LLC, VLAN
• Throughput - LLC Type 1
• Availability - Vsync, still MST
• Security - Ensemble Security
• Storage Sharing - Vsync, cctrl
• Intelligence - Protocol Composition

18
IMPLEMENTATION

• Platform, Software Developed, Ported, Used
• Traces

19
Implementation

• Platform
• Linux 2.4.
• Tools
• Developed
• Ensemble Compatible SCSI Target Emulator
• LLC transport for Ensemble CEnsemble
• Ported
• CEnsemble to Linux 2.4.
• Used
• PGP, OpenSSH Lib

20
Implementation

• iSAN Configuration
• iSAN Script
• static configuration
• read during Sanlet initialization
• two flavours c like, value param pairs
• iSAN in-band control messages
• dynamic control
• iSAN script dictates if the Sanlet should be
  dynamically controllable

21
Implementation

• Traces
• HP Trace
• HP, Multiple Disks, Development Environ
• Postmark
• NetApp, Single Disk, Heavily loaded Web Server
• GCC unpack-configure-build
• Single Disk, Long Lived Large Transfers
• SSH /SSL unpack-configure-build-unlink
• Single Disk, Short, metadata intensive load

22
Performance
23
CASE STUDIES

• Concurrency Control, Storage Security
• Stable Store

24
Case Studies -1

• Concurrency Control
• Logical Vs Physical View of the Storage
• multiple levels of abstractions
• Theory Practice - Strict Schedules
• Locking (2PL)
• Time Stamp Ordering (BTO)
• Optimisitc
• Hybrid
• Time Stamp Ordering BTO1, BTO2
• High contention - most effective of the cctrl
  protocols

25
Case Studies -1

• Time Stamp Ordering
• clk synchronization
• high speed is actually high throughput!
• also, depends on msg delay variance, not speed
• mis-ordered transactions
• receive order being different from sent order
• ordering violations
• causality violation
• differing transaction semantics
• File Systems and Databases

26
Msg Delay Variance - No Hop
27
Msg Delay Variance One Hop
28
Msg Delay Variance Two Hops
29
Case Studies -1

• Soln Message Ordering Protocols
• No clock synchronization, mis-ordered
  transactions, causality violation
• An Intelligent Solution
• Causally Constrained Total Order - Replication
• Unconstrained Total Order - DB, Replication
• Causal Order - Hybrid Storage
• FIFO - PFS, VxFS

30
Case Studies 1p

• Concurrency Control - Performance
• Relative Costs of Ordering Protocols
• FIFO Ordering of Journal Records

31
Case Studies - 2

• SAN Security
• Zoning
• Soft Hard VLANs
• LUN Masking
• Switch Based
• Cryptography
• Fortress Model - Ensemble
• Multicast and Membership Aware
• 3DES MD5
• Smart Security
• CBC DES for Journals, Selective Encryption

32
Case Studies 2p

• iSAN Security - Performance
• CBC for Journal
• Selective Encryption 7

33
Case Studies - 3

• Stable Storage
• Ideal, fault tolerant store
• Lampson, RAID
• Persistent Vs Transient Stable Storage
• Redundancy disk, remote memory
• Virtual Synchrony
• mutual consistency - dynamic non-uniform
  guarantee
• Data with temporal Validity
• Journals metadata intensive/synchronous work
  loads

34
Case Studies - 3

• Intelligent Stable Storage
• Transient Stable Storage Journals
• Journal data replicated in remote Sanlets
• Partition Tolerant - 2 f 1 Sanlets
• Switch to persistent stable storage mode if
  Group cardinality is less than 2 f 1
• Safe delivery on demand

35
Case Studies 3p

• Stable Storage - Performance
• Transient Stable Storage for VxFS Journal Records

36
CONCLUSIONS

• iSAN in retrospection, Improvements
• Future Directions

37
Conclusions

• iSAN is not a panacea, not yet!
• Interface Design Issues export, instruct,
  collob
• Network Processor Realization, MST
• Cost / Complexity Tradeoffs
• Future Directions
• Attribute Based Storage QoS, Migration
• Prefetching / (Exclusive) Caching at the Edge
  Switches
• Automatic Protocol Composition

38
Thanks!

• The Road goes ever on and onDown from the door
  where it began.Now far ahead the Road has
  gone,And I must follow, if I can,Pursuing it
  with eager feet,Until it joins some larger
  wayWhere many paths and errands meet.And
  whither then? I cannot say.
• J.R.R.Tolkien
• The Lord Of The Rings -- A Long Expected Party

39
Surplus..

• Extra Slides

40
Experimental Setup

• Machines
• 1 - 2 GHz, 1 1.5G (embedded trace feeding)
• Network
• 1Gbps shared Ethernet
• Disk
• FC JBOD, soft RAID 5 over 5 disks QLogic HBA
• GCS
• Ensemble v1.39, cEnsemble v0.10
• Crypto
• PGP v6.5.8 OpenSSL v0.96b
• Traces
• HP traces, ssh, ssl, gcc compilations, postmark

41
Related Work

• Interposed request routing for scalable network
  storage
• OSDI 2000
• Façade virtual storage devices with performance
  guarantees
• FAST 04
• NASD
• FAST 04

42
References

• BTO1
• Highly concurrent shared storage
• ICDCS 00
• BTO2
• Network-Aided Concurrency Control in Distributed
  Databases
• phD thesis, vtech 02

43
Design Criteria Revisited