MicroTerabyte

1
MicroTerabyte

• Leveraging InfiniBand to Build a Powerful,
  Scalable Oracle Database and Application Platform
• Brian Dougherty
• Chief Architect, CMA

2
Background

• Exploding data volumes are presenting new
  challenges to small and medium sized
  organizations.
• These organizations need a new generation of
  technology that delivers powerful analytical
  capability with reduced cost and complexity.
• CMA, in partnership with Dell, QLogic, Oracle,
  and EMC, has developed a sophisticated solution
  to address this growing need.

3
MicroTerabyte PlatformWhat is It ?

• A pre-configured, integrated, fault tolerant,
  high performance, commodity hardware and software
  based runtime environment
• Leverages the power of the Oracle database
  running on commodity Linux servers, and a unified
  InfiniBand fabric over QLogic multi-protocol
  fabric directors
• Provides a scalable, reliable, lower cost
  platform for Business Intelligence, Custom
  Software, or Commercial Software

4
MicroTerabyte PlatformFeatures

• High Performance
• Scalable
• Fault Tolerant
• Commodity Hardware
• Lower Cost
• Smaller Footprint
• Reduced Power Consumption

5
MicroTerabyte PlatformAttributes

• Red Hat Linux Operating System
• Commodity Hardware (Dell / EMC CX / QLogic)
• 42U Rack Footprint with Fault Tolerant 42U rack
• Clustered Oracle Database
• Clustered Storage Provisioning Layer
• Unified Storage and Interconnect Fabric

6
MicroTerabyte Single Rack Configuration

• Sample Hardware Components (per rack)
• (2) Dell 1950 Database Servers
• 32GB RAM Each
• 16 Processor Cores
• (1) Dell 1950 ETL Server
• 16GB RAM
• 8 Processor Cores
• (1) Dell 1950 Business Intelligence Server
• 16GB RAM
• 8 Processor Cores
• (1) QLogic 9020 InfiniBand Fabric Director
• (2) FVIC Modules
• (2) EMC CX3-40 Storage Arrays
• 2TB 5TB Storage
• (1) Dell PowerConnect 48 Port GigE Switch
• (1) Belkin KVM
• (1) Dell Console

Physical Data Guard
7
MicroTerabyte Mid-Range Configuration

• Hardware components (two racks)
• (4) Dell 1950 Database Servers
• 32GB RAM Each
• 32 Processor Cores
• (1) Dell 1950 ETL Server
• 32GB RAM
• 8 Processor Cores
• (1) Dell 1950 Business Intelligence Server
• 32GB RAM
• 8 Processor Cores
• (1) QLogic 9040 InfiniBand Fabric Director
• (2) FVIC Modules
• (2) EMC CX3-40 Storage Arrays
• 8TB 15TB Storage
• (2) EMC SATA 1TB Drives (for backup)
• (1) Dell PowerConnect 48 Port GigE Switch
• (1) Belkin KVM
• (1) Dell Console

8
MicroTerabyte Large Scale Configuration

• Hardware components (three racks)
• (8) Dell 1950 Database Servers
• 32GB RAM Each
• 64 Processor Cores
• (2) Dell 1950 ETL Server
• 32GB RAM Each
• 8 Processor Cores
• (2) Dell 1950 Business Intelligence Server
• 32GB RAM Each
• 8 Processor Cores
• (1) QLogic 9040 InfiniBand Fabric Director
• (4) FVIC Modules
• (4) EMC CX3-40 Storage Arrays
• 24TB 32TB Storage
• (2) EMC SATA 1TB Drives (for backup)
• (1) Dell PowerConnect 48 Port GigE Switch
• (1) Belkin KVM
• (1) Dell Console

9
MicroTerabyte ArchitectureMid-Level Diagram
Dell 1950 PowerEdge Servers / RHEL v5
QLogic Multi-Protocol Fabric Director
EMC CX3 / CX4 Flare OS / Navisphere
10
MicroTerabyte ArchitectureDetailed Diagram
Applications
Oracle Grid Control 11g
MT Clustered Database / Storage Provisioning
Oracle EE 11g w/ RAC and Partitioning
Oracle ASM 11g
MicroTerabyte Platform
QLogic OFED InfiniBand Drivers
Oracle Clusterware 11g
MT O/S Binding
Red Hat Linux O/S
Oracle ASMlib
Public Networking
Dell PowerConnect 6248
Server Compute
Dell 1950/2950 QLogic 7104 HCA
MT Core
QLogic 9020/9040 Multi-Protocol Director
Unified Fabric
Storage Infrastructure
EMC CX3/CX4
11
MicroTerabyte PlatformOracle Software
12
MicroTerabyte Compute NodesDell 1950/2950

• MicroTerabyte solution consists of 2/4/8 RAC
  Database nodes
• 2 ETL and BI Nodes
• Each node is a Dell 1950/2950 consisting of
• Processor One/Two quad-core Intel Xeon X5355 _at_
  2.66GHz
• Memory 16-64 GB
• Hard drives 218GB-3TB Internal Storage
• RAID Controller PERC 5/i
• 1 DDR InfiniBand HCA
• Network interface cards Dual gigabit NICs
  (100baseTx-FD)
• Power supply 670W, optional hot-plug redundant
  power (11)
• Operating system RedHat Enterprise Linux v5

13
MicroTerabyte Storage NodesEMC CX4 Model 480

• Power Consumption (Disk Expansion Chassis)
• 440 VA (425W max)

14
MicroTerabyte Unified Fabric LayerWhat is it?

• QLogic 9020 9040 Multi-Protocol Fabric
  Directors
• 9020 with two (2) FVIC IB-FC Virtual I/O
  Controllers
• 9040 with up to four (4) FVIC IB-FC Virtual I/O
  Controllers
• each FVIC provides 10 DDR (20Gb) IB ports 8 4Gb
  FC ports
• supports up to 128 Virtual HBA ports per module
• automatic sensing 1/2/4 Gb/s
• load balancing
• automatic port and module fail-over
• LUN mapping and masking features
• QLogic 7104 Host Channel Adapters
• Dual Port, DDR
• IPoIB, RDS, SRP

15
MicroTerabyte Unified Fabric LayerGeneral
Benefits

• Managing one fabric
• Reduced footprint
• Compact implementation
• Fewer host components needed to support I/O and
  interconnect
• Increased bandwidth and reduced latency
• Reduced host resources (1 HCA vs. several HBAs)
• Path failover through SRP protocol
• Well positioned to take advantage of advances in
  InfiniBand technology

16
MicroTerabyte Unified Fabric LayerOracle RAC
Benefits

• Scalable platform to support Oracle RAC
• More predictable response times
• Capability to drive more Oracle I/O through fewer
  compute nodes
• Ability to exploit storage capability at a lower
  cost
• Reduced Oracle messaging latency via RDS/IB

17
MicroTerabyte ORION Benchmarks
CONFIGURATION A mid-size MicroTerabyte
configuration, including Servers Four (4) Dell
1950 Intel quad core servers. Each 1950 includes
8 cores, 16GB memory and 1 dual channel HCA. The
server is running Red Hat Linux 5 update 1
(2.6.18-53 kernel). Storage Two (2) EMC
CX3-40 storage arrays. Each storage array
includes 8 4Gbps front-end fiber channel
connections for a total of 16 4Gbps FE adapters
-- approx 7.5TB usable storage configured in 41
RAID sets. Unified Interconnect and Storage IB
fabric One (1) QLogic 9040 Multi-Protocol
Director with 2 FVIC modules HARDWARE
COST Approximate total cost (market)
500,000 TEST METHOD AND RESULTS Testing
simulator Oracle ORION -- Oracle I/O Numbers
Calibration Tool for Linux gtgt Random I/O Test
Results Test Type random I/O I/O size
8K Rate observation 1 9,600 sustained IOPS
_at_ 1.87 ms per I/O avg node latency Rate
observation 2 30,566 sustained IOPS _at_ 2.96 ms
per I/O avg node latency Rate observation 3
42,615 sustained IOPS _at_ 2.96 ms per I/O avg node
latency gtgt Sequential I/O Test Results
Test Type sequential I/O I/O size
1MB Rate observation 1 677MB/sec
sustained I/O seq. throughput _at_ 26.48 ms per 1MB
I/O avg node latency Rate observation 2
2.098GB/sec sustained I/O seq. throughput _at_ 91.48
ms per 1MB I/O avg node latency
18
MicroTerabyte Oracle Benchmarks
Hardware
Servers (4) Dell 1950
Dual Socket Quad Core 16 GB RAM per
server Single dual port IB HCA Unified
I/O Fabric QLogic 9040 (2) FVIC
Modules Storage (2) EMC CX-3 Model 40 (16)
Fibre Channel ports
Software
O/S Red Hat Linux v 5.1 Kernel
2.6.18-51 UDEV Unified I/O Fabric QLogic IB
drivers v 4.2.0.0.39 IB/RDS IB/SRP IPoIB Da
tabase Oracle RAC 10gR2 Oracle Clusterware
10gR2 Oracle ASM Oracle RMAN Oracle
DataGuard Storage Flare O/S EMC
Navisphere Backup EMC Networker
19
MicroTerabyte Benchmark Comparisons
20
Summary

• As demonstrated in CMAs MicroTerabyte platform,
  Infiniband can provide an extremely capable
  transport mechanism for unifying interconnect and
  i/o traffic
• Increases bandwidth and reduces latency
• Reduces Oracle messaging latency via RDS/IB
• Provides more predictable response times
• Reduces host resource requirements (i/o
  processing workload off-loaded to HCA card)
• Consistent with Oracles strategic technology
  direction

21
For More Information
Brian Dougherty Chief Architect,
CMA bdougherty_at_cma.com