ITEP computing center and plans for supercomputing

1
ITEP computing center and plans for supercomputing

• Plans for Tier 1 for FAIR (GSI) in ITEP
• 8000 cores in 3 years, 2000-3000 in this year
• Distributed parallel filesystem of 1 PB in 3
  years, 100-200 TB in this year
• 150-200 kW of power (35-40 kW in this year)

2
ITEP computing center and plans for supercomputing
3
ITEP computing center and plans for supercomputing
4
ITEP computing center and plans for supercomputing

• Hardware
• 7U blade system with 10 twin modules (20 nodes
  per blade chassis)
• 3 blade chassis enclosures with power supply per
  42U rack. 20-25 kW per 42U rack.
• Two 36 QSFP ports infiniband switches per blade
  chassis
• 36 QSFP porrts infiniband switches for the second
  level of fat tree
• 2 x AMD 12 cores CPU per node
• 64 GB of RAM per node
• Two channel of 4xQDR Infiniband per node for
  interprocess communication

5
ITEP computing center and plans for supercomputing

• Infiniband topology
• Two levels fat tree using 36 x QSFP ports
  infiniband switches

6
ITEP computing center and plans for supercomputing

• Software
• RedHat based distribution (Scientific Linux or
  CentOS) for x86_64 architecture.
• TORQUE batch system with maui as scheduler.
• OpenMPI with TORQUE integration (mvapich and
  mvapich2 is under consideration)
• OpenMP
• BLAS, lapack including ATLAS versions, ACML

7
ITEP computing center and plans for supercomputing

• Prototype
• 7U blade system
• 10 twin blade modules, 20 nodes
• 36 QSFP ports switch module
• 22x1GbE 3x10GbE ports Ethernet switch module
• Node characteristics
• Dual Xeon X5650 2.67GHz 6 cores
• 32GB RAM
• 500 GB disk
• One 4xQDR Infiniband port
• Dual 1GbE Ethernet (one channel connected)

8
ITEP computing center and plans for supercomputing

• Prototype software configuration
• CentOS 5.6 x86_64
• TORQUE batch system
• maui scheduler
• OpenMPI 1.4 integrated with TORQUE
• BLAS, lapack including ATLAS version

9
ITEP computing center and plans for supercomputing

• Benchmarking
• Single node (12 processes)
• Linpack (N60000, NB128, P4, Q3) - 98 Gflops
  
• (77 of theoretical performance)
• One process per node (12 processes)
• Linpack (N60000, NB128, P4, Q3) - 100
  Gflops
• (78 of theoretical performance)
• Cluster full load (240 processes)
• Linpack (N250000, NB128, P16, Q15) - 1800
  Gflops
• (70 of theoretical performance)

10
ITEP computing center and plans for supercomputing

• Distributed parallel filesystem
• Glusterfs-3.2.1 with RDMA and TCP as transport
• Local disks of the nodes are used for glusterfs
  volumes
• Replication (mirroring) of data provides fault
  tolerance
• Linear speed for 1 file Write 46 MB/s, Read 84
  MB/s
• Bandwidth for cluster Write 450 MB/s, Read 839
  MB/s

11
ITEP computing center and plans for supercomputing

• Infiniband bandwidth measurements for prototype
• Approximated bandwidth for cluster 56 GByte/s
  for 20 nodes (bidirectional)

12
ITEP computing center and plans for supercomputing

• Zabbix monitoring system
• Active client with vast possibility of
  customization
• SNMP monitoring and traps
• IPMI monitoring and control
• Triggers and events for group of hosts including
  the usage of the aggregate functions
• Powerful and flexible tools for triggers and
  actions description
• Presentation of data in many ways

13
ITEP computing center and plans for supercomputing

• Example of presentation data from different
  sources in ZABBIX

14
ITEP computing center and plans for supercomputing

• Next steps
• Dedicated storage with lustre filesystem
• First stage of the supercomputer with AMD
  processors
• Two levels fat tree infiniband topology
• Moving to RHEL 6 based operating system

15
ITEP computing center and plans for supercomputing

• Thank you