Optimisation of Grid Enabled

1

• Optimisation of Grid Enabled
• Storage at Small Sites
• Jamie K. Ferguson
• University of Glasgow
• email J.Ferguson_at_physics.gla.ac.uk
• Jamie K. Ferguson University of
  Glasgow
• Graeme A. Stewart - University of
  Glasgow
• Greig A. Cowan - University of
  Edinburgh

2
Introduction

• Typical Tier 2 Purpose of the Inbound Transfer
  Tests
• Details of the hardware/software configuration
  for the File Transfers
• Analysis of Results

3
LHC and the LCG

• LHC most powerful instrument ever built in the
  field of physics
• Generate huge amounts of data every second it is
  running
• Retention of 10PB annually to be processed at
  sites

• Use case is typically files of size GB, many of
  which are cascaded down to be stored at T2s until
  analysis jobs process them

4
Typical Tier2 - Definition

• Limited Hardware Resources
• (In GridPP) Using dCache or dpm as SRM
• Few (one or two) Disk Servers
• Few Terabytes of RAIDed Disk
• Limited Manpower
• Not enough time to Configure and/or Administer a
  Sophisticated Storage System
• Ideally want something just to work out of the
  box

5
Importance of Good Write (and Read) Rates

• Experiments Desire Good in/out Rates
• Write more stressful than read, hence our focus
• Expected data transfer rates (T1gtT2) will be
  directly proportional to the storage at a T2 site
• Few 100Mbps for small(ish) sites up to several
  Gbps for large CMS sites
• Limiting Factor could be one of many things
• I know this from recently coordinating 24hour
  tests between all 19 of the GridPP T2 member
  institutes
• We also yielded file transfer failure rates

6
Glite File Transfer Service

• Used FTS to manage transfers
• Easy to use file transfer management software
• Uses SURLs for source and destination
• Experiments shall also use this software
• Able to set channel parameters Nf and Ns
• Able to monitor each job, and each transfer
  within each job.
• Pending, Active, Done, Failed, etc.

7
What Variables Were Investigated?

• Destination srm
• dCache (v1.6.6-5)
• dpm (v1.4.5)
• The underlying File system on the destination
• ext2, ext3, jfs, xfs
• Two Transfer-Channel Parameters
• No. of Parallel Files
• No. of GridFTP Streams

• Example gt Nf5, Ns3

8
Software Components

• Dcap and rfio are the transportation layers for
  dCache and dpm respectively
• Under this software stack is the filesystem
  itself. e.g. ext2
• Above this stack was the filetransfer.py script
• See http//www.physics.gla.ac.uk/graeme/scripts/
  filetransferfiletransfer

9
Software Components - dpm

• All the daemons of the destination dpm were
  running on the same machine
• dCache had a similar setup in terms everything
  housed in a single node

10
Hardware Components

• Source was a dpm.
• High performance machine
• Destination was single node dual core Xeon CPU
• Machines were on same network.
• Connected via 1GB link which had negligible other
  traffic.
• No firewall between source and destination
• No iptables loaded
• Destination had three 1.7TB partitions
• Raid 5
• 64K stripe

11
Kernels and Filesystems

• A CERN contributed rebuild of the standard SL
  kernel was used to investigate xfs.
• This differs from the first Kernel only in the
  addition of xfs support
• Instructions on how to install kernel at
  http//www.gridpp.ac.uk/wiki/XFS_Kernel_Howto
• Necessary RPMs available from ftp//ftp.scientific
  linux.org/linux/scientific/305/i386/contrib/RPMS/x
  fs/

12
Method

• 30 source files, each of size 1GB were used
• This size is typical of the sizes of LCG files
  that shall be used by LHC experiments
• Both dCache and dpm were used during testing
• Each kernel/Filesystem was tested - 4 such pairs
• Values of 1,3,5,10 were used for No. Files and
  No. Streams - giving a matrix of 16 test results
• Each test was repeated 4 times to attain a mean.
• Outlying results ( lt 50 of other results) were
  retested
• This prevented failures in higher level
  components e.g. FTS adversely affecting results

13
Results Average Rates

• All results are in Mbps

14
Average dCache rate vs. Nf
15
Average dCache rate vs. Ns
16
Average dpm rate vs. Nf
17
Average dpm rate vs. Ns
18
Results Average Rates

• In our tests dpm outperformed dCache for every
  average Nf, Ns

19
Results Average Rates

• Transfer rates are greater when using jfs and xfs
  rather than ext2 or ext3
• Rates for ext2 are better than ext3 due to the
  fact that ext2 does not suffer from journalling
  overheads

20
Results - Average Rates

• Having more than one Nf on the channel
  substantially improves the transfer rate for both
  SRMs and for all filesystems. And for both SRMs,
  the average rate is similar for Nf 3,5,10
• dCache
• Ns 1 is the optimal value for all filesystems
• dpm
• Ns 1 is the optimal value for ext2 and ext3
• For jfs and xfs rate seems independent of Ns
• For both SRMs, the average rate is similar for Ns
  3,5,10

21
Results Error (Failure) Rates

• Failures, in both cases tended to be caused by a
  failure to correctly call srmSetDone() in FTS
  resulting from a high machine load
• Recommended to separate the SRM daemons and disk
  servers, especially at larger sites

22
Results Error (Failure) Rates

• dCache
• small number of errors for the ext2 and ext3
  filesystems
• caused by high machine load
• No errors for the jfs and xfs filesystems
• dpm
• all filesystems had errors
• As in dCache case, caused by high machine load
• Error rate for jfs was particularly high, but
  this was down to many errors in one single
  transfer

23
Results FTS Parameters

• Nf
• Initial tests indicate that Nf set at a high
  value (15) causes a large load on machine when
  first batch of files completes. Subsequent
  batches time-out.
• Caused by post-transfer SRM protocol negotiations
  occurring simultaneously
• Ns
• gt 1 caused slower rates for ¾ of the
  SRM/filesystem combinations
• Multiple streams causes a file to be split up and
  sent down different TCP channels
• This results in random writes to the disk.
• Single streams cause the data packets to arrive
  sequentially and can be written sequentially also

24
Future Work

• Use SL4 as OS
• allows testing of 2.6 kernel
• Different stripe size for RAID configuration
• TCP read and write buffer sizes
• Linux kernel-networking tuning parameters
• Additional hardware, e.g. More disk servers
• More realistic simulation
• Simultaneous reading/writing
• Local file access
• Other filesystems?
• e.g. reiser, but this filesystem is more
  applicable to holding small files, not the sizes
  that shall exist on the LCG

25
Conclusions

• Choice of SRM application should be made at site
  level based on resources available
• Using newer high performance filesystem jfs or
  xfs increases inbound rate
• Howto move to xfs filesystem without loosing data
  http//www.gridpp.ac.uk/wiki/DPM_Filesystem_XFS_Fo
  rmatting_Howto
• High value for Nf
• Although too high will cause other problems
• Low value for Ns
• I recommended Ns1 and Nf8 for GridPP inter-T2
  tests that I'm currently conducting