The Storage Resource Broker and the NGS

1
The Storage Resource Brokerand the NGS
Slides from Wayne Schroeder, SDSC andPeter
Berrisford, RAL
2
Policy for re-use

• This presentation can be re-used for academic
  purposes.
• However if you do so then please let
  training-support_at_nesc.ac.uk know. We need to
  gather statistics of re-use no. of events,
  number of people trained. Thank you!!

3
Acknowledgements

• This tutorial selects slides from several
  sources, specifically from talks given by Wayne
  Schroeder (SDSC) and Peter Berrisford (RAL)

4
Goal

• Introduce use of the SRB for distributed file
  management on the NGS
• This is the focus of the practical that follows
• NOTE
• Wayne Schroeder SRB does so much, people tend
  to learn subsets and are often unaware of useful
  features
• So explore further!
• http//www.sdsc.edu/srb/
• For a full SRB tutorial, see http//www.niees.ac.
  uk/events/srb

5
What is SRB? (1 of 3)

• The SDSC Storage Resource Broker (SRB) is
  client-server middleware that provides a uniform
  interface for connecting to heterogeneous data
  resources over a network and accessing unique or
  replicated data objects.
• SRB, in conjunction with the Metadata Catalog
  (MCAT), provides a way to access data sets and
  resources based on their logical names or
  attributes rather than their names and physical
  locations.

6
What is SRB? (2 of 3)

• The SDSC SRB system is a comprehensive
  distributed data management solution, with
  features to support the management, collaborative
  (and controlled) sharing, publication, and
  preservation of distributed data collections.
• The SRB also serves as middleware via a rich set
  of APIs available to higher-level applications
  and by providing a management layer on top of a
  wide variety of storage systems.

7
What is SRB? (3 of 3)

• The SRB is an integrated solution which
  includes
• a logical namespace,
• interfaces to a wide variety of storage systems,
• high performance data movement (including
  parallel I/O),
• fault-tolerance and fail-over,
• WAN-aware performance enhancements (bulk
  operations),
• storage-system-aware performance enhancements
  ('containers' to aggregate files),
• metadata ingestion and queries (a MetaData
  Catalog (MCAT)),
• user accounts, groups, access control, audit
  trails, GUI administration tool
• data management features, replication
• user tools (including a Windows GUI tool (inQ), a
  set of SRB Unix commands, and Web (mySRB)), and
  APIs (including C, C, Java, and Python).
• SRB Scales Well (many millions of files,
  terabytes)
• Supports Multiple Administrative Domains / MCATs
  (srbZones)
• And includes SDSC Matrix SRB-based data grid
  workflow management system to create, access and
  manage workflow process pipelines.

8
SRB Projects

• Digital Libraries
• UCB, Umich, UCSB, Stanford,CDL
• NSF NSDL - UCAR / DLESE
• NASA Information Power Grid
• Astronomy
• National Virtual Observatory
• 2MASS Project (2 Micron All Sky Survey)
• Particle Physics
• Particle Physics Data Grid (DOE)
• GriPhyN
• SLAC Synchrotron Data Repository
• Medicine
• Digital Embryo (NLM)
• Earth Systems Sciences
• ESIPS
• LTER
• Persistent Archives
• NARA
• LOC

Over 90 Tera Bytes in 16 million files
9
SRB Scalability
10
What is SRB?

• Storage Resource Broker (SRB) is a software
  product developed by the San Diego Supercomputing
  Centre (SDSC).
• Allows users to access files and database objects
  across a distributed environment.
• Actual physical location and way the data is
  stored is abstracted from the user
• Allows the user to add user defined metadata
  describing the scientific content of the
  information

11
How SRB Works

• 4 major components
• The Metadata Catalogue (MCAT)
• The MCAT-EnabledSRB Server
• The SRB Storage Server
• The SRB Client

12
SRB Client Tools

• Provide a user interface to send requests to the
  SRB server.
• 4 main interfaces
• Command line (S-Commands)
• MS Windows (InQ)
• Web based (MySRB).
• Java (JARGON)
• Web Services (MATRIX)

13
Planned Deployment on NGS
Database Servers _at_ Manchester
SRB Server
MCAT
DB1
DB n
MCAT Server _at_ Manchester
Online Replication
Failover link
User
MCAT Server _at_ RAL
SRB Server
MCAT
DB1
DB n
Database Servers _at_ RAL
Disk Farm
Disk Farm
Disk Farm
Disk Farm
14
Summary

• SRB provides NGS users with
• a virtual filesystem
• Accessible from all core nodes and from the UI
  / desktop
• (will provide) redundancy mirrored catalogue
  server
• Replica files
• Support for application metadata associated with
  files

15
Tutorial Overview

• Use of the Scommands
• Commands for unix based access to srb
• Strong analogy to unix file commands
• Accessing files from multiple (two) sites

16
Getting Started

• Launch two putty connections to pub-234
• One for commands run on pub-234
• One for connecting to grid-data.rl.ac.uk and
  running commands from there
• Open browser and follow link from
  http//homepages.nesc.ac.uk/gcw/NGS/
• Follow the instructions from there.
• Your SRB name is the same as your account on
  pub-234