GridSite Storage

1
GridSite Storage

• Andrew McNab
• University of Manchester

2
Outline

• GridSite's evolution
• File servers
• htcp command
• Storage farms
• File location
• POSIX access
• SRM etc

3
GridSite evolution

• Started as web content management
• www.gridpp.ac.uk
• library-ised, for reuse of GridSite components
• EDG/LCG Logging Bookkeeping LCAS
• GridSite CGI becomes GridSite Apache module
• 3rd party CGI/PHP on top of this GOC etc
• Web Services like gLite WM Proxy on CE's
• Storage is current expansion area for GridSite

4
GridSite philosophy

• Aim to reuse as much as possible from mainstream
  Web and Web Services worlds
• Applies both to software and standards
• Reduces work needed and ongoing support overhead
• We use Apache, OpenSSL, curl, gSOAP, libxml, ...
• Aim for ease of configuration and operation
• Try to keep everything in httpd.conf file
• Autoconfigure hostname etc as much as possible

5
File servers

• Apache web servers are already simple file
  servers
• GridSite adds directory or file level access
  control, in terms of certificate DN, lists of DNs
  or VOMS attributes
• Also allows PUT, MOVE and DELETE http(s) methods
  for writing to disk
• All specified by the RFCs, but usually not
  implemented
• Along with third-party COPY method, this gives
  Apache mod_gridsite very similar functionality
  to GridFTP
• but with the fine grained, VOMS-aware access
  control

6
htcp command

• htcp is similar to scp (or globus-url-copy)
• Allow copy of files to or from remote server,
  using grid proxy, VOMS etc credentials
• htcp /tmp/myfile.txt https//grid0.hep.man.ac.uk/d
  ir/
• Variants htls, htll, htrm, htmv also allow users
  to examine remote directories, delete files or
  rename them.
• htls https//grid0.hep.man.ac.uk488/dir/
• Support GridSite's GridHTTP mode for
  authentication via HTTPS but bulk file copy via
  HTTP.

7
Storage farms

• As we've gone up in storage, have gone up in
  number of nodes with storage disk
• BaBar UK c.2000 with a Sun raid array and farm of
  Linux CPU nodes at each site
• Sites like Manchester, all storage now on CPU
  nodes
• Traditional cluster mechanisms like NFSautomount
  don't scale up to 1000 nodes
• So dCache, DPM, xrootd etc have emerged to give
  access to these disks from other CPU nodes

8
GridSite storage

• Our idea is to use GridSite/Apache file servers
  on the CPU/Disk nodes
• Aim to be as democratic as possible, since it
  removes single points of failure/overload
• Query the state of the files on the disk rather
  than duplicate this information in a database
• This will require use of lock-files on disk for
  some meta data (compare pool accounts)
• Provide access via HTTP(S), GridFTP, htcp, POSIX
  files

9
File location

• How to find files on a node without a database?
• By querying the nodes directly.
• We do this with multicast
• RFC2756 describes HyperText Cache Protocol which
  we use to format Do you have? queries and
  responses
• Added a multicast UDP responder to
  GridSite/Apache module, configured via 2
  SiteCast lines in httpd.conf
• File server just looks for file and replies if
  it has it
• HTCP round trip time between client and server is
  usually between 200 and 900 microseconds

10
POSIX file access

• htcp command supports SiteCast file location
• htcp --domain sitecast.hep.man.ac.uk
  --groups 224.0.0.111777 https
  //sitecast.hep.man.ac.uk488/file.txt
  /tmp/file.txt
• But would like to be able to access files on
  other nodes
• without applications having to know about this
• or without having to copy files temporarily to
  this CPU node
• Need POSIX-like access, as we had with NFS etc
• So we've revived the SlashGrid part of GridSite
• This hasn't been actively developed since 2003

11
SlashGrid

• Use FUSE kernel module (mainstream in Linux
  2.6.14)
• Connects slashgrid daemon to the /grid part of
  the filesystem
• Daemon acts on open(), read(), write(), unlink()
  etc.
• We use the code from htcp commands to generate
  HTTP(S) GET, HEAD, PUT, MOVE, DELETE requests
• either absolute URLs or SiteCast location URLs
• Uses GSI proxies (including VOMS) if present
• emacs /grid/https/n0.hep.man.ac.uk488/mcnab/notes
  .txt
• TFileOpen(/grid/https/sitecast.hep.man.ac.uk/d1
  /file34.root)

12
GridFTP access

• Clients currently assume GridFTP access is
  available
• So we want to give GridFTP access to files in
  /var/www/html/
• But without breaking .gacl access control
• We've added a /grid/local/ filesystem
• maps requests to local /var/www/html/ directory
• enforces any .gacl access restrictions
• identifies user from pool accounts, applies DN
  Lists (including from VOMS)
• Run standard GridFTP server on this filesystem,
  in chroot mode
• SiteCast works with gsiftp// URLs and
  /grid/local/ directories

13
So what we have...

• Transparent access to files on any other CPU/Disk
  node on the local farm
• No need to maintain a database of file location
• No resyncing, backing up database, building DB
  farm etc
• Files on dead nodes automatically disappear
• Unless there is another replica, which is used
  automatically
• Read/write access via HTTP(S) (htcp, wget,
  Firefox), GridFTP (globus-url-copy, lcg-cp, ...)
  and POSIX (/grid/...)
• All the fine-grained, VOMS-aware access control
  from GridSite is available, irrespective of the
  access protocol

14
What's needed

• SRM of course!
• We are designing an SRM using SiteCast as a
  backend instead of a database
• Map SRM chmod functions to modification of GACL
  policies
• This gives us VOMS level access control of files
• Be able to use SiteCast to locate free space
  reservation
• Create space lockfiles to reserve space (ie
  sparse files)
• Global disk quotas across the site
• Allocate disk on N nodes to a VO?

15
What's needed (2)

• A set of scripts or services which
• monitor SiteCast requests to identify busy
  files and make more replicas of them
• remove unused/expired files
• enforces changes to global quotas by shifting /
  expiring files
• This way of working means there is no state
  stored in a site management box
• it can go down, be rebooted, reinstalled etc and
  the day to day business of running jobs carries
  on.

16
Advantages

• SiteCast makes it easy to find replicas,
  ignoring dead nodes
• GridSite supports searching multiple multicast
  groups in order
• Query this rack, then other racks, then other
  machine room.
• Could form virtual Tier-2's by sharing multicast
  groups
• SRM at each physical site is able to find files
  located in the others. CPU nodes can
  transparently access them via /grid
• SlashGrid retries on server error, including the
  SiteCast query
• Will automatically switch to another replica even
  during read()
• Admins can replicate files off nodes to be taken
  down without disturbing running jobs

17
Architecture
Client application
GridFTP
htcp cmds
/grid/
HTTP(S)
Queries by multicast
GridFTP in chroot
Apache/mod_gridsite UDP responder and HTTP(S)
file access
/grid/local/
Node1 /var/www/html/...
Node2
Node3
Node4
18
Conclusion

• Have combined GridSite file servers and SlashGrid
  clients to provide transparent access to files on
  a storage farm.
• Access also by HTTP/HTTPS/GridHTTP and GridFTP
• This uses multicast HTCP queries (SiteCast) to
  find replicas of files
• Have avoided the need for a database, by ensuring
  file operations on servers are atomic and using
  actual file states
• Now looking at adding an SRM interface and space
  location reservation via SiteCast