SlashGrid (

1
SlashGrid (/grid)
a framework for Grid-aware filesystems

• Motivation dynamic-accounts issues
• Local storage implementation alternatives
• Generalisation remote file access
• Implementation Coda, ACLs, plugins
• Current status
• Future work

Andrew McNab - Manchester HEP - 5 March 2002
2
Motivation dynamic accounts

• For TB1 we provided a patch for Globus
  gatekeeper, gsi_wuftpd etc to associate Unix UIDs
  from a pool with the Grid DN identities of
  incoming requests.
• This is ok when all jobs do on the machine in
  question is computation.
• But (1) any files created by pool UID need to
  cleaned up before account can be reallocated.
• But (2) no good for long term storage, since no
  promise to maintain UID-DN association in long
  term.
• But (3) what if malicious user creates a cron
  entry, writes to some obscure writeable directory
  we didnt think of, etc?

Andrew McNab - Manchester HEP - 5 March 2002
3
Solution get away from UID filesystems

• All these problems are fundamentally because
  files are owned according to UID, but we want UID
  to have no long term meaning.
• Obvious solution is to have a filesystem where
  file ownership depends on Grid DNs not temporary
  UIDs.
• Can then ban user processes from writing anywhere
  else (straightforward to impose this with a
  modified ext2 device driver eg no disk files can
  be created if UID gt 99)
• UID becomes as transitory as Process Group ID.
• Problem now becomes how to implement a DN/Grid
  aware filesystem?

Andrew McNab - Manchester HEP - 5 March 2002
4
Implementation alternatives

• 1) Fake a filesystem by making user process use
  modified versions of open(), read() etc system
  calls.
• Can do this by relinking, or by an interposition
  / bypass library that is preloaded before real,
  shared libc.
• But, this cannot enforce access restrictions on
  files accessible on local disk (since you can use
  a static binary and ignore permissions)
• Need to put filesystem behind a server, accessed
  via TCP ports, named pipe, or shared memory (all
  the usual X tricks.) This going to be slow for
  streaming large files the very thing we need to
  be fast.
• 2) Put filesystem into kernel
• Lets kernel enforce access control. Potentially
  as fast as normal disk.
• User space daemon useful to parse proxies, and do
  any remote IO.

Andrew McNab - Manchester HEP - 5 March 2002
5
Coda

• A suitable kernel module already exists for
  Linux Coda
• introduced into main kernel tree in 1997 (during
  2.1) and present in all 2.2 and 2.4 stable
  kernels.
• This is part of the Coda project at CMU, an open
  source fork of AFS2.
• Very similar architecture to AFS
• Kernel module and client side cache daemon
  (Venus)
• Kerberos based
• Already used parasitically by other Linux
  projects
• eg AVFS maps files to virtual filesystems (eg cd
  into a tar file)
• Coda kernel module / Venus also available for
  BSD and Windows 98/NT upwards.

Andrew McNab - Manchester HEP - 5 March 2002
6
Implementation with Coda

• Coda kernel module talks to client cache daemon
  by exchanging messages via /dev/cfs0
• Since we already have the kernel module, we just
  need to write a Venus-like daemon SlashGrid
  (/grid)
• Coda implementation allows efficient streaming
• open(), close(), stat() handled by calls to
  Venus/SlashGrid daemon
• coda_open call returns the inode of the cached
  copy to the kernel
• subsequent read() and write() operations handled
  by kernel itself, without daemon being involved.
• So streaming a local copy is just as fast as
  reading/writing a normal disk file.
• Since SlashGrid called for open()s etc, can
  enforce DN based access control at that point.

Andrew McNab - Manchester HEP - 5 March 2002
7
System calls with SlashGrid
Standard Unix
User process
User process
ordinary directory
/grid/...
SlashGrid
open() read() stat()
read() write()
open() stat()
/dev/cfs0
/var/spool/slashgrid/fcache
kernel
a real (ext2) disk
Andrew McNab - Manchester HEP - 5 March 2002
8
Remote file access?

• Another idea that has been around a while
  AFS-like system using Grid protocols.
• All the usual advantages of a global filesystem
• Makes a lot of the tedious management of
  parameter files needed by jobs just another
  operating system service.
• Very useful for interactive users they just see
  the Grid as one big file system.
• Makes all applications (even ls) Grid-enabled
  immediately.
• Already using URLs to refer to remote files, so
  easy to find an appropriate mapping into a
  filesystem space.
• So we want to design a system that can be
  generalised to remote file access too.

Andrew McNab - Manchester HEP - 5 March 2002
9
ACL format

• Need to specify permissions in some way.
• Commonly used compromise between granularity and
  simplicity is the per-directory ACL (cf AFS)
• Prototype uses the same format as the GridSite
  website management system (used for WP6 and
  GridPP websites)
• admin can modify ACL
• write can write/create files
• list can get a directory listing
• read can read a named file
• ACL consists of lines ltlevelgt ltDN/groupgt
• Currently only implement ltDNgt but in future can
  add VO groups, CAS authorisation symbols etc
  (when dust settles?)

Andrew McNab - Manchester HEP - 5 March 2002
10
ACL implementation

• Each directory has, or appears to have, a
  read-only file .grid-acl specifying ACL.
• Can easily be transferred via existing protocols
• eg if cache daemon fetches a file from a remote
  gsi-ftp server, can fetch the .grid-acl from the
  same directory without modifying gsi_wuftpd or
  GridFTP protocol.
• Modification of ACL done by accessing virtual
  files - these operations are trapped by
  SlashGrid and ACL updated
• cf. Codas .CONTROL mechanism
• eg remove file .grid-acl-write-url-encoded-DN
  to change the DNs permission level to write
• Provide grid-acl command line tool to hide this
  from users

Andrew McNab - Manchester HEP - 5 March 2002
11
XML Grid ACL?

• Current SlashGrid prototype uses same ACL format
  as GridSite web managament system
• text file with lines of form LEVEL DN
• where LEVEL is admin, write, list, read, none
• This could be extended to also recognise VO
  groups, CAS symbols etc as well as DNs.
• However, something in XML format would be more
  easily extensible

ltgridaclgt ltpersongt ltdngt/OGrid/OUKHEP/OUhep.man.
ac.uk/DNAndrew McNablt/dngt ltlevelgtadminlt/levelgt lt/
persongt lt/gridaclgt
Andrew McNab - Manchester HEP - 5 March 2002
12
Plugin framework

• Avoid making a monolithic system since
• Lots of interesting filesystems possible anon
  ftp, http, https, gsi-ftp, rfio, ldap, SQL
  databases (cf. Oracle 8i)
• Lots of uncertainty about which caching
  strategies to use.
• Some people will want some but not all of this on
  their systems.
• Have /etc/slashgrid.conf that specifies mount
  points and then which loadable module handles
  which part of the file system (cf. /etc/fstab)
• At start time, load dynamic modules which all
  export a common API.
• SlashGrid daemon hands each request to the right
  plugin
• user stat() gt coda_getattr gt PluginStat() gt
  plugin stat()

Andrew McNab - Manchester HEP - 5 March 2002
13
Current status

• Have implemented SlashGrid daemon and one plugin
  to provide local file storage with ACLs
  (certfs.so)
• SlashGrid obtains DN of a UID from
  /tmp/x509up_uUID
• so you do grid-proxy-init to get started
• stat / read / creat / mkdir / write / remove /
  rename / chmod system calls working for files and
  directories
• Can already do normal shell commands (ls etc),
  edit files with emacs, even copy the SlashGrid
  and certfs sources into the filesystem and build
  them with make and gcc.
• Can modify ACLs with grid-acl tool
• eg grid-acl admin /OGrid/CNJ.Smith

Andrew McNab - Manchester HEP - 5 March 2002
14
Future work

• Finish certfs
• Implement XML ACLs.
• Implement an example remote IO plugin
• probably read-only, anonymous http, since
  simplest
• Encourage other people to write plugins
• documentation for Plugin API exists
• Investigate specialised filesystems for dynamic
  accounts, automated cleanup, extra logging /
  auditing, ...
• Look at porting to other OSs
• Coda kernel module exists for BSD and Windows
  already
• The Linux Coda module was only 4000 lines of C...

Andrew McNab - Manchester HEP - 5 March 2002
15
Conclusion

• Have implemented a read/write filesystem for
  Linux, based on Grid DNs rather than Unix UIDs.
• Have done this in an extendable way using plugins
  for different filesystem types.
• Should also be straightforward to write a plugin
  for your favourite remote file access protocol.
• System is efficient for streaming local copies of
  files
• But can still accommodate many different
  strategies for fetching, caching and streaming
  files from remote servers.
• (Thanks to Anders, Cal and Fabio of Integration
  Team for useful discussions about all these
  issues.)
• http//www.gridpp.ac.uk/slashgrid/ has source,
  notes etc

Andrew McNab - Manchester HEP - 5 March 2002