The functionality of a Replica Registration Service

1
The functionality of a Replica Registration
Service
Attendees Michael Haddox-Schatz, JLAB Ann
Chervenak, USC/ISI Jean-Philippe Baud, CERN Timur
Perelmutov, Fermi Don Petravick, Fermi Doug
Olson, LBNL Alex Sim, LBNL Arie Shoshani, LBNL
Meeting Location LBNL Sept 18, 2003
2
Generalized Replica ManagementService

• Replicate LFN ? site
• Lookup LFNs find out which RLS to use
• Planning select PFNs / SURLs for accessing files
  
• Allocate space
• Move files robustly
• Register files

3
Simple Replica Management focus on three
services

• Dynamic Space Allocation
• Now space is pre-allocated
• Future request space reservation
• Multi-file replication
• Managing the entire multi-file request
• Use file transfer services
• Guarantee robustness
• Replica Registration
• Into one or more catalogs
• Can register into standard Replica Catalogs
• Can register into specialized experiment file
  catalogs

4
Replica ManagementArchitecture
Client
Replica Management Service
Multi-file Replication Service
Replica Registration Service
File Transfer Services
Replica/File catalog Services
5
Replica Managementfor STAR Experiment
Client
Replica Management Service
Replica Registration Service
GridFTP (bbftp)
STAR File Catalog
6
Replica Managementfor STAR Experiment
Client
DataMover
Replica Registration Service
HRM
HRM
GridFTP (bbftp)
STAR File Catalog
7
Generalizing the ConceptNeed to Standardize APIs
8
Three use cases

• Client generated a bunch of files
• Call a RepReg service to register to one or more
  RepCats
• Client want to replicate a bunch of files (or
  directory)
• Call RepReg with different modes
• Want to delete one or more files
• Should a RepReg service to cleanup catalogs?
• --------------------------------------------------
  -------------------------------------
• Question how the RepCats chosen?
• We propose to use the MDS for that
• Qusetion should RRS be a separate (external)
  service
• Yes, because it is too much to impose on
  underlying RepCats
• Yes, because it is a common functionality to all
  RepCats
• Yes, because it has value as a separate service

9
Replica ManagementArchitecture
Find RepCats (optional)
Client
Monitoring Discovery Service
Replica Management Service
Default RepCats
Multi-file Replication Service
Replica Registration Service
File Transfer Services
Replica/File catalog Services
10
Functionality ofReplica Registration

• Registration modes
• File-at-a-time registration
• Mode 1 register all the files that were
  replicated successfully, and report which failed.
• Mode 2 stop registration process if there is a
  failure in registration, report which failed.
• Global registration after the entire multi-file
  replication finished
• Mode 3 register only if all file replications
  are successful.
• Mode 4 register all files that were successfully
  replicated, and report which failed, if any.
• Failures are in terms of registering to a RepCat
• E.g. RepCat database is full or down
• Assume immutable files, LFN-PFN duplication is
  error
• Register to any catalogs
• Standard catalogs e.g. RLS
• Specialized catalogs e.g. experiments file
  catalogs
• Issue need a standard for RepCat interface
• In the meantime we need have a plug-in type
  service

11
Functionality ofReplica Registration (Contd)

• Queue registration requests
• Request token
• Multi-file reg capability
• Permit multiple registration modes
• File-a-time (as quickly as you can) mode 1 or
  mode 2
• Bulk (do it all or do none gt roll back) mode 3
  or mode 4
• Failure no-response, no space,
  file-already-registered
• Permission security GSI proxy user name for
  now
• Access control specify r/w for catalog (limited
  today, ACLs later)
• Abort and Roll back registration
• Commit registration
• Global reg request (commit implied)
• Robustness
• Recover from transient failures
• Retry per request with time limit (in seconds)

12
Functionality ofReplica Registration (Contd)

• Register to multiple catalogs simultaneously
• Ability to specify which catalogs to register to
• Refer to RepCats by URLs
• Manage multiple registration requests for
  multiple clients
• Implementation dependent

13
Functionality ofReplica Registration (Contd)

• Asynchronous service
• Provide dynamic status on registration
• Success, failed, pending per file per RepCat
• Failed already-in-catalog, ill-formed (keep
  RepCat error string), not-authorized
• RepCat not responding per RepCat
• RepCat database full per RepCat
• Waiting for commit per request
• Request timed-out per request
• Full Success / partial success (but completed)
  per request
• Provide statistics and failure information
• Summary per request
• (History activity information)

14
Un-register

• Problem want to delete a file and un-register it
  from all RepCats that know about it.
• 1) Need to find out where file is registered
  or2) broadcast to all RepCat in the VO
• Prefer 2) because we avoid keeping a state of
  where files are registered
• Who issues the notification to un-register a
  file?
• SRMs need to have permission
• Maybe RepCats should check with SRM before
  removal
• Should this be a service of RRS?
• Probably, but not in first version