Use of SRM File Streaming by Gateway

1
Use of SRM File Streaming by Gateway

• Alex Sim
• Arie Shoshani
• May 2008

2
File Streaming how it works
Small file request
user quota
user quota
Get file
user quota
user quota
Release file
3
File Streaming whats the advantages

• Can accommodate very large requests with a
  limited quota
• No waste of space use only space needed for
  files
• Reuse space as soon as files are transferred and
  released
• Share files that multiple users ask for
• Keep popular (hot) files in cache as long as
  space permits
• Can have smaller quotas to accommodate more users
• Length of lifetime can be longer, as long as
  files are released(can avoid cutting off request
  before they finish)
• With DML, transfer can be started right away,
  even if only someof the files are in cache
• Overlap transfer to cache (from archive or
  another site) with transfer to User

4
Scenario 1 Simple Scenario for User File Access

• Simple http or wget download from ESG Gateway
  site
• User goes to Gateway, selects files, requests
  files
• Users quota must be sufficient to hold all
  requested files (otherwise request refused)
• Gateway gets files into SRM disk from local or
  remote sites
• User finds out the status from the Gateway either
  by browser or email
• User downloads files either by clicking file
  links on the browser or by wget
• User clicks Gateway to release files that are
  downloaded (no way to enforce that)

LBNL/NERSC
ORNL
NCAR
Users machine
ESG Gateway
Users browser
SRM
SRM
Browser Or wget
SRM
http transfer
Disk Cache
Disk Cache
Disk Cache
Disk Cache
NERSC/HPSS
ORNL/HPSS
NCAR/MSS
5
Scenario 2 Download with DML from ESG - No File
Streaming

• User downloads DataMoverLite (DML)
• User goes to Gateway, select files, requests
  files
• Users quota must be sufficient to hold all
  requested files (otherwise request refused)
• Gateway submits the request to the SRM.
• SRM returns the request token to the user.
• User launches DML with request token.
• DML checks the status of the request and starts
  transferring files from Gateway to the users
  local disk(right away for files already in
  cache)
• DML downloads files and sends release to SRM
  through the Gateway, so space is not wasted
• After file transfers are completed, DML sends
  request completed to Gateway)

NCAR
Users machine
LBNL/NERSC
ORNL
request
ESG Gateway
Users browser
request token
release
DataMover Lite
SRM
SRM
SRM
Disk Cache
Disk Cache
HTTP/HTTPS transfer
Disk Cache
Disk Cache
NERSC/HPSS
ORNL/HPSS
MSS
6
Scenario 3 Download with DML from ESG - File
Streaming

• File Streaming
• When files fit user quota space DML starts
  downloading files (right away for file already in
  the cache)
• When files do not fit the user quota space, DML
  uses the file streaming feature (DML repeatedly
  sends Status to Gateway, and downloads available
  files )
• SRM brings files into Gateways SRM cache only as
  many files as it can fit into the users space
  quota.
• As DML downloads files and sends release to SRM
  through the gateway, more files are being brought
  in, and streamed to the client.

NCAR
Users machine
LBNL/NERSC
ORNL
request
ESG Gateway
Users browser
request token
Status
DataMover Lite
release
SRM
SRM
SRM
File transfer
Disk Cache
Disk Cache
Disk Cache
Disk Cache
NERSC/HPSS
ORNL/HPSS
MSS
7
File Streaming
Users machine
Browser
DataMover Lite
Users browser
Disk Cache
NCAR
Data Nodes
LBNL/NERSC
ORNL
ESG Gateway
SRM
SRM
SRM GridFTP FTP HTTP
SRM
Disk Cache
Disk Cache
Disk Cache
Disk Cache
NERSC/HPSS
ORNL/HPSS
MSS
8
Scenario 4 Download with DML from any data
source locations

• User downloads DataMoverLite
• User goes to Gateway, select files
• Gateway gets files information with source
  location (Gateway does not request any files)
• User launches DML with files info provided by the
  Gateway
• DML contacts Data Source with appropriate
  transfer protocol(security mechanism needs to be
  worked out for GSI access)
• When a file is available, DML downloads to its
  local disk
• DML releases files when done transferring.
• DML reports statistics to the Gateway

Gateway
Users machine
SRMs with MSS
(1) Find file(s), Get SURLs
ESG Gateway
Users browser
SRMs with Disk
(4) Release file(s)
(2) Request file(s) From NCARs MSS
HTTP
DataMover Lite
(2) Request file(s)
SRM
(4) Release file(s)
GridFTP
(3) transfer file(s)
Disk Cache
Disk Cache
FTP
(3) transfer file(s)
NCARs MSS
9
Status of BeStMan Java Interfaces

• Provide API to find out cache usage
• Done srmPing provides Total_space and Used_space
• Provide API for request status summary
• Done srmRequestSummary provides
  no_of_files(requested, in-cache, released,
  in-queue)
• Provide API to get all available Transfer URLs
  within a users quota
• Done srmRequestStatus provides that.
• Provide API to abort a request
• Done srmAbortRequest aborts request and releases
  all files
• Gateway needs to estimate total size of request
• in order to check if request fits in quota
• Accordingly, it advise user what to use
• Discuss should wget be used in streaming mode
  (or al-in-cache mode)?