GridSAM

1
GridSAM

• A Standards Based Approach to Job Submission
• A. Stephen MCGough
• Imperial College London

2
Common Grid Systems

• There are many Grid Systems.
• Though they are not compatible.
• Globus
• Condor
• gLite
• Grid Engine
• LSF
• PBS
• LoadLeveler
• Unicore

• Different job description languages
• Different submission interfaces

3
If all DRM systems supported the same interface

• If we had
• One interface definition for job submission
• One job description language
• Then life would be easier!
• Were getting there
• JSDL is a proposed standard job submission
  description language
• OGSA-BES is a proposed standard basic execution
  service interface
• One day hopefully everyone will support this
• Till then

4
GridSAM Job Submission and Monitoring Web
Service
5
GridSAM OverviewGrid Job Submission and
Monitoring Service

• What is GridSAM?
• A Job Submission and Monitoring Web Service
• Funded by the Open Middleware Infrastructure
  Institute (OMII) managed programme
• V2.0.0 Available as part of the OMII 2.x release
• Open source (BSD)
• One of the first system to support the GGF Job
  Submission Description Language (JSDL)

6
Whats not?

• GridSAM is not
• a scheduling service
• Thats the role of the underlying launching
  mechanism
• Thats the role of a super-scheduler that brokers
  jobs to a set of GridSAM services
• a provisioning service
• GridSAM runs whats been told to run
• GridSAM does not resolve software dependencies
  and resource requirements

7
GridSAM OverviewGrid Job Submission and
Monitoring Service

• What is GridSAM to the resource owners?
• A Web Service to expose heterogeneous execution
  resources uniformly
• Single machine through Forking or SSH
• Condor Pool
• Grid Engine 6 through DRMAA
• Globus 2.4.3 exposed resources
• OR use our plug-in API to implement

8
GridSAM OverviewGrid Job Submission and
Monitoring Service

• What is GridSAM to end-users?
• A set of end-user tools and client-side APIs to
  interact with a GridSAM web service
• Submit and Start Jobs
• Monitor Jobs
• Terminate Jobs
• File transfer
• Client-side submission scripting
• Client-side Java API

9
Integration with OMII Distribution
10
GridSAM Implementation

• Virtual File System API (Apache VFS)
• FTP / GSIFTP / HTTP / WEBDAV / SFTP
• POSIX Shell API
• Fork / SSH
• Event dispatches (OpenSymphony Quartz)
• Job Persistence (Hibernate - JDBC databases)
• Runtime Monitoring and Control (Java Management
  Extension)

11
GridSAM Architecture

• A staged event-driven architecture
• Submission pipeline is constructed as a network
  of stages connected by event queues
• Each stage perform specific action upon incoming
  events

M. Welsh and D. Culler and E. Brewer. Seda An
architecture for well-connected scalable internet
services. In Eighteenth Symposium on Operating
Systems Principles (SOSP-18), October 2001.
12
Deployment Scenario Forking
Local FS
HTTP WS-Sec./ HTTPS WS-Sec. / HTTPS mutual.
13
Deployment Scenario Secure Shell (SSH)
HTTP WS-Sec./ HTTPS WS-Sec. / HTTPS mutual.
SFTP - FS
14
Deployment Scenario Condor Pool
Condor command-line wrapper
Network FS
HTTP WS-Sec./ HTTPS WS-Sec. / HTTPS mutual.
15
Deployment Scenario Globus 2.4.3
16
Deployment Scenario Grid Engine 6
Network FS
17
Latest Features

• Available in v2.0.0 (released 1/7/06)
• MPI Application through GT2 plugin
• Simple non-standard JSDL extension
  ltmpiMPIApplication/gt that extends
  ltposixPOSIXApplication/gt with a
  ltmpiProcessorCount/gt element
• Authorisation based on JSDL structure
• Allow / deny submission based on a set of XPath
  rules and the identities of the submitter (e.g.
  distinguished name).
• Prototype Basic Execution Service (ogsa-bes)
  interface
• Demonstrated in the mini face-to-face in London
  last December
• Shown interoperability with the Uni. Of Virginia
  BES (.NET based) implementation.

18
Upcoming Features

• New DRMConnectors
• PBS, EGEE, LSF, CCS
• GridBS
• Using the Condor matchmaker with GridSAM
• With Oxford University
• Basic Execution Service
• Continue tracking the changes in the ogsa-bes
  specification
• Support dual submission WS-interfaces

19
GridSAM deployment

• UK NGS
• Supported by the Belfast e-Science Centre
• UCL
• Newcastle
• Reading University
• Imperial College London
• JSDL Portlet from DL

20
GridSAM and AHE

• The Application Hosting Environment (AHE) uses
  GridSAM for deploying their application wrappers
  to resources on the Grid.
• As part of the SPICE (Simulated Pore Interactive
  Computing Environment) demonstrator for
  SuperComputing05 GridSAM (along with the AHE)
  was used to deploy exeutables on resources both
  part of the UK National Grid Service and the US
  TeraGrid.
• This helped towards SPICE winning the HPC
  Analytics Challenge for demonstrating the use of
  innovative techniques in rigorous data analysis
  and high-end visualisation to solve a complex,
  real-world problem.

21
GridSAM and e-protein

• e-protein is performing protein annotation
• Using GridSAM for executing bioinformatics jobs
  on the Grid

Level 1
Web Portal
Web Interface
Sequence files
Control
3D- GENOMICS
Workflow
Java Control
Workflow
Program
Pipeline
Program
Job Description document
Level 2
SSH tunnel
Job Identifier
DRM
DRM
Level 3
Scheduler
Job
Job
Job
Job
Dasty Viewer
22
GridSAM and MicroArray

• Simple, robust access to up-to-date resources for
  microarray data storage and analysis
• Distributed availability of the portal, allowing
  access to large compute power facilities required
  for microarray analysis and storage

23
GridSAM and eMinerals
24
GridSAM at UCL

• Integrating GridSAM with BPEL
• Providing Condor WS DRMConnector

Client
JSDL
SOAP
Middle-tier
GridSAM
SGE
Fork
Condor
25
GridSAM and NeuroGrid

• The aim of the NeuroGrid consortium is to enhance
  collaboration between clinical, imaging and
  e-scientists to create a Grid-based network of
  neuroimaging centres and a neuroimaging tool-kit.
  Sharing data, experience and expertise will
  facilitate the archiving, curation, retrieval and
  analysis of imaging data from multiple sites and
  enable large-scale clinical studies.

26
GridSAM and HPCP Interop

• GridSAM was one of over 10 OGSA-BES (HPCP
  profile) services demonstrating interop
• Managed to interop with all but one other system
• Profile was restrictive
• Simple Security
• No data staging
• Taking part in the SC07 interop

27
Questions?

• Official Download
• http//www.omii.ac.uk
• Project Information and Documentation
• http//gridsam.sourceforge.net