Towards Grid Interoperability

1
Towards Grid Interoperability

• Richard Boardman, Stephen Crouch, Hugo Mills,
  Steven Newhouse, Juri Papayand the OMII-UK
  TeamAll Hands Meeting11/09/07

2
Contents

• Introduction background
• Interoperability and standards
• Job Submission
• Implementations
• Build and test
• Job brokering using job submission standards

3
Introduction What is the Grid?

• The grid many definitions!
• Grid computing offers a model for solving
  massive computational problems by making use of
  the unused CPU cycles of large numbers of
  disparate, often desktop, computers treated as a
  virtual cluster embedded in a distributed
  telecommunications infrastructure Wikipedia
• An infrastructure for coordinated resource
  sharing and problem solving in dynamic,
  multiinstitutional virtual organizations
  Foster, Kesselman, Tuecke.
• A service for sharing computer power and data
  storage capacity over the Internet. CERN
  (European Organisation for Nuclear Research)
• Differing perceptions of the Grid
• Particle physics community massive, loosely
  coupled, distributed computing environment with
  computing capability, bandwidth and storage.
• Bio-informaticians global virtual federated
  database of experimental data, research papers
  and laboratory records.
• Suggested common view
• The Grid provides secure virtualisation of
  resources and enables collaboration and
  establishment of virtual organisations.

4
Background

• Many Grid infrastructures have been developed,
  but, traditionally, with little interoperability
• Policies governing access/use of distributed
  resources
• Lack of adherence to emerging common standards
• Interoperability offers huge benefits for
  categories of users within user community
• e-Infrastructure providers easier
  deployment/management of software distributions
• e-Science users freedom to choose services
  deployed in different Grids based on
  functionality, not deployed on a particular Grid
• e-Science application developers portability of
  apps across multiple Grids to increase uptake

5
Standards the key to Interoperability

• Adoption of common standards strongly supported
  and implemented by OMII-Europe and GIN (Grid
  Interoperability Now) for
• Job Submission, Accounting, Virtual Organisation
  Management
• Standards from OGF, OASIS, W3C, DMTF
• Across platforms EGEE, Globus, UNICORE others
• Will focus on Job Submission (OGSA-BES, JSDL)
  standards

6
Emerging Job Submission Standards

• Two key standards for two key elements
• The Basic Execution Service interface (OGSA-BES)
• Simplified version of OGSA-EMS (Execution
  Management Service)
• Handles basic job lifecycle management
• Defines simple (but extendable) job state model
• Pending, running, cancelled, failed or finished
• The Job Submission Description Language (JSDL)
• Specify job executable, data staging and resource
  requirements

7
Summary of OGSA-BES
BES-Management Port-type BES-Management Port-type
StopAcceptingNewActivities Request that the BES stop accepting new activities
StartAcceptingNewActivities Request that the BES start accepting new activities
BES-Factory Port-type BES-Factory Port-type
CreateActivity Request the creation of a new activity
GetActivityStatuses Request the status of a set of activities
TerminateActivities Request that a set of activities be terminated
GetActivityDocuments Request the JSDL documents for a set of activities
GetFactoryAttributesDocument Request XML document containing BES properties
BES-Activity Port-type (optional) BES-Activity Port-type (optional)
GetStatus Request the status of an activity
Terminate Request that an activity be terminated
GetDocument Request the JSDL document for an activity
GetActivityAttributesDocument Request XML document containing activity properties
8
Interface/Interaction Standards not Enough

• BES JSDL alone not enough for real
  interoperability
• JSDL is extensible
• Differing security models across Grid
  infrastructures
• HPC-Profile proposes Grid interoperability
  through
• Restricted OGF Job Submission Description
  Language (JSDL)
• OGF OGSA Basic Execution Service (BES)
• WS-I Basic Profile
• In addition, an agreed security framework between
  participants
• Via HTTPS transport (server offers certificate)
  username/password (client) for user
  authentication
• OMII-UK has implemented the HPC-Profile within
• GridSAM funded by OMII-UK, first to adopt BES
• CROWN with OMII-Europe, in collaboration with
  Beihang University, China

9
Build and Test

• To gain confidence of compliance with
  interoperability standard, test against other
  standards-compliant infrastructures
• Test multiple BES/JSDL clients against multiple
  service endpoints
• Mechanistic process automation advantageous
• ETICS (CERN) provides test automation framework
• e-Infrastructure for Testing, Integration and
  Configuration of S/W
• Leverages Metronome (formerly NMI Build Test)
  across Condor cluster
• Controls management of software builds and
  testing
• Create project configurations, maintain
  historical records
• ETICS deployed to enable automated compliance
  testing

10
NMI Build and Test
11
ETICS
12
Build and Test

• BES method sequence for testing (core
  functionality)
• Get attributes document
• Create job
• Query job status
• Show job output
• Get jobs JSDL document
• Terminate job
• Scenarios implemented in ETICS to test
  interoperability
• Testing across HPC-Profile endpoints (with simple
  config)
• Test OMII-UK client component against service

13
Job Brokering using the CROWN Scheduler

• CROWN Grid developed by Beihang University
• With OMII-Europe as part of Component Exchange
  activity with OMII China
• Identified CROWN Meta Scheduler
• Integrated into OMII release (interoperation)
• Usage of OMII-UK Grimoires service registry
• Coordinated implementation of BES interface to
  the Scheduler

14
CROWN BES Scheduler
BES
Registry
submit
submit
monitor
BESClient
monitor
CROWN Scheduler
BES
submit
monitor
Supported Platform

• 2 jobs, different requirements
• Delegate BES resource selection by submitting
  both jobs to Scheduler
• Monitor each until completion

15
CROWN BES Scheduler Demo
16
Conclusions

• First phase of projects dealt with
  infrastructures, test beds and application
  software
• Led to greater understanding of key issues
• Next phase concentrates on interoperability and
  providing solutions-based approaches
• Interoperability offers key benefits to the
  community
• Ease of management
• Choice
• Simplicity of implementation
• Have illustrated a process of standards adoption,
  build and compliance testing, and usage within a
  scheduling application