Condor BirdBath SOAP Interface to Condor

1
Condor BirdBathSOAP Interface to Condor

• Charaka Goonatilake
• Department of Computer Science
• University College London
• c.goonatilake_at_cs.ucl.ac.uk

2
Background
3
Motivations

• Adding Web Service interfaces to Condor daemons
• Facilitate the development of third-party
  applications capable of interacting with Condor
  (remotely).
• E.g. build higher-level application specific
  scheduler that submits jobs to multiple Condor
  pools based on application semantics
• These can be built using a wide range of
  languages/SOAP packages
• BirdBath has been tested on
• Java (Apache Axis, XSUL)
• Python (ZSI)
• C (.Net)
• C/C (gSOAP)
• Condor accessible from platforms where its
  command-line tools are not supported/installed

4
The (Hypothetical) Big Picture
Site D
Condor-G
Globus/gridSAM
Client
SOAP
SOAP
flocking
Site A
Site B
Site C
5
Condor SOAP Interfaces

• Now available in Condor 6.7.5

• Obtain resource ClassAds
• Obtain all daemon ClassAds

• Begin/commit/abort transaction
• Submit Job
• Obtain Job ClassAds
• Get/Send file
• Removed/hold/release job

Execution machine
Execution machine
Submission machine(s)
Central manager
Schedd
Collector
Negotiator
Execution machine
Execution machine
6
Clients for BirdBath
Custom Client holdJob()
Submission machine(s)
SOAP library
Schedd

• WSDLs available that describe each daemon
  interface
• SOAP libraries (stub code) can be generated
  automatically from WSDLs

7
Example Query Collector

• Obtain information (ClassAds) about all resources
  that have over 512MB RAM
• collectorLocator new CondorCollectorLocator()
• collector locator.getcondorCollector(new
  URL(http//kotturoti.cs.ucl.ac.uk9618)
• classAds collector.queryStartdAds(Memorygt512)

8
Example Job Submission

• Create New Transaction
• Create New JobId
• Send Files (input files, binaries)
• Create Job Description
• Submit Job
• Commit Transaction
• Transaction based process
• Uses two phase commit to promote reliability and
  robustness
• Files transferred in chunks Base64 encoded SOAP
  attachments
• DAGMan jobs supported

9
OMII Collaboration gridSAM plugin

• Job Submission Monitoring Web Service that
  submits to range of resource managers (e.g.
  Condor, SGE, Globus)
• Uses JSDL (Job Submission Description Language)
  for standardised job description

10
gridSAM plugin

• Existing Condor plugin uses shell-based
  submission via command-line tools and monitoring
  of log files

Client
JSDL
SOAP
Middle-tier
GridSAM
Condor
SGE
Fork

11
Future Work

• Bundle plug-in into gridSAM
• Also provide separate JSDL to ClassAd conversion
  library
• Extensions to other daemons (what functionality
  to expose?)
• For example, allow checkpoint data to be
  retrieved through the shadow in order to migrate
  jobs across pools
• Roll out daemons on e-minerals mini-grid