GAT Einf

1
GAT
Alexander Beck-Ratzka
OFFIS, 23.06.06
2
GAT Agenda

• Introduction
• Why a new Grid-API?
• What GAT offers, what not?
• Examples
• Architecture
• The single API groups
• File (-stream) management, logical file
  management,
• AdvertService management,
• Job management
• Adaptors
• Conclusion

3
Why a new Grid-API?
GAT as a uniform API for the access to
heterogeneous grid technologies / grid
middleware.
GAT is only a framework the real grid
operationen must be done by adaptors. GAT offers
the possibility for accessing adpaptors.
New grid technologies only needs to be coupled to
GAT via adaptors -gt No changes in the source sode
necessary, even not If using new grid
technologies.
GAT enables a simple access to grid technologies.
GAT uses the grid middleware, which is available
at the moment, where is should be used, and this
by only typing a single statement for
GAT_JobSubmit or GATFileCopy...
4
File copy CoG/RFT
package org.globus.ogsa.gui import java.io.BufferedReader import java.io.File import java.io.FileReader import java.net.URL import java.util.Date import java.util.Vector import javax.xml.rpc.Stub import org.apache.axis.message.MessageElement import org.apache.axis.utils.XMLUtils import org.globus. import org.gridforum.ogsi. import org.gridforum.ogsi.holders.TerminationTimeTypeHolder import org.w3c.dom.Document import org.w3c.dom.Element public class RFTClient public static void copy (String source_url, String target_url) try File requestFile new File (source_url) BufferedReader reader null try reader new BufferedReader (new FileReader (requestFile)) catch (java.io.FileNotFoundException fnfe) Vector requestData new Vector () requestData.add (target_url) TransferType transfers1 new TransferTypetransferCount RFTOptionsType multirftOptions new RFTOptionsType () multirftOptions.setBinary (Boolean.valueOf ( (String)requestData.elementAt (0)).booleanValue ()) multirftOptions.setBlockSize (Integer.valueOf ( (String)requestData.elementAt (1)).intValue ()) multirftOptions.setTcpBufferSize (Integer.valueOf ( (String)requestData.elementAt (2)).intValue ()) multirftOptions.setNotpt (Boolean.valueOf ( (String)requestData.elementAt (3)).booleanValue ()) multirftOptions.setParallelStreams (Integer.valueOf ( (String)requestData.elementAt (4)).intValue ()) multirftOptions.setDcau(Boolean.valueOf( (String)requestData.elementAt (5)).booleanValue ()) int i 7 for (int j 0 j lt transfers1.length j) transfers1j new TransferType () transfers1j.setTransferId (j) transfers1j.setSourceUrl ((String)requestData.elementAt (i)) transfers1j.setDestinationUrl ((String)requestData.elementAt (i)) transfers1j.setRftOptions (multirftOptions) TransferRequestType transferRequest new TransferRequestType () transferRequest.setTransferArray (transfers1) int concurrency Integer.valueOf ((String)requestData.elementAt(6)).intValue() if (concurrency gt transfers1.length) System.out.println ("Concurrency should be less than the number" "of transfers in the request") System.exit (0) transferRequest.setConcurrency (concurrency) TransferRequestElement requestElement new TransferRequestElement () requestElement.setTransferRequest (transferRequest) ExtensibilityType extension new ExtensibilityType () extension AnyHelper.getExtensibility (requestElement) OGSIServiceGridLocator factoryService new OGSIServiceGridLocator () Factory factory factoryService.getFactoryPort (new URL (source_url)) GridServiceFactory gridFactory new GridServiceFactory (factory) LocatorType locator gridFactory.createService (extension) System.out.println ("Created an instance of Multi-RFT") MultiFileRFTDefinitionServiceGridLocator loc new MultiFileRFTDefinitionServiceGridLocator() RFTPortType rftPort loc.getMultiFileRFTDefinitionPort (locator) ((Stub)rftPort)._setProperty (Constants.AUTHORIZATION, NoAuthorization.getInstance()) ((Stub)rftPort)._setProperty (GSIConstants.GSI_MODE, GSIConstants.GSI_MODE_FULL_DELEG) ((Stub)rftPort)._setProperty (Constants.GSI_SEC_CONV, Constants.SIGNATURE) ((Stub)rftPort)._setProperty (Constants.GRIM_POLICY_HANDLER, new IgnoreProxyPolicyHandler ()) int requestid rftPort.start () System.out.println ("Request id " requestid) catch (Exception e) System.err.println (MessageUtils.toString (e))
5
What GAT offers? (1)
GAT is not a new grid middleware.

• GAT enables an easy to different grid
  infrastructures.
• GRAM
• Condor
• Unicore
• GridFTP
• ...

GAT is an OpenSource project.
6
What GAT offers? (2)

• Applications call the GAT-API for grid
  operations.
• Applications hav to be linked against the GAT.

• Applications are independend of the underlying
  grid infrastructure.
• GAT Engine loads available adaptors at runtime
• During a calls of the GAT-API the GAT engine
  decides, which adaptor performs the grid
  operation.
• In error case during a grid operation, another
  adaptor will be accessed.

• default adaptors offer local operations
• grid applications can be compiled, linked and
  tested without available grid services.
• The same application can run in a complete grid
  environment without a new build.

7
What GAT offers? (3)
The GAT-API doesnt change. Changes e.g. in
Globus job submit are included in the GAT Globus
adaptor.
GAT offers reliability. If one grid service is
not available, another grid service is used.
GAT is much eaysier to install as Globus.
GAT offers grid with a minimum effort for the end
user.
8
What GAT not offers?
GAT doesnt replace any function of a grid
middleware.
Without an adaptor for a grid middleware GAT is
useless for this middleware.
GAT has no resource broker.
9
Examples (Java-GAT)
File operations
Job submit
10
File copy Java-GAT
import org.gridlab.gat. import
org.gridlab.gat.io.File public class RemoteCopy
public static void main(String args) throws
Exception GATContext context new
GATContext() URI src new URI(args0) URI
dest new URI(args1) File file
GAT.createFile(context, src) // create file
object file.copy(dest) // and copy
it GAT.end()
11
File Write (Java-GAT)
import java.io.PrintWriter import
org.gridlab.gat.GAT. import org.gridlab.gat.io.F
ileOutputStream public class FileStreamSimple
public static void main(String args)
GATContext context new GATContext() URI
src null FileOutputStream stream null
PrintWriter p src new URI(args0)
stream GAT.createFileOutputStream(context,
null, src) p new PrintWriter(stream)
String toBeStreamed "hello world\n"
p.println(toBeStreamed) p.close()
GAT.end()
12
Job Submit (Java-GAT)
public class SubmitJobToHost public static
void main(String args) GATContext
context new GATContext() Preferences
prefs new Preferences() prefs.put("ResourceBro
ker.adaptor.name", "globus")
prefs.put("ResourceBroker.jobmanager", "sge")
URI exe null File outFile
null File errFile null File
pre1 null File pre1Dest null
try exe new URI("file////bin/ho
stname") outFile
GAT.createFile(context, prefs, new URI(
"any//fs0.das2.cs.vu.nl/out"))
errFile GAT.createFile(context, prefs, new
URI( "any//fs0.das2.cs.vu.nl/err"
)) pre1 GAT.createFile(context,
prefs, new URI(
"any//fs0.das2.cs.vu.nl//bin/echo"))
pre1Dest GAT.createFile(context, prefs, new
URI( "any//fs0.das2.cs.vu.nl//hom
e/rob/my_temp_file")) catch (Exception
e) System.err.println("error "
e) System.exit(1)

SoftwareDescription sd new
SoftwareDescription() sd.setLocation(exe)
sd.setStdout(outFile)
sd.setStderr(errFile) sd.addPreStagedFile
(pre1, pre1Dest) Hashtable
hardwareAttributes new Hashtable()
hardwareAttributes.put("machine.node",
"fs0.das2.cs.vu.nl") ResourceDescription
rd new HardwareResourceDescription(
hardwareAttributes) JobDescription jd
null ResourceBroker broker null
try jd new
JobDescription(sd, rd) broker
GAT.createResourceBroker(context, prefs)
catch (Exception e)
System.err.println("Could not create Job
description " e) System.exit(1)
Job job null try
job broker.submitJob(jd)
catch (Exception e)
System.err.println("submission failed " e)
e.printStackTrace()
System.exit(1)
13
GAT architecture

• API thin layer only a frame.
• Adaptor implements grid functionality as
  ordered by the API.
• Engine mediates betwen API u. adaptor
• the adaptors are called at runtime
• error tracing and Fallback (default local
  adaptors)
• CPI for Implementation of single adaptors.

14
GAT architecture
User Space
Grid Space
15
Implementation (Engine)
C version fully implemented C wrapper fully
implemented Python wrapper fully implemented JAVA
version fully implemented
16
Implementation (Adaptors) C-GAT

• Globus
• gram, gridftp, RLS, gsiscp, gsissh

• Unicore
• Job Submit, Job Monitoring

DRMAA (Distributed Resource Management
Application API)
SGE (Sun Grid Engine)
PBS (Portable Batch System)
17
Implementation (Adaptors) Java-GAT

• Globus
• With Java CoG Kit Globus 3.y complete, GTK 4 up
  to now without WebServices.

• Unicore
• In work

SGE
PBS
18
Implementation (Adaptors) Java-GAT and C-GAT

• Local adpaptors
• ssh, scp, ftp, sftp,
• File-Adapter (cp, mv, read, write, etc...)
• Job-Adapter fork, exec, auch über ssh...

19
File.copy schematic run through
20
GAT API overview
File operations
(Monitoring and events)
Resources, jobs

(Information exchange)
(Utility classes error handling, security,
preferences)
21
API sub systems
File sub system GATFile GATFilestream GATLogica
lFile GATEndpoint GATPipeListener GATPipe Monito
ring and event sub system GATRequestListener GATRe
questNotifier GATAction GATMetricListener GATMet
ric GATMetricEvent Information exchange sub
system GATAdvertisable GATAdvertService Resource
management sub system GATSoftwareDescription GATRe
sourceDescription GATResoure GATJobDescription GA
TResourceBroker GATReservation GATJob Utility sub
system GATSelf GATContext GATSecurityContext GA
TStatus GATPreferences URL,Time, ...
22
File sub systemGATFile class
GATObject
GATFile
Create Destroy Copy Move Delete IsReadable IsWrita
ble GetLength LastWriteTime GetLocation
23
File sub systemGATFileStream class
GATObject
GATFileStream
Create Destroy Read Write Seek
24
File sub systemGATLogicalFile class
GATObject
GATLogicalFile
Create Destroy GetFiles GetLocations Remove Remove
File AddFile AddLocation Replicate
25
Advert package
Goal Advertisement of important informations at
a central place, e.g. job finished, job rc...
Realized via an SQL adaptor in C-GAT.
26
Advert package
GATObject
GATAdvertService
Add Delete AddMetadata GetMetadata GetAdertisable
Find SetPWD GetPWD
27
Job managementclasses
GATResourceBroker
ReserveResource FindResources SubmitJob
28
Job managementavailable job status
GATJobState value meaning
GATJobState Unknown Status not retrievable
GATJobState Initial At initialization
GATJobState Scheduled e.g. queued
GATJobState Running Job is executing
GATJobState Stopped Job read or stopped
29
Job managementstructures
GATHardwareResourceDescription
Machine type Memory CPU-Time Nodes needed
30
Job managementexample (PBS adaptor)
GATHardwareResourceDescription
machine.queue destination (-q) jobname
jobname (-N) yeo join (-j) memory.size mem
(-l) file.size file (-l) cpu.walltime
walltime (-l) cpu.nodes nodes (-l)
31
Job managementexample (PBS adaptor)
GATSoftwareResourceDescription
Stdin stdin (exec lt input) Stdout
stdout (-o) Stderr stderr (-e) Executable
executable Arguments arglist
32
Job managementexample (PBS adaptor)
Realization in QSUB script
machine.queue new_at_url.org jobname
TestJob yeo eo memory.size 1G file.size
2G cpu.walltime 120000 cpu.nodes 8
stdin input stdout out.testjob stderr
err.testjob executable /bin/prog arglist0
arg1 arglist1 arg2 arglist2 lastarg
33
Job managementlacks
Problem Defintions in HardwareResourceDescriptio
n quite arbitrarily.
Solution Orientation at the JSDL standard.
34
Job managementlacks
Problem To less JOB status.
Solution Orientation at DRMAA.
35
GAT future
C-GAT is now in maintenance mode
Java-GAT is still further developed
SAGA (Simple API for Grid Applications) is the
upcoming GGF standard. First SAGA engine with GAT
wrapper and GTK4 adaptor in autumn 2006
36
Usage exampleProC MPA Garching
37
SAGA
To be integrated GAT, Java-CoG, DRMAA,
RealityGrid, JSDL, GridRPC, OSGA-BES, GridCPR,
gLite, HDF5
Developed by GAT, RealityGrid UK Science, OMII
Grid UK Science, CCT Louisana, VU Netherlands,
NAREGI Japan, Globus/CoG, GGF DRMAA, GGF GridRPC
Important Please remarks needs on adaptors!
38
GAT users
C-GAT
Java-GAT
MPA in Garching ProC-Workflows on the Grid LSU in
Baton Rouge Chemistry project AMOLF NL Fourier
Transform Mass Spectrometry (FTMS) Analyse.
FTMS data are delivered to the Grid with JavaGAT
(ssh, sftp, gridftp). Multimedian Project
NL Start of parallel Jobs ZIB Berlin INRIA
Frankreich
SuperScalar (Univ. of Barcelona, ESP)
SURA-Grid (63 partners, US) SCOOP project (LSU
9 partners, US) UCOMS project (LSU 4
partners, US) Cactus (LSU, US) Clusterix
Project (PSNC, Univ. of Krakow, PL) Amolf
(Vl-e, NL) Frank Seinstra (UvA, NL) Triana
group (Cardiff, UK)
39
Links for GAT
GAT overall http//www.gridlab.org/WorkPackages/
wp-1/
GAT CVS cvs.gridlab.org cvsroot
pserverreadonly_at_cvs.gridlab.org/cvs/gridlab Pass
word anon GAT sources wp-1/Codes GAT
documents wp-1/Documents Only GATEngine
wp-1/Codes/GATEngine
Download tarball http//www.gridlab.org/WorkPack
ages/wp-1/gatreleases.html http//www.gridlab.org/
WorkPackages/wp-1/adaptorreleases.html
German GAT Mailing-Liste GAT_at_d-grid.de
(www.listserv.dfn.de)