Title: Grid Programming in
1- Grid Programming in
- Java Python
- Prepared by
- Gregor von Laszewski
- Argonne National Laboratory
- gregor_at_mcs.anl.gov
- Keith R. Jackson
- Lawrence Berkeley National Laboratory
- krjackson_at_lbl.gov
2Chapter Outline
- Introduction to Grids
- What is a Grid?
- What is the Globus Toolkit?
- What is a Commodity Grid Kit?
- Using and Programming Grids with the JavaTM and
Python CoG Kits - Secure access to remote resources
- Remote job submission and monitoring
- Distributed grid information management and
remote data access - Graphical component to access a Grid
- Conclusion for this part of the presentation
- Exercises
3Integration of High-end Resources
Supercomputers
Software catalogs
Sensor nets
Colleagues
Data archives
On-demand creation of powerful virtual computing
systems
4Grid Computing and Existing Technologies
- Commonalities between Grid computing and major
industrial thrusts - Business-to-business, peer-to-peer, application
service providers, storage service providers,
distributed computing, Internet computing - Differences between Grid computing and existing
technologies - Complicated requirements Run program X at site
Y subject to community policy P, providing access
to data at Z according to policy Q - High performance unique demands of advanced
high-performance systems
5What challenges do we have to solve?
- Authenticate once
- Specify simulation (code, resources, etc.)
- Locate resources
- Negotiate authorization, acceptable use, etc.
- Acquire resources
- Initiate computation
- Steer computation
- Access remote datasets
- Collaborate on results
- Account for usage
Domain 1
Domain 2
6An Example ApplicationAn Advanced Scientific
Instrument
Virtual Reality Cave
Advanced Photon Source
Scientist
Avatar
Supercomputer
Electronic Library and Databases
Computing Portal Clients
7The Globus Toolkit
- The Globus Toolkit provides a range of basic Grid
Services - Security, information, fault detection,
communication, resource management - These services are simple and orthogonal
- Independently use mix and match
- Programming model independence
- For each service there is generally a
well-defined API - Standards are used extensively
- E.g. LDAP, GSS-API, X.509
8Globus Approach
- A toolkit and collection of services addressing
key technical problems - Modular bag of services model
- Not a vertically integrated solution
- General infrastructure tools (aka middleware)
that can be applied to many application domains - Interdomain issues, rather than clustering
- Integration of intradomain solutions
- Distinction between local and global services
9Globus Hourglass
- Focus on architecture issues
- Propose set of core services as basic
infrastructure - Use to construct high-level, domain-specific
solutions - Design principles
- Keep participation cost low
- Enable local control
- Support adaptation
- IP hourglass model
A p p l i c a t i o n s
Diverse global services
Core Globus services
Local OS
10Layered Grid Architecture
Applica- tion
Application
Internet Protocol Architecture
Collective
Resource
Connectivity
Transport
Internet
Link
Fabric
11Production Grids Testbeds
GUSTO Testbed
The Alliance National Technology Grid
NASAs Information Power Grid
12Key Grid Services
- Resource Discovery
- List all of the solaris machines with at least 16
processors and 2 Gigs of memory - Resource Acquisition
- Submit my job to this machine
- Data Management/Movement
- Locate, or create, a replica of this data set
- Move this data set to this host
- Security
- Authentication
- Authorization
13Grid Security Infrastructure
- Standards based infrastructure that provides
- Single sign-on
- Authentication data integrity/confidentiality
- Delegation
- Based on
- X.509 Proxy certificates (draft ietf pkix
standard) - TLS
- GSS-API
14Delegation in GSI
Client
GSI Proxy
3) Submit Job
5) Retrieve Data Set
Compute Resource
Tertiary Storage
6) Run computation
15GridFTP
- Implements many of the optional RFCs for the ftp
protocol. - Authentication using the Grid Security
Infrastructure through the GSS-API - Restart markers
- Third-party transfers
- Adds some new extensions to support
high-performance transfers. - Parallel streams
- Striped servers
- TCP Buffer Tuning
16Definition CoG Kit
- A Commodity Grid (CoG) Kit defines and implements
a set of general components that map Grid
functionality into a commodity environment. - Web/CGI, Java, Python, CORBA, DCOM, ....
- CoG Kits help us build applications, Problem
Solving Environments, and Portals. - CoG Kits take the good things from framework
Grid
17To further elaborate
- Computational Grids
- Globus Infrastructure
- Commodity Technologies
- CORBA, Java, Python, Perl, Web technologies, ...
- Science Application
- Computing Portal
- Problem Solving Environments
- Workbenches
18Motivation Java Python CoG Kits
- Use and leverage existing technologies for Grid
programming - - The capabilities of the framework onto which
Grid Services are mapped can be exploited - Objects, Events, Exceptions, JNDITM, ...
- - Objects like jobs/tasks can be defined.
- - XML support is provided.
- - GUI's, ...., IDE's can be used (Forte, BOA
Constructor) - Maximize software flexibility, extensibility, and
reusability - Provide foundations for application developer
teams that are familiar to develop applications
in this framework - - Reduce development and maintenance cost
- Use as glue for many technologies
- Python is well suited to tying together many
different languages/technologies.
19What is the Java CoG Kit ?
- The Java CoG Kit provides a mapping between Java
and the Globus Toolkit. It extends the use of
Globus by enabling to access advanced Java
features such as events and objects for Grid
programming. - The Java CoG Kit is implemented in pure Java. It
speaks the Grid protocols. - It is not a wrapper of the C Globus Toolkit
- This allows integration within applets.
- Mostly client side support
20What is the Python CoG Kit?
- Similarly the Python CoG Kit provides a mapping
between Python and the Globus Toolkit. It extends
the use of Globus by enabling to access advanced
Python features such as events and objects for
Grid programming. - The Python CoG Kit is implemented as a series of
Python extension modules that wrap the Globus C
code. - Uses SWIG (http//www.swig.org) to help generate
the interfaces.
21Status Java CoG Kit
- Modified core Globus components (Protocols)
- Basic services are provided accessing
- Security
(GSI) - Remote job submission and monitoring (GRAM)
- Quality of service
(GARA) - Remote Data Access (GSIFTP)
- Information Service Access (MDS)
- Certificate store
(myProxy) - Current 100 client side components includes
- Reusable Grid GUI components
22Status Python CoG Kit
- Basic services are provided accessing
- Security
(security) - Remote job submission and monitoring
(gramClient) - Gass Server (gassServerEZ)
- Secure high-performance network IO (io)
- Protocol independent data transfers (gassCopy)
- High performance Grid FTP transfers (ftpClient)
- Support for building Grid FTP servers (ftpControl)
- Remote file IO (gassFile)
- Replica Catalog (replicaCatalog)
- Replica Management (replicaManagement)
- GSI enabled SOAP (GSISOAP)
23Python CoG Kit basics
- Everything is contained in the pyGlobus package.
- The low-level C-wrappers are all named modulec,
e.g., ftpClientc, gramClientc - Shouldnt need to access these directly, instead
use the Python wrappers (ftpClient, gramClient) - Exceptions are used to handle error conditions
- pyGlobus.util.GlobusException is the base class
for all pyGlobus exceptions. It extends the base
Python exceptions.Exception class - The Python wrappers classes manage the underlying
memory
24Building the Python CoG Kit
- Uses standard GNU tools (autoconf, make)
- Download the latest source from
http//www-itg.lbl.gov/Grid/projects/pyGlobus/ - Tar zxvf pyGlobus-VERSION.tar.gz
- cd pyGlobus mkdir build cd build
- ../configure
- Use with-flavorltGlobus flavor stringgt to
override the default flavor - Use with-pythonltpath to pythongt to override the
default python installation - make make install
- Installs into your site-packages directory (No
need to edit PYTHONPATH)
25Where do we go from here?
- On which computers can I perform my task?
- Use a Grid information service (MDS) to answer
this. - Use the LDAP browser as a good example of how to
develop a Grid-based Portal to a directory based
information service. - How can I execute a job on a remote machine?
- Use the job submission API or GUI
- How can I access data on a remote machine?
- Use Grid FTP.
- Use the protocol independent gassCopy module.
- Remote file IO using the gassFile module.
- Secure high-performance network IO.
26LDAP Browser and Editor
- User-friendly Windows Explorer-like interface to
LDAP directories - It allows browsing and modification of
information - It is written entirely in Java
- Uses JFC (SwingSet) for the GUI
- Uses JNDI class library for LDAP access
- Connects to LDAP v2 and v3 servers.
- Note v3 standard is defined as we speak
- Recognition
- More than 1600 commercial user licenses
- Many users in academia (license for free)
- Winner of the Best Student Application award in
the Novell Developers' Contest - Jars top 25
27Features
- Browsing, editing, searching
- Export
- Binary attributes, customizable attribute editors
- Object templates
- LDAP v3 aware, SSL support
- Drag and drop, copy-and-paste interface
- Named sessions
- Applet support
28Features
- Browsing, editing, searching
- Export
- Binary attributes, customizable attribute editors
- Object templates
- LDAP v3 aware, SSL support
- Drag and drop, copy-and-paste interface
- Named sessions
- Applet support
29Features
- Browsing, editing, searching
- Export
- Binary attributes, customizable attribute editors
- Object templates
- LDAP v3 aware, SSL support
- Drag and drop, copy-and-paste interface
- Named sessions
- Applet support
30Screenshot
31Tiny Information Query Program
- MDS mds new MDS ("mds.globus.org", 389,
"oGrid") - MDSresult result mds.search
- ("(objectclassGridComputeResource)
- (freenodes64))",
- "contact freenodes totalnodes")
- String contact result.get("contact")
- System.out.println(result.print())
- This can also be done with JNDI or Netscape SDK
- We have this layer for portability
32Remote Job Submission
- Access to remote resources is an elementary Grid
problem - Supports delegation to the remote resource
33Gram Submission Demo
Hello World 1 2 3 Bang
A test to stderr 3 2 1 Bang
34C Job Submission Example
callback_func(void user_arg, char job_contact,
int state, int errorcode)
globus_i_globusrun_gram_monitor_t monitor
monitor (globus_i_globusrun_gram_monitor_t )
user_arg globus_mutex_lock(monitor-gtmutex)
monitor-gtjob_state state
switch(state) case GLOBUS_GRAM_PROTOCOL_
JOB_STATE_PENDING globus_i_globusrun_gram_m
onitor_t monitor monitor
(globus_i_globusrun_gram_monitor_t ) user_arg
globus_mutex_lock(monitor-gtmutex)
monitor-gtjob_state state switch(state)
case GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED
if(monitor-gtverbose)
globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE
_FAILED\n") monitor-gtdone
GLOBUS_TRUE break case
GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE
if(monitor-gtverbose)
globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE
_DONE\n") monitor-gtdone
GLOBUS_TRUE break
globus_cond_signal(monitor-gtcond)
globus_mutex_unlock(monitor-gtmutex)
35C Job Submission Example (cont.)
globus_l_globusrun_gramrun(char request_string,
unsigned long options, char rm_contact)
char callback_contact GLOBUS_NULL char
job_contact GLOBUS_NULL
globus_i_globusrun_gram_monitor_t monitor
int err monitor.done GLOBUS_FALSE
monitor.verboseverbose globus_mutex_init(mo
nitor.mutex, GLOBUS_NULL) globus_cond_init(m
onitor.cond, GLOBUS_NULL) err
globus_module_activate(GLOBUS_GRAM_CLIENT_MODULE)
if(err ! GLOBUS_SUCCESS) err
globus_gram_client_callback_allow(
globus_l_globusrun_gram_callback_func,
(void ) monitor,
callback_contact) if(err !
GLOBUS_SUCCESS) err
globus_gram_client_job_request(rm_contact,
request_string, GLOBUS_GRAM_PROTOCOL_JOB_STAT
E_ALL, callback_contact,
job_contact) if(err ! GLOBUS_SUCCESS)
globus_mutex_lock(monitor.mutex)
while(!monitor.done) globus_cond_wait(m
onitor.cond, monitor.mutex)
globus_mutex_unlock(monitor.mutex)
globus_gram_client_callback_disallow(callback_cont
act) globus_free(callback_contact)
globus_mutex_destroy(monitor.mutex)
globus_cond_destroy(monitor.cond)
36Gram
- Creating a job
- GramJob job new GramJob ("(executable/bi
n/sleep)(arguments15)") - Listening to state changes via Listeners
- job.addListener( new GramJobListener()
public void statusChanged(GramJob job)
System.out.println(Job
job.getIDAsString() " " Status
" job.getStatusAsString()) )
37gramClient
- from pyGlobus import gramClient
- from threading import
- cond 0
- condV Condition(Lock())
- try
- gramClient GramClient()
- callbackContact gramClient.set_callback(fun
c, condV) - jobContact gramClient.submit_request(rm,rsl,
- gramClient.JOB_STATE_ALL,
callbackContact) - condV.acquire()
- while cond 0
- condV.wait()
- condV.release()
- except GramClientException, ex
- print ex
38gramClient Job State Callback
- def func(cv, contact, state, error)
- global cond
- if state gramClient.JOB_STATE_FAILED
- print "Job failed"
- cv.acquire()
- cond 1
- cv.notify()
- cv.release()
- elif state
- gramClient.JOB_STATE_DONE
- print "Job is done"
- cv.acquire()
- cond 1
- cv.notify()
- cv.release()
39Redirecting stdout with gramClient
- from pyGlobus import gassServerEZ
- opts gassServerEZ.STDOUT_ENABLE
- server gassServerEZ.GassServerEZ(opts)
- url server.getURL()
- rsl "(executable/bin/sleep)(arguments15)
- (stdouts/dev/stdout)
url - (normal job submission from here)
40Remote Data Transfer
- Directly use the Grid FTP protocol to transfer
the data. - Use the gassCopy module for protocol independent
transfers. It currently supports the ftp, gsiftp,
http, and https protocols.
41Java UrlCopy (protocol independent file transfer)
import org.globus.io.urlcopy.
UrlCopy c new UrlCopy()
c.setSourceUrl(from) c.setDestinationUrl(to
) c.setUseThirdPartyCopy(true) // hint to
enable thridparty transfer // register a
transfer listener.... c.setListener(new
UrlCopyListener() public void transfer(int
total, int current)
System.out.println(total " " current)
public void transferError(Exception e)
System.out.println("transfer failed "
e.getMessage()) ) c.copy() //
this starts the copy
42GSI FTPClient (not GridFTP)
import org.globus.io.ftp. GSIFTPClient ftp
new GSIFTPClient(host, port) ftp.authenticate()
ftp.setType(FTPClient.BINARY) // we have
convenient calls like ftp.makeDir(dir)
ftp.deleteDir(dir3) // the listener for the
transfers.... listener new TransferProgressList
ener public void transfer(int total, int
current, String from, String to)
System.out.println(total " " current " "
from " " to) public void
transferError(String from, String to, Exception
e) System.out.println("transfer failed "
from " " to)
43Simple Examples
- Retrieve a Makefile from a remote site and save
it to Makefile.bak - File dst new File (Makefile.bak)
- ftp.get (Makefile, dst, listener)
- Retrieve all files in remote directory to a
directory called Destination - File destinationDir newFile(Destination)
- ftp.Get(, destinationDir, true, listener)
- Disconnect
- ftp.disconnect()
- Examples in source code apis/examples, UrlCopy,
FTPClient
44ftpClient
- Use Grid FTP to transfer a file.
- from pyGlobus import ftpClient
- from pyGlobus.util import Buffer
- handleAttr ftpClient.HandleAttr()
- opAttr ftpClient.OperationAttr()
- marker ftpClient.RestartMarker()
- ftpClnt ftpClient.FtpClient(handleAttr)
- ftpClnt.get(url, opAttr, marker, done_func,
condV) - buf Buffer(641024)
- handle ftpClnt.register_read(buf, data_func, 0)
- def data_func(cv, handle, buffer, bufHandle,
bufLen, offset, eof, error) - g_dest.write(buffer)
- if not eof
- try
- handle g_ftpClient.register_read(g_b
uffer, data_func, 0) - except Exception, e
-
45Performance Options in ftpClient
- Setting tcpbuffer size
- from pyGlobus import ftpControl
- battr ftpControl.TcpBuffer()
- battr.set_fixed(641024)
- Or
- battr.set_automatic(161024, 81024, 641024)
- opAttr.set_tcp_buffer(battr)
- Setting parallelism
- para ftpControl.Parallelism()
- para.set_mode(ftpControl.PARALLELISM_FIXED)
- para.set_size(3)
- opAttr.set_parallelism(para)
46Using ftpClient plugins
- Debug Plugin
- from pyGlobus import ftpClient
- f open(/tmp/out, w)
- debugPlugin ftpClient.DebugPlugin(f, foobar)
- handleAttr.add_plugin(debugPlugin)
- RestartMarker Plugin
- restartPlugin ftpClient.RestartMarkerPlugin(begi
nCB, - markerCB,
doneCB, None)
47gassCopy
- Provides a protocol independent API to transfer
remote files. - from pyGlobus import gassCopy
- from pyGlobus import ftpClient
- srcAttr gassCopy.Attr()
- handleAttr gassCopy.HandleAttr()
- destAttr gassCopy.Attr()
- ftpSrcAttr ftpClient.OperationAttr()
- ftpDestAttr ftpClient.OperationAttr()
- srcAttr.set_ftp(ftpSrcAttr)
- destAttr.set_ftp(ftpDestAttr)
- copy gassCopy.GassCopy(handleAttr)
- copy.copy_url_to_url(srcUrl, srcAttr, destUrl,
destAttr)
48Remote File IO
- Supports reading and writing remote files from
http, https, ftp, gsiftp servers. - Can be opened either as normal Python file
objects or as int file descriptors suitable for
use with the os module. - from pyGlobus import gassFile
- gassFile gassFile.GassFile()
- f gassFile.fopen(gsiftp//foo.bar.com/file,
r) - lines f.readlines()
- gassFile.fclose(f)
49Secure High-Performance IO.
- Uses the Grid Security Infrastructure to provide
authentication. - Provides access to the underlying network
parameters for tuning performance.
50TCP Server example
-
- attr NetIOAttr.TCPIOAttr()
- attr.set_authentication_mode(
io.GLOBUS_IO_SECURE_AUTHENTICATION_MODE_GSS_API) - authData AuthData.AuthData()
- authData.set_callback(auth_callback, None)
- attr.set_authorization_mode(
io.GLOBUS_IO_SECURE_AUTHORIZATION_MODE_CALLBACK,
authData) - attr.set_channel_mode( io.GLOBUS_IO_S
ECURE_CHANNEL_MODE_GSI_WRAP) - soc GSITCPSocket.GSITCPSocket()
- port soc.create_listener(attr)
- soc.listen()
- childSoc soc.accept(attr)
- buf Buffer.Buffer(size)
- bytesRead childSoc.read(buf, size, size)
We will develop a similar example for Java, It is
already available as part of the GASS server.
51TCP Client example
- attr NetIOAttr.TCPIOAttr()
- attr.set_authentication_mode(
io.GLOBUS_IO_SECURE_AUTHENTICATION_MODE_GSS_AP) - authData AuthData.AuthData()
- attr.set_authorization_mode( io.GLOBUS_I
O_SECURE_AUTHORIZATION_MODE_SELF, authData) - attr.set_channel_mode(
io.GLOBUS_IO_SECURE_CHANNEL_MODE_GSI_WRAP) - soc GSITCPSocket.GSITCPSocket()
- soc.connect(host, port, attr)
- nBytes soc.write(str, len(str))
52Web Services in Grids
- Provides a uniform, lightweight, standards based,
framework for - Service description (WSDL)
- Discovery (UDDI / WSIL)
- Invocation (SOAP)
- Allows the use of standard tools
- Separates the interface from implementation
details - Including separating service definition from
protocol bindings - Language/Programming model agnostic
- Reduces the cost of building scientific
applications - Uses SOAP over HTTP over the Grid Security
Infrastructure - Similar to HTTPS, but supports delegation
53GSISOAP
- Built on SOAP.py, looking at switching to ZSI
- GSISOAP client example
- from pyGlobus import GSISOAP, ioc
- proxy GSISOAP.SOAPProxy(https//host.lbl.gov80
81, namespaceurngtg-Echo) - proxy.channel_mode ioc.GLOBUS_IO_SECURE_CHANNEL_
MODE - proxy.delegation_mode
- ioc.GLOBUS_IO_SECURE_DELEGATION_MODE_NONE
- print proxy.echo(spam, spam, spam, eggs, and
spam)
54GSISOAP (cont.)
- GSISOAP server example
- from pyGlobus import GSISOAP, ioc
- def echo(s, _SOAPContext)
- cred _SOAPContext.delegated_cred
- Do something useful with cred here
- return s
- server GSISOAP.SOAPServer(host.lbl.gov, 8081)
- server.channel_mode ioc.GLOBUS_IO_SECURE_CHANNEL
_MODE_GSI_WRAP - server.delegation_mode ioc.GLOBUS_IO_SECURE_DELE
GATION_MODE_FULL_PROXY - server.registerFunction(SOAP.MethodSig(echo,
keywords0, context1), urngtg-Echo) - server.serve_forever()
55Reliable File Transfer Service
- A Grid Web Service that
- Provides a simple interface to reliably transfer
data between GridFTP servers - Supports reliable transfers in the face of
network outages, machine crashes, and other
faults - Supports the automated tuning of network
parameters to optimize bandwidth
56RFT Usage
Client
WSDL Repository (UDDI?, WSIL?)
GSI Proxy
Reliable File Transfer Service
GridFTP Server
GridFTP Server
57Fault Recovery
Reliable File Transfer Service
GridFTP Server
GridFTP Server
58Fault Recovery
Reliable File Transfer Service
GridFTP Server
GridFTP Server
59Fault Recovery (cont.)
Machine crashes
Reliable File Transfer Service
DB
GridFTP Server
GridFTP Server
60Fault Recovery (cont.)
Reliable File Transfer Service
DB
After machine reboot, recover incomplete
transfers from the DB
GridFTP Server
GridFTP Server
61RFT WSDL
ltdefinitions name"FileTransferService namespace
stuff gt ltmessage name "submitTransferRequest
"gt ltpart name "srcUrl" type
"xsdstring"/gt ltpart name "destUrl"
type "xsdstring"/gt ltpart name
"priority" type "xsdint"/gt lt/messagegt
ltmessage name "submitTransferResponse"gt
ltpart name "return" type "xsdstring"/gt
lt/messagegt ltmessage name "checkStatusRequest"gt
ltpart name "handle" type
"xsdstring"/gt lt/messagegt ltmessage name
"checkStatusResponse"gt ltpart name
"return" type "xsdstring"/gt lt/messagegt
62RFT Internals
Client
GSISOAP Interface
Transfer Queue
Persistent DB
Manager Thread
Transfer Threads
Optimization Module
GridFTP Server
GridFTP Server
63RFT Usage Example
from rftClient import handle srcUrl
gsiftp//host.lbl.gov/data/climate_model_1234 de
stUrl gsiftp//lorenz.ncar.edu/tmp/climate_mode
l_1234 try handle submit_transfer(srcUrl,
destUrl) except TransferRequestException, ex
print ex try status check_transfer_status
(handle) Except TransferRequestException, ex
print ex print status
64Many more CoGs
- Perl
- JSP
- CORBA
- .
- GUIs
- Applications
65Java CoG Requirements
- JDK 1.2
- Security
- Needs security package such as IAIK
- We are able to replace security packages based on
a couple of common functions with public domain
package. - We hope that these package will reach the
maturity of iaik
66Python CoG Requirements
- Python 2.0
- The GT2-beta (or better) release of Globus.
- Support for dynamic libraries.
- Currently pyGlobus is tested on
- Linux
- Solaris
- Should compile
- Will support win32 when Globus is ported
67Review
- Modifications to core Globus
- Http based protocols
- APIs are not sufficient we need services
- Components useful to enhance grid infrastructure
- JavaBeansTM can be used to integrate with COTS.
- Java and Python have a rich set of libraries
(network, ldap, ...). - Grid programming in Java and Python is easier!
- Java uses event model - not callbacks
- GUIs do not have everything we need scripting!
68References
- Java CoG Kit
- http//www.globus.org/cog.
- Python CoG Kit
- http//www-itg.lbl.gov/Grid/projects/pyGlobus
- Globus
- http//www.globus.org
- SWIG
- http//www.swig.org
- krjackson_at_lbl.gov
- gregor_at_mcs.anl.gov
69Acknowledgements
- This research is supported by the Mathematical,
Information, and Computational Science Division
subprogram of the Office of Advanced Scientific
Computing Research, U.S. Department of Energy,
under Contract W-31-109-Eng-38 and under Contract
DE-AC03-76SF00098 with the University of
California. Globus research and development is
supported by DARPA, DOE, and NSF.