The Java CoG Kit

1
The Java CoG Kit

• Gregor von Laszewski
• Argonne National Laboratory
• University of Chicago
• gregor_at_mcs.anl.gov
• http//www.cogkit.org
• Updated slides will be available on the CoG Kit
  web site

The Globus Alliance contains the following
members in alphabetical order
2
Funding sources Acknowledgement

• The Java CoG Kit receives funding from the
  following sponsors
• DOE MICS
• NSF NMI
• Previous versions of the CoG Kit also received
  funding from
• NCSA Alliance
• Please, contact gregor_at_mcs.anl.gov in case you
  like to work with us more closely.
• Acknowledgement
• CoG Team, Globus Team, Globus Alliance, many
  others as listed on www.cogkit.org

3
Community

• Call on the community to help us with extending
  and improving the CoG Kit

4
Outline

• What is the CoG Kit?
• Basic definitions
• History of the CoG Kit
• CoG Kit in action
• Relationship to GT versions
• Selected Project Components
• Design Abstractions
• Programming with Abstractions (Task Graphs)
• Visual components Portals Applications
• Conclusion

5
Introduction
6
Observation

• Problem
• Many application developers desire to program the
  Grid in familiar higher level frameworks that
  allow rapid prototyping.
• Solution
• We propose to reuse a variety of commodity
  tools, protocols, approaches, methodologies,
  while integrating Grid software based on the
  Globus Toolkit
• Easier development of advanced Grid services
• Easier and more rapid application development
• Easier deployment of Grid services
• Code reuse and use of component repositories
• Use of Web services as part of the Grids
• Widespread use of the Grid
• Use of commodity technology is not limited to the
  client!

7
Abstractions

• Hypothesis
• With rapidly changing technologies it may be
  beneficial to have an abstraction that can be
  assisting in this technical challenge.
• Solution
• CoG Kit abstractions are defined for precisely
  that reason.

8
Result CoG Kits

• CoG Kits make Grid programming simple and new
  technologies are easy to integrate
• We focus on a CoG Kit for Java
• Others are possible Python,
• Availability Java CoG Kit since 1997
• Our designs are based on experience reaching back
  to the beginnings of Meta-computing and
  Grid-computing

9
Relationship towards GT

• Since GT3 CoG Kit is an essential part of GT
• CoG Kit protects from an evolving standard
• CoG Kit provides simple programming model
• CoG Kit supports portal and GUI developers
• CoG Kit is a bridge between application and Grid
  middleware developers.
• CoG Kit has known to be working with
• GT1.0 GT2.4, GT3.0.2, GT3.2, GT3.21, SSH
• (under dev.) GT3.9.x, GT4, Condor
• (community) Unicore

10
Relationship to WS-RF

• Because
• (Quote Steve Tuecke, at a GGF meeting)WS-RF
  is still under development. The OASIS standards
  process has just begun.
• COG Kit
• Provides investment protection while standards
  are developed.
• Provides a more sophisticated programming model
  than just services
• Focus on what many high end-users need
• You can influence the direction of the CoG Kit by
  partnering with us
• Will work with future versions of GT, SSH, Condor
  (planed),
• We intend to support and integrate with upcoming
  new standards

11
History
12
History (cont.)
1992 Term Metacomputer is introduced
1994 von Laszewski Graphical Meta-computing environment
1995 I-Way
Nov. 1996 von Laszewski joins Argonne
1997 Globus version 1 / first release of jglobus based on concepts of protocols and services includes a high throughput fault tolerant workflow prototype
1998 CoG Team LDAP browser wins Novel developers award
1998 Term Grid is introduced
1999 Term Java CoG Kit is introduced to include jglobus and other components in a single toolkit
2000 CoG Team The experimental personal gatekeeper of the Java CoG Kit was been able to be installed in less than 30 seconds on a PC including Windows, a similar Globus service installed by an experienced administrator required multiple days.
2001 Cog Team The Java CoG Kit experimental Infogram Service architecture was defined combining execution and information Services as a single Grid service.
2001 CoG Kit was selected by IBM to demonstrate Grid computing in Boards of Directors meeting
2002 Globus Team defines OGSI / Java CoG Kit for GT2.x and GT3.0/OGSI based, includes visual components such as the CoG Kit Desktop, GridFTP interface, GRAM interface
13
History
2003 WSRF is defined
2002 and 2003 CoG team rewrites the workflow component and introduces GridAnt and a new workflow engine called Karajan that contains flow and structural control (DAGs, conditions, loops). The workflow concept is expandable. Check pointing and minimal features for fault tolerance are available. Result caching is possible based on method signatures.
2004 CoG team introduces the concept of Grid providers making it possible that the CoG Kit can in principal submit to GT2, GT3, GT4, or SSH. Community demonstrates also UNICORE provider.
2004 A class project shows it is possible to define PBS and LSF providers (not distributed with the CoG Kit)
2004 CoG Kit receives best research poster award at SC 2004
2005 Major new Java CoG Kit release. GT2, GT3, GT4, SSH providers Workflow Graphical components New manual
14
Use of CoG Kits
Frameworks
OGCE
Python
Java
Astronomy
Portalware
OGCE
Applications
Climate
Chef
Cactus
GridLab
Nano Materials
HEP
GridSphere
Genome
GADU
Chemistry
Launchpad
CoG
GRIP/Unicore
Chimera
Pegasus
Nimrod/G
GAF4J
XCAT
Workflow
Globus Toolkit Version 3
Parameter Studies
Java CoG Kit
GridAnt
Task Management
OGSA/OGSI
File Transfer GridFTP RFT
Job Submission
Globus Toolkit Version 2
Perl CoG Kit
Grid Services
Security
Middleware
DOE
CERN
Information Services
NPACI
PACI
Access Grid
Production
15
Design
16
Design

• Based on layered model
• Flexible
• Expandable
• Based on Java interfaces
• Abstracts protocols
• Abstracts services
• Provides workflow

17
CoG Kit is more than jglobus
Essential part of GT3.02 GT3.2, GT3.2.1 GT3.9.x,
GT4.0
18
CoG Kits further the Grid
Application Grid Portals
Java CoG Kit
Python CoG Kit
Grid Applications
Biology
Chemistry
High Energy Physics
Earth Science

CoG Portal Services
Grid Portals
Advanced CoG Services (File Management, Job
Management, Security Management)
Grid Management Services
Grid Middleware
SWIG
Globus Toolkit
Execution Security
GT2
GT3
OGSA
GT4
Grid Primitives
Grid prototyping Web Services
19
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 OGSI classic
GT4 WS-RF
Condor
Unicore
SSH
Others Avaki SETI
20
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 OGSI classic
GT4 WS-RF
Condor
Unicore
SSH
Others Avaki SETI
21
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 OGSI classic
GT4 WS-RF
Condor
Unicore
SSH
Others Avaki SETI
22
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications
GT2
GT3 OGSI classic
GT4 WS-RF
Condor
Unicore
SSH
Others Avaki SETI
23
CoG Abstraction Layers
Development Support
Nano materials
Bio- Informatics
Disaster Management
Portals
Applications

• // Check whether you can submit a job to a
  particular gatekeeper.
• Gram.ping( proxy, hot.mcs.anl.gov)
• // Create a job
• GramJob job new GramJob(proxy, rsl.toRSL())
• // Add a status change listenerclass
  GramJobListenerImpl implements GramJobListener
  public void statusChanged(GramJob job)
  String status job.getStatusAsString()
  System.out.println(status)
  job.addListener(new GramJobListenerImpl())
• // Submit the job to a GRAM resource
  managerjob.request(hot.mcs.anl.gov) //
  default IANA port 2119
• //Cancel the job, if need be.job.cancel()

GT2
GT3 OGSI classic
GT4 WS-RF
Condor
Unicore
SSH
Others Avaki SETI
24
Rapid Prototyping Job Submission
callback_func(void user_arg, char job_contact,
int state, int errorcode)
globus_i_globusrun_gram_monitor_t monitor
monitor (globus_i_globusrun_gram_monitor_t )
user_arg globus_mutex_lock(monitor-gtmutex)
monitor-gtjob_state state
switch(state) case GLOBUS_GRAM_PROTOCOL_
JOB_STATE_PENDING globus_i_globusrun_gram_m
onitor_t monitor monitor
(globus_i_globusrun_gram_monitor_t ) user_arg
globus_mutex_lock(monitor-gtmutex)
monitor-gtjob_state state switch(state)
case GLOBUS_GRAM_PROTOCOL_JOB_STATE_FAILED
if(monitor-gtverbose)
globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE
_FAILED\n") monitor-gtdone
GLOBUS_TRUE break case
GLOBUS_GRAM_PROTOCOL_JOB_STATE_DONE
if(monitor-gtverbose)
globus_libc_printf("GLOBUS_GRAM_PROTOCOL_JOB_STATE
_DONE\n") monitor-gtdone
GLOBUS_TRUE break
globus_cond_signal(monitor-gtcond)
globus_mutex_unlock(monitor-gtmutex)
globus_l_globusrun_gramrun(char request_string,
unsigned long
options, char
rm_contact) char callback_contact
GLOBUS_NULL char job_contact
GLOBUS_NULL globus_i_globusrun_gram_monitor_t
monitor int err monitor.done
GLOBUS_FALSE monitor.verboseverbose
globus_mutex_init(monitor.mutex, GLOBUS_NULL)
globus_cond_init(monitor.cond, GLOBUS_NULL)
err globus_module_activate(GLOBUS_GRAM_CLIENT
_MODULE) if(err ! GLOBUS_SUCCESS)
err globus_gram_client_callback_allow(
globus_l_globusrun_gram_callback_func,
(void ) monitor,
callback_contact) if(err !
GLOBUS_SUCCESS) err
globus_gram_client_job_request(rm_contact,
request_string, GLOBUS_GRAM_PROTOCOL_JOB_STAT
E_ALL, callback_contact,
job_contact) if(err ! GLOBUS_SUCCESS)
globus_mutex_lock(monitor.mutex)
while(!monitor.done) globus_cond_wait(m
onitor.cond, monitor.mutex)
globus_mutex_unlock(monitor.mutex)
globus_gram_client_callback_disallow(callback_cont
act) globus_free(callback_contact)
globus_mutex_destroy(monitor.mutex)
globus_cond_destroy(monitor.cond)
25
CoG integrates technologies
26
Selected Project Components
27
Focus on Reusable APIs Components

• Abstractions
• Provide a simple programming model
• Workflow
• Workflow abstraction
• Portals
• Supporting APIs, abstractions and implementations
  for portals.
• jglobus1.2
• GSI security in Java
• GRAM protocal client
• gridFTP protocol client
• Myproxy client
• Not just APIs but also their implementation

28
Focus on Abstractions and Patterns

• Abstraction above Grid Toolkits
• Task Model
• Jobs, information query, file transfer,
  authentication, others
• Gridfaces model
• Abstract views of GUIs to the Grid in different
  implementations (SWING, JSP, Portlets, )
• Data Types
• Queues, Sets, Brokers, Schedulers. Based on Task
  model

29
Java CoG Kit Abstractions

• A programming model based on a task model that
  simplifies elementary Grid patterns such as job
  execution, file transfer, and file operations.
• A programming model that includes execution flows
  in the form of directed acyclic graphs (DAG).
• The programming model is decoupeling the
  definition from the implementation, thus
  providing independence from current and future
  Grid versions.
• Only elementary Grid patterns are considered.
• It makes programming the Grid simple
• It makes developing Grid portals more easy
• Focus is selected functionality

30
Design

• ExecutableObject
• Task
• TaskGraph
• Handlers
• Events
• Service

31
Design
1
1
ExecutableObject
1
1
Identity
Status
SecurityContext
1

1

1
Task
Service
1
TaskGraph

1

1
1
Dependency
ServiceContact
1
Queue
Set
Specification
1
1
JobSpecification
FileTransferSpecification
FileOperationSpecification
1
1
1
1
TaskHandler
TaskGraphHandler
32
Programming with Abstractions
33
A simple Programming Pattern

• public class COG implements StatusListener
• public void create()
• public void submit ()
• public void statusChanged (StatusEvent e)
  
• public static void main (String arg)
• try
• COG cog new COG()
• cog.create()
• cog.submit()
• catch (Exception e)
• logger.error(Something went wrong,
  e)

34
Executing a Simple TaskGraph

• TaskGraph tg new TaskGraphImpl()
• public void create ()
• // define tasks
• ..
• / Add the tasks to the TaskGraph /
• tg.add(task1)
• tg.add(task2)
• tg.add(task3)
• tg.add(task4)
• tg.addDependency(task1, task2)
• tg.addDependency(task1, task3)
• tg.addDependency(task2, task4)
• tg.addDependency(task3, task4)
• public void submit()
• TaskGraphHandler handler new
  TaskGraphHandlerImpl()
• try
• handler.submit(tg)

Task 3
Task 1
Task 4
Task 2
35
Create a task

• Task task1 new Task()
• JobSpecification spec new JobSpecificationImpl()
  
• spec.setExecutable(/bin/ls)
• spec.addArguments(-la)
• spec.setStdOutput(output.txt)
• task1.setSpecification(spec)
• // bind the task (late binding)

36
Status Monitoring

• public void statusChanged (StatusEvent event)
• Status status event.getStatus( )
• logger.debug(Status changed to ''
  status.getStatusCode())
• if (status.getStatusCode( )
  Status.COMPLETED)
• logger.info(Task Done'')
• elsif (status.getStatusCode( )
  Status.FAILED)
• logger.info(Task Failed'')
• System.exit(1)

Users can design their own Event handeling logic
based on status changes
37
Using the Handler
Detailed information Can be retrieved if
exceptions are used

• try
• handler.submit (cog)
• catch (InvalidSecurityContextException ise)
  
• logger.error(Security Exception'', ise)
• System.exit(1)
• catch (TaskSubmissionException tse)
• logger.error(TaskSubmission Exception'',
  tse)
• System.exit(1)
• catch (IllegalSpecException ispe)
• logger.error(Specification Exception'',
  ispe)
• System.exit(1)
• catch (InvalidServiceContactException
  isce)
• logger.error(Service Contact Exception'',
  isce)
• System.exit(1)

38
Bind a Task to a Service

• Service service new ServiceImpl(Service.JOB_SUBM
  ISSION)
• service.setProvider(GT3_2_1'')
• // Set Security Context e.g. certificates and
  such
• SecurityContext securityContext
  CoreFactory.newSecurityContext(GT3_2_1'')
• securityContext.setCredentials(null) // e.g. set
  it to default in ./globus
• service.setSecurityContext(securityContext)
• // Set Contact e.g. where to go to
• ServiceContact serviceContact
• new ServiceContactImpl( http//127.0.0.18080
  /ogsa/services/base/gram/
• MasterForkManagedJobFactoryService)
• service.setServiceContact(serviceContact)
• task.setService(Service.JOB_SUBMISSION_SERVICE,
  service)

ServiceContact serviceContact new
ServiceContactImpl( http//127.0.0.18080)
39
Bind a Task to a Service

• Service service new ServiceImpl(Service.JOB_SUBM
  ISSION)
• service.setProvider(GT3_2_1'')
• // Set Security Context e.g. certificates and
  such
• SecurityContext securityContext
  CoreFactory.newSecurityContext(GT3_2_1'')
• securityContext.setCredentials(null) // e.g. set
  it to default in ./globus
• service.setSecurityContext(securityContext)
• // Set Contact e.g. where to go to
• service.setServiceContact(serviceContact)
• task.setService(Service.JOB_SUBMISSION_SERVICE,
  service)

40
Build Higher Level Abstractions Grid Resource
With a Queue(not yet released)

• // pseudocode
• interface QueueEnabledExecutionResource extends
  GridResource
• void setQueue (Queue queue)
• Queue getQueue ()
• void submit (String provider,
  ExecutableObject executableObject)
• void remove (Identity identity)
• void setServiceContact (ServiceContact
  serviceContact)
• ServiceContact getServiceContact (String
  provider, int serviceType)
• void setSecurityContext (String provider,
  SecurityContext securityContext)
• SecurityContext getSecurityContext (String
  provider)
• Enumeration getAllSubmittedTasks ()

41
Build Higher Level Abstractions Grid Resource
With QoS(not yet released)

• // pseudocode
• interface QoSEnabledExecutionResource extends
  GridResource
• void setQueue (Queue queue)
• Queue getQueue ()
• void submit (Date startTime, Date EndTime,
  String provider, ExecutableObject
  executableObject,)
• void remove (Identity identity)
• void setServiceContact (ServiceContact
  serviceContact)
• ServiceContact getServiceContact (String
  provider, int serviceType)
• void setSecurityContext (String provider,
  SecurityContext securityContext)
• SecurityContext getSecurityContext (String
  provider)
• Enumeration getAllSubmittedTasks ()

42
Workflow

• Java CoG Kit Abstractions
• Gridant/Karajan
• Grid Shell

43
A simple Grid job
ltset nameremote valuehot.mcs.anl.gov/gt
ltgridTransfer srchost"localhost"
srcdir"/tmp" srcfile"in"
desthost"remote" destdir"/tmp"/gt
ltgridExecute host"remote"
executable"/usr/bin/climate"
args512 512"
stdin"/tmp/in"
stdout"/tmp/out"/gt ltgridTransfer
srchost"remote" srcdir"/tmp"
srcfile"out"
desthost"localhost" destdir"/tmp"/gt
transfer the input data
do the heavy processing
transfer back the results
44
A simple Grid job
ltset nameremote valuehot.mcs.anl.gov/gt
ltgridTransfer srchost"localhost"
srcdir"/tmp" srcfile"in"
desthost"remote" destdir"/tmp"/gt
ltgridExecute host"remote"
executable"/usr/bin/climate"
args512 512"
stdin"/tmp/in"
stdout"/tmp/out"/gt ltgridTransfer
srchost"remote" srcdir"/tmp"
srcfile"out"
desthost"localhost" destdir"/tmp"/gt
transfer the input data
do the heavy processing
transfer back the results
45
CoG Kit Desktop
Job Icons
Grid Shell
Machine Icons
File Transfer GUI
Grid Log
Native Icons
46
Portlets OGCE.org
47
Contributing

• You can contribute
• We have a module concept allowing components to
  be integrated in the distribution easily

48
Conclusion

• Programming with CoG abstractions is simple
• We envision multiple programming models in CoG
• We envision multiple backend services
• We can support multiple protocols
• We like to engage the community
• Contributions
• CA management, Unicore provider, gsissh
• These contributions are being integrated.