Java CoGKit Provider for CondorG

1
Java CoGKit Provider for Condor-G

• Final Presentation
• By
• Vedang Joshi
• May 5, 2005

2
Background

• Grid / MetaComputing Infrastructure
• Advanced infrastructures designed to enable the
  coordinated use of distributed high-end resources
  for scientific problem solving
• Commodity Grid Toolkit
• Provides a commodity interface to Grid
  environment i.e, provides a mapping of grid
  services
• Commodity Java, CORBA, DCOM, etc.
• Leverages the power of Grid computing while
  remaining independent of the implementation
  details of the underlying Grid
• Much like an outlet for electricity
• PBSWeb is similar attempt for PBS-managed Grids

3
Resource Management Systems

• Condor
• Popular workload management system developed at
  Univ. of Wisconsin
• Two parts Job Management, Resource Management
• Counterparts LSF, Sun GRID Engine, PBS, MAUI,
  NQE.
• Condor-G Job Management part
• List the jobs in the queue,
• Detail the life cycle of a job
• Submit new jobs, put jobs on hold
• Manage input, output files, etc.
• Most like globus-job-submit
• Setup job dependencies for complex job flow

4
Running Jobs with Condor

• Code Preparation
• Jobs are not run interactively
• Console output and Keyboard input redirected to
  and from files suitably
• Files that contain keyboard entries for the
  program input need to be created
• Universe
• Runtime environment for Condor
• Most frequently used Standard and Vanilla
• Checkpointing and migration of jobs possible with
  Standard but not with Vanilla
• Vanilla less restrictive compared to Standard

5
Running Jobs with Condor

• Job Submission through Submit Description File
• Controls the details of job submission
• Contains the commands and keywords to direct the
  job execution
• What executables to run
• The files to use for Stdin (Keyboard and screen
  data)
• Platform type required to run the job
• Whom to notify by email upon job completion
• No. of times the program is to be run can be a
  great time saver
• Condor_submit ltltsubmit description filegtgt

6
Submit Description File (1)

• Example demonstrate use of multiple
• directories for data organization.
• Executable mathematica
• Universe vanilla
• input test.data
• output loop.out
• error loop.error
• Log loop.log
• Initialdir run_1
• Queue
• Initialdir run_2
• Queue

7
Submit Description File (2)

• Example 2 Features including Requirements and
  Rank
• Executable foo
• Requirements Memory gt 32 OpSys "IRIX65"
  Arch "SGI"
• Rank Memory gt 64
• Image_Size 28 Meg
• Error err.(Process)
• Input in.(Process)
• Output out.(Process)
• Log foo.log
• Queue 150

8
Condor Job Management

• Remove job (condor_rm)
• Place job on hold (condor_hold)
• Changing the priority of job (condor_prio)
• Does not change user priority
• Job analysis (condor_q analyze )
• x of machines rejected because of job
  requirements or their own requirements
• Y of machines serving users with better
  priority, etc.
• Job completion
• Job removed from queue and inserted in history
  file

9
Project work

• Goal Take Java CoG Kit, and add a Provider for
  Condor-G - the job management part of Condor
• Literature Review
• Determine what functionality could be mapped into
  CoGKit framework
• Implementation
• Classes for dealing with properly constructing
  Submit Description file and running the jobs
  accordingly.
• Integration with classes for Task Handling and
  Job submission
• Testing In progress

10
References

• Gregor von Laszewski, Ian Foster, Jarek Gawor,
  and Peter Lane, "A Java Commodity Grid Kit,"
  Concurrency and Computation Practice and
  Experience, vol. 13, no. 8-9, pp. 643-662, 2001,
  http/www.cogkit.org/.
• Condor version 6.6.8 Manual, Condor Team,
  University of Wisconsin, March 10, 2005.