John Kewley

1
GROWL Scripts Lightweight Access to Grid
Resources

• John Kewley
• Grid Technology Group
• e-Science Centre
• STFC Daresbury Laboratory
• j.kewley_at_dl.ac.uk

2
GROWL

• Collaborative project between CCLRC (now STFC)
  Daresbury Laboratory and the Universities of
  Cambridge and Lancaster, funded by the JISC VRE
  programme.
• Project Objectives to produce a lightweight
  client-side Grid connection toolkit.
• Project completed January 2007

3
GROWL

• GROWL addresses the three barriers that newcomers
  find when using the Grid for the first time
• Setting up the client-side middleware
• Handling of certificates
• Job submission in the presence of firewalls
• This talk looks at GROWL Scripts, GROWL Web
  Services takes a different approach

4
Client Middleware Problems

• Typically need to be root to install (according
  to documentation)
• Software must be downloaded from various
  locations
• There are many choices for type of installation
  (too many options?)

5
Installation

• GROWL scripts provide an alternative way of
  installing Grid middleware on your client Linux
  machine to that given on the NGS website
• Advantages
• Don't need to be a privileged user
• Will download client middleware packages for your
  system (assuming it is supported)
• Minimal setup/configuration
• About 1015 mins (if all goes well !)

6

• The Virtual Data Toolkit (VDT) is an easy to
  install and configure ensemble of grid middleware
• http//vdt.cs.wisc.edu
• GROWL Scripts installs the pre-WS globus client
  from VDT, as well as gsi-enabled openssl and the
  best known (IGTF accredited) CA certificates.

7
Installing Grid Client using GROWL

• Download GROWL Scripts
• cd
• wget http//www.growl.org.uk/Growl.tar.gz
• Install into home directory
• tar -zxvf Growl.tar.gz
• Build VDT client (a software distribution that
  includes globus)
• cd Growl make VDT
• Before using any GROWL Scripts, bash users should
• source /Growl/setup.sh
• while csh users should
• source /Growl/setup.csh

8
GROWL Scripts Contents

• Certificate helper scripts
• mk-cert
• growl-info, growl-login, growl-logout
• VDT client installation of globus and MyProxy
• grid-proxy-init, grid-proxy-info
• globus-job-submit, globus-job-run
• gsissh, gsiscp, openssl
• myproxy-init, myproxy-info, myproxy-logon
• GROWL wrapper scripts
• growl-submit, growl-status, growl-get-output,
• growl-sh, growl-cp, growl-mkdir, growl-rm,
  growl-mv,
• growl-pwd, growl-which, growl-get-jobmanager,
  growl-queue

9
Certificate Manipulation

• Hard to remember openssl commands are wrapped for
  you
• Fewer passwords need to be entered
• Correct file and directory permissions are applied

10
mk-cert

• openssl pkcs12 in mykey.p12 \
• -clcerts nokeys
• -out usercert.pem
• ltPass1gt
• openssl pkcs12 in mykey.p12 \
• nocerts -out userkey.pem
• ltPass1gt
• ltPass2gt
• ltPass2gt confirm
• chmod 444 usercert.pem
• chmod 400 userkey.pem
• mv userkey.pem /.globus
• mv usercert.pem /.globus
• chmod 700 /.globus

• mk-cert mykey.p12
• ltPass1gt
• ltPass2gt

11
growl-login

• If you need to upload your certificate to MyProxy
  and generate a local proxy, growl-login is
  provided

grid-proxy-init ... ltgrid-passwordgt
myproxy-init Your identity /CUK/OeScience/OUCL
RC/LDL/CNjohn kewley Enter GRID pass phrase for
this identity ltgrid-passwordgt ... Enter MyProxy
pass phrase ltmyproxy-passgt Verifying - Enter
MyProxy pass phrase ltmyproxy-passgt
growl-login Password to protect MyProxy
credential ltmyproxy-passgt Enter GRID pass phrase
for this id ltgrid-passwordgt
12
growl-info

• A wrapper for grid-cert-info, grid-proxy-info and
  myproxy-info

growl-info Certificate Information (including
validity) ----------------------------------------
---- subject /CUK/OeScience/OUCLRC/LDL/CNjoh
n kewley notBeforeJun 15 161035 2006
GMT notAfterJun 15 161035 2007 GMT Local
proxy certificate(s) -------------------------- su
bject /CUK/OeScience/OUCLRC/LDL/CNjohn
kewley issuer /CUK/OeScience/OUCLRC/LDL/CN
john kewley identity /CUK/OeScience/OUCLRC/L
DL/CNjohn kewley type Proxy draft
(pre-RFC) compliant impersonation proxy strength
512 bits path /tmp/x509up_u13445 timeleft
115719
13
GROWL job submission

• Help with transparency - user shouldn't really
  need to know
• Machine's jobmanager
• Home directory location
• Location in your path of executable
• Firewall problems minimised

14
Running a grid job (1)
growl-submit dl1.nw-grid.ac.uk
hostname https//dl1.nw-grid.ac.uk64010/792/11647
5/ growl-status https//dl1.nw-grid.ac.uk64010
/792/116475/ PENDING growl-status
https//dl1.nw-grid.ac.uk64010/792/116475/ DONE
growl-get-output https//dl1.nw-grid.ac.uk64010
/792/116475/ comp023.nw-grid.ac.uk
growl-submit -c dl1.nw-grid.ac.uk hostname
growl-status PENDING growl-status DONE
growl-get-output comp021.nw-grid.ac.uk
15
Globus Firewalls
Grid Resource
Client
globus-job-submit
jobmanager
globus-job-get_result
Results
gsiscp
sshd
gsissh /GSI-SSHTerm
16
GROWL Firewalls
Client
Grid Resource
growl-submit
jobmanager
globus-job-get-output
growl-get-output (using gsissh)
sshd
17
Advantages

• growl-submit
• uses growl-get-jobmanager to obtain default
  parallel queue, rather than defaulting to
  jobmanager-fork
• uses growl-which to get full path of executable,
  ensuring it is in your path
• growl-get-output
• uses gsissh to do remote retrieval, avoiding
  client firewall problem

18
Remote filestore manipulation

• Equivalents of many of the standard unix command
  tools are provided for remote filestore
  manipulation.
• growl-ls contents of directory
• growl-mkdir (sub)directory creation
• growl-rm file removal
• growl-mv renaming/moving files
• growl-which finds executable in your path
• growl-pwd prints your home directory on the
  grid resource
• growl-sh gsissh wrapper (using default ports)
• growl-cp remote file copying, including "3rd
  party"
• An additional parameter (the grid resource) is
  required

19
Remote file copying using growl-cp

• growl-cp can be used to stage and retrieve files.
  The syntax follows that of scp. It can also be
  used for "3rd party" file transfers
• For 3rd party transfers to work, there has to be
  a route through all firewalls between the 2
  remote resources in one direction or the other

growl-cp my_input_file.txt dl1.nw-grid.ac.uk.
growl-cp dl1.nw-grid.ac.ukmy_output.txt .
growl-cp lv1.nw-grid.ac.ukmy_file.txt
dl1.nw-grid.ac.uk.
20
growl-cp (1)
Client
Grid Resources
B
A
21
growl-cp (2)
Client
Grid Resources
B
A
22
growl-cp (3)
Client
Grid Resources
B
A
23
Usage patterns

• Easy way to build VDT
• As above certificate scripts
• As above use of job submission features

24
Current/future work

• Scripting help for a simplistic DIY
  meta-scheduler
• Use of Java CoG-kit GSI-SSHTERM on Windows

25
GROWL Firewalls (3-tier)
Client
GROWL Server
Grid Resource
jobmanager
WS
WS I/F
growl-submit
GROWL Scripts
growl-get-output growl-cp
sshd
26
Summary

• Useful as an easy way to build VDT
• Simpler job submission
• less need be known about Grid resources
• less firewall pain for retrieving data
• Firewall-aware 3rd party file copying
• http//www.growl.org.uk/

27
GROWL Web Services

• Analogous interface to those we have seen on
  previous slides.
• Remote file manipulation
• Job submission
• Interfacing to SRB
• Wrappers being developed for access from R,
  Fortran, C and C
• Users at both Lancaster and Bristol using GROWL
  Web Services