BigBen and the TeraGrid

1
BigBen and the TeraGrid

• Derek Simmellt dsimmel_at_psc.edu gt
• Raghu Reddylt rreddy_at_psc.edu gt

2
Overview

• Teragrid _at_ PSC
• Motivation BigBen TeraGrid ???
• portal.teragrid.org
• User Authentication / Proxy Certificates
• PSC grid-mapfile Web Interface
• Examples
• GSISSH
• globus-url-copy
• Globus job submission

3
TeraGrid _at_ PSC
PSC is a TeraGrid Resource Provider (RP),
specializingin large HPC systems including
BigBen, LeMieux andRachel. PSC also provides
data staging on Linux CacheNodes (LCNs) and
archival data storage via Slash/DMF.
4
MotivationBigBen TeraGrid ???

• Computational Workflows
• TeraGrid sites host a variety of resources
  including databases and computing systems suited
  to different kinds of computation
• Clusters, SMPs, MPPs, graphics/vis engines
• TeraGrid sites are connected via large-capacity
  networks
• Data copying / relocation / remote access
• Single Sign-On (with a some effort)
• Common grid software among systems
• Common environment variables among systems
• TG_CLUSTER_SCRATCH, TG_COMMUNITY, TG_EXAMPLES,
• Science Gateways
• Discipline- or application-specific web portals
• Emerging Web Services
• Information, data indexing and resource matching,
  reservations

5
portal.teragrid.org
6
Accounts Alloc Usage
7
TG Resource Status
8
TG User Documentation
9
TG Help Desk
10
Allocations
11
TeraGrid User Portal

• Single login username and password
• Unique user principal in .TERAGRID.ORG domain
• Used for both User Portal and myproxy server
  access
• Generates a short term NCSA CA user certificate
  upon login
• Stored in myproxy.teragrid.org for later
  retrieval
• Saves users from having to maintain a long-term
  certificate
• Future enhancement
• Single point of user certificate DN management
  for all TeraGrid resources

12
User Authentication /Proxy Certificates

• Globus, GSISSH and other GSI-enabled software
  rely on X.509 certificates for user, host and
  service authentication
• User certificate management hassle
• Avoid long-term user certificates!!
• Use short term PSC KCA or TeraGrid Portal (NCSA)
  certificates
• User certificate Distinguished Names (DNs) need
  to be installed in target hosts
  /etc/grid-security/grid-mapfile (
• NCSA CA certificate DNs distributed by default
  for all TG users
• PSC KCA certificate DNs deployed on all PSC
  systems
• Ways to generate a certificate proxy for
  authentication
• grid-proxy-init (for long-term user certs -
  yuck!)
• PSC KCA kinit PSCuser_at_PSC.EDU kx509 kxlist -p
• myproxy-get-delegation -L TUPuser -s
  myproxy.teragrid.org

13
Adding a certificate DN toTeraGrid host
grid-mapfiles

• For now, at most other TeraGrid sites, users must
  log into each system and execute gx-map -i to
  enter their certificate DN (a.k.a. Subject) into
  the/etc/grid-security/grid-mapfile on that
  system
• PSC and Purdue do NOT support gx-map
• Ways to find out your user certificate DN
  (Subject)
• grid-cert-info -subject -file usercert.pem
• openssl x509 -in usercert.pem -subject -noout
• (for PSC or USC KCA certificates only) kxlist

14
PSC grid-mapfileWeb Interface
http//dirs.psc.edu/cgi-bin/teragrid/userpage/list
.pl
15
Adding a DN toPSC grid-mapfiles
16
Adding a DN toPSC grid-mapfiles
17
Adding a DN toPSC grid-mapfiles
18
Adding a DN toPSC grid-mapfiles
19
Adding a DN toPSC grid-mapfiles
20
Checking DN entries

• grep username /etc/grid-security/grid-mapfile

dsimmel_at_tg-login2gt grep dsimmel
/etc/grid-security/grid-mapfile "/CUS/ONational
Center for Supercomputing Applications/CNDerek
Simmel" dsimmel "/CUS/OPittsburgh
Supercomputing Center/OUPSC Kerberos
Certification Authority/CNdsimmel/UIDdsimmel/em
ailAddressdsimmel_at_PSC.EDU" dsimmel "/CUS/OPitt
sburgh Supercomputing Center/OUPSC Kerberos
Certification Authority/CNdsimmel/USERIDdsimmel
/Emaildsimmel_at_PSC.EDU dsimmel "/CUS/OSDSC/OU
SDSC/CNDerek Simmel/UIDux454184"
dsimmel dsimmel_at_tg-login2gt
21
Examples

• Using Grid Security Infrastructure (GSI)-based
  tools
• Start by generating/obtaining a (proxy)
  certificate
• Use GSI-based clients with GSI-based services
• GSISSH
• globus-url-copy, tgcp,
• globus-job-run, globusrun, condor
• Finish by destroying (proxy) certificate
• grid-proxy-destroy
• Short-term (proxy) certificates will expire on
  their own

22
Getting a (proxy)certificate on BigBen

• PSC KCA short term user certificates
• Login to BigBen and execute
• kinit kx509 kxlist -p
• Remote users can do the same with KCA clients
  (kx509, kxlist) installed
• kinit PSCuser_at_PSC.EDU kx509 kxlist -p
• TeraGrid User Portal NCSA short term cert
• Log into the TeraGrid User Portal webpage
• Login to BigBen and execute
• myproxy-get-delegation -l TUPuser \-s
  myproxy.teragrid.org(note first argument is
  dash ell TeraGrid-User-Portal-username)

23
Getting a (proxy)certificate on BigBen

• PSC KCA certificate method

bash-2.04 kinit kx509 kxlist
-p dsimmel_at_PSC.EDU's Password Service
kx509/certificate issuer /CUS/OPittsburgh
Supercomputing Center/CNPSC Kerberos CA 1
subject /CUS/OPittsburgh Supercomputing
Center/OUPSC Kerberos Certification
Authority/CNdsimmel/UIDdsimmel/emailAddressdsim
mel_at_PSC.EDU serial32BE hasha31d5407 bash-2.04
grid-proxy-info subject /CUS/OPittsburgh
Supercomputing Center/OUPSC Kerberos
Certification Authority/CNdsimmel/UIDdsimmel/em
ailAddressdsimmel_at_PSC.EDU issuer
/CUS/OPittsburgh Supercomputing Center/CNPSC
Kerberos CA 1 identity /CUS/OPittsburgh
Supercomputing Center/OUPSC Kerberos
Certification Authority/CNdsimmel/UIDdsimmel/em
ailAddressdsimmel_at_PSC.EDU type end entity
credential strength 512 bits path
/usr/users/0/dsimmel/.globus/userproxy.pem timelef
t 95951
24
Getting a (proxy)certificate on BigBen

• TeraGrid User Portal certificate method
• Log in to the TeraGrid User Portal website, and
  then back on BigBen

bash-2.04 myproxy-get-delegation -l dsimmel -s
myproxy.teragrid.org Enter MyProxy pass phrase A
credential has been received for user dsimmel in
/usr/users/0/dsimmel/.globus/userproxy.pem. bash
-2.04 grid-proxy-info subject
/CUS/ONational Center for Supercomputing
Applications/CNDerek Simmel issuer
/CUS/ONational Center for Supercomputing
Applications/CNCertification Authority identity
/CUS/ONational Center for Supercomputing
Applications/CNDerek Simmel type end
entity credential strength 1024 bits path
/usr/users/0/dsimmel/.globus/userproxy.pem timelef
t 115953 bash-2.04
25
GSISSH

• GSISSH is an enhanced edition of OpenSSH
• Adds support for GSI (X.509) authentication
• Includes PSCs HPN-SSH
• http//www.psc.edu/networking/projects/hpn-ssh/
• No password required after proxy cert
  initialization
• A new proxy certificate is automatically
  generated upon successful remote login on the
  remote host
• Continue to use GSISSH, Globus commands, etc., on
  the remote host
• Tip Add host aliases to your /.ssh/config to
  avoid having to type in long TeraGrid hostnames

26
GSISSH Example
dsimmel_at_tg-login2gt myproxy-get-delegation -l
dsimmel -s myproxy.teragrid.org Enter MyProxy
pass phrase A credential has been received for
user dsimmel in /tmp/x509up_u17780. dsimmel_at_tg-log
in2gt gsissh tg-login1.lemieux.psc.teragrid.org
Compaq Tru64 UNIX V5.1B (Rev. 2650) Compaq
AlphaServer SC V2.6 UK1 This system is for the
use of authorized users only. Unauthorized use
may be monitored and recorded. In the course of
such monitoring or through system maintenance,
the activities of authorized users may be
monitored. By using this system you expressly
consent to such monitoring. If there are any
problems, please contact remarks_at_psc.edu. bash-2
.04 uname -a OSF1 iam763 V5.1 2650
alpha bash-2.04 grid-proxy-info subject
/CUS/ONational Center for Supercomputing
Applications/CNDerek Simmel/CN448983302 issuer
/CUS/ONational Center for Supercomputing
Applications/CNDerek Simmel identity
/CUS/ONational Center for Supercomputing
Applications/CNDerek Simmel type Proxy
draft (pre-RFC) compliant impersonation
proxy strength 512 bits path
/usr/users/0/dsimmel/.globus/userproxy.pem timelef
t 115935
27
globus-url-copy

• globus-url-copy is the Globus toolkit
  command-line client for GridFTP
• Supports 3rd-party transfers
• Supports striped transfers with multiple servers
• TeraGrid hosts have an scp-like wrapper, tgcp,
  that you can also use to copy files via GridFTP
• TeraGrid configurations help to pick optimal
  parameters
• BigBen has two dedicated GridFTP servers, each
  with 10Gb Ethernet interfaces
• gsiftp//tg-gridftp.bigben.psc.teragrid.org

28
globus-url-copy example

• 3rd-party transfer initiated on
  lemieux,transferring 100MB file from PSC
  archiverto user home directory on BigBen

bash-2.04 globus-url-copy -vb -stripe
\ gsiftp//tg-gridftp.psc.teragrid.org//100MB
\ gsiftp//tg-gridftp.bigben.psc.teragrid.org//te
st Source gsiftp//tg-gridftp.psc.teragrid.org/
/ Dest gsiftp//tg-gridftp.bigben.psc.teragrid.
org// 100MB -gt test 104857600 bytes
35.71 MB/sec avg 35.71 MB/sec
inst bash-2.04
29
Globus Job Submission

• PSC operates a centralized Globus Job Submission
  server
• gt4-submit.psc.teragrid.org
• Jobs may be submitted to BigBen at
• gt4-submit.psc.teragrid.org/jobmanager-bigben-pbs
• Current support is for Pre-webservices GRAM
• Webservices deployment is currently under way

30
Globus Job gt BigBen

• Globus GRAM Job Submission example initiated from
  Lemieux to BigBen

dsimmel_at_tg-login2gt globus-job-run
gt4-submit.psc.teragrid.org/jobmanager-bigben-pbs
\ -q debug /opt/xt-catamount/default/bin/cnos64/h
ello_qk DATE Thu Aug 24 130659 2006 PBS
JOB ID 61019 SCRATCH
/scratcha/dsimmel hello, world from node
2770, aka 0x0AD2 dsimmel_at_tg-login2gt
31
Clean-up

• When you have completed yourGSI-authenticated
  tasks, its a good idea to remove your (proxy)
  certificate
• grid-proxy-destroy

32
Questions?

• Feel free to contact us if you have questions
  regarding BigBen and the TeraGrid
• Derek Simmel lt dsimmel_at_psc.edu gt
• Raghu Reddy lt rreddy_at_psc.edu gt
• Thanks!