Introduction to the NERSC J90 Cluster

1
Introduction to the NERSC J90 Cluster

• David Turner Tom DeBoni
• NERSC User Services Group
• April 1999

2
Hardware Overview

• J90se CPU
• 100 MHz, 200 MFlop, 64-bit vector processor
• "Scalar enhanced"
• J90 System
• Multiple J90se CPUs
• 1 GWord shared memory
• Large, fast RAID filesystem (/tmp)

3
Hardware Overview (cont.)

• NERSC J90 Cluster
• Machine CPUs /tmp
• killeen 20 161 GB
• bhaskara 28 368 GB
• franklin 28 371 GB
• seymour 28 334 GB
• fcrick 28 229 GB
• jwatson 28 229 GB

Interactive System ssh, telnet, cqsub

Batch Systems cqsub
4
File Systems

• HOME
• permanent (but not archival)
• 5 GB quota, regular backups, file migration
• 93.7 GB total
• local to killeen, NFS-mounted on batch systems
• poor performance for batch jobs
• /u/repo/user
• /Un/user
• /u/ccc/dpturner
• /U0/dpturner

5
File Systems (cont.)

• TMPDIR
• temporary (created/destroyed each session)
• no quota (but NQS limits 4 GB - 32 GB)
• no backups, no migration
• local to each machine
• high-performance RAID arrays
• system manages this for you
• A.K.A. BIG

6
File Systems (cont.)

• /tmp
• location of TMPDIR
• 14-day lifetime
• A.K.A. /big
• you manage this for yourself
• HPSS
• archival tape storage (and file migration)
• no quota (but tracked by SRU)
• access with hsi, pftp, or ftp

7
Environment

• Shells
• Supported
• sh
• csh
• ksh (same as sh)
• Unsupported
• tcsh (module load tcsh)
• bash (module load tools)

8
Environment (cont.)

• Modules
• Found on many Unix systems
• Sets environment variables, aliases, executable
  search paths, man search paths, header file
  include paths, and/or library load paths
• Exercise care modifying startup files!
• Useful options
• module list
• module avail
• module load modfile
• module switch modfile modfile.rev
• module display modfile
• module help modfile

9
Compiling and linking

• Programming
• Fortran 90 - f90
• No Fortran 77 compiler
• C/C - cc, CC
• Assembler - as
• Cray Message Passing Toolkit
• Use compiler (f90, cc, CC) for linking also
• f90 file naming conventions
• Use for Fortran 77-style code
• filename.f - fixed form
• filename.F - fixed form, run preprocessor first
• Use for Fortran 90-style code
• filename.f90 - free form
• filename.F90 - free form, run preprocessor first

10
Compiling and linking (cont.)

• Useful compiler options
• -dp Disable double precision
• -rlistop Controls content of listing file
• -Gn Debugging level
• 0 Full debugging (same as -g)
• 1 Block by block debugging
• -Rrunop Run-time checking
• a Argument number and type
• b Array bounds

11
Compiling and linking (cont.)

• Useful compiler options (cont.)
• -ev Static storage
• -On Optimization level
• 0 None
• 1 Conservative global scalar optimization
• 2 Moderate loop nest restructuring
• 3 Aggressive autotasking

12
Compiler Options Comparison

• Feature cf77 f90
• Static storage -a -ev
• Autotasking -Zp -Otask3
• Vectorization -Zv -Otask0,scalar3,vector3
• Optimization -Zp -Zv -Otask3,scalar3,vector3
• Overindexing (default) -Ooverindex

13
Compiling and linking (cont.)

• Use make for large projects
• setenv NPROC 2
• Fortran 90 modules
• Don't confuse with module command
• Each Fortran 90 module must be compiled before
  any routine that uses it
• Useful linker options
• -Mopt Load map options

14
Execution

• Multiprocessing
• setenv NCPUS 4 (csh)
• export NCPUS4 (ksh)
• "a.out Command not found."
• ./a.out ...

15
Execution(cont.)

• Interactive
• killeen, during business hours
• Small (.le. 80MW), short (.le. 10 hours) jobs
• Batch
• killeen, night and weekends
• Small (.le. 40MW), short (.le. 6 hours) jobs
• bhaskara, franklin, seymour, fcrick, jwatson
• Big (.le. 512MW), long (.le. 168 hours) jobs

16
Debugging

• totalview -h
• Display brief summary of commands
• totalview -L
• Line-mode interface (similar to cdbx)

17
Libraries

• Mathematics
• default includes nag, imsl, slatec
• modules for lsode, harwell
• Graphics
• ncar
• gnuplot
• I/O
• HDF - module load hdf
• netCDF - module load netcdf

18
Applications

• Amber
• module load amber41
• Ansys
• module load ansys54
• Basis
• module load basis11.8
• Gamess
• module load gamess
• Gaussian
• module load g98
• Nastran
• module load nastran

19
Batch Computing

• User creates shell script myscript
• Submits to NQE with cqsub myscript
• Returns NQE task id (e.g., t1234)
• NQE selects machine and forwards to NQS
• Job remains pending (NPend) until resources
  available
• NQS runs the job
• Assigns NQS job id (e.g., 5678.bhaskara)
• Run job in appropriate batch queue
• Job log returned upon completion

20
Tools

• ja
• ./name
• ja -cst -n name
• hpm
• prof
• flowview
• atexpert