PPT – UNIX PowerPoint presentation | free to view

About This Presentation

Title:

UNIX

Description:

... Cygwin, Xming UNIX Workstation Included in OS Apple Mac Exodus X Client runs on remote machine Graphical Application xterm xcalc Modelling and visualisation ... – PowerPoint PPT presentation

Number of Views:86

Avg rating:3.0/5.0

Slides: 57

Provided by: cics8

Category:

more less

Transcript and Presenter's Notes

Title: UNIX

1
Getting Started with HPC On Iceberg
Michael Griffiths and Deniz Savas Corporate
Information and Computing Services The University
of Sheffield www.sheffield.ac.uk/wrgrid
2
Outline

Introduction to Iceberg
Getting connected
Basic Linux
Using the Linux Operating System
Running Applications and Jobs on Iceberg
How to distribute work across the cluster
The N8 Facility
Getting Help

3
Types of Grids

Cluster Grid
Computing clusters ( e.g. iceberg )
Enterprise Grid, Campus Grid, Intra-Grid
Departmental clusters, servers and PC network
Cloud, Utility Grid
Access resources over internet on demand
Global Grid, Inter-grid
White Rose Grid, National Grid Service, Particle
Physics Data Grid

4
Iceberg Cluster
There are two head-nodes for the iceberg cluster

login
login
login
HEAD NODE1 Iceberg(1)
HEAD NODE2 Iceberg(2)
qsh,qsub,qrsh
qsh,qsub,qrsh
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
There are 232 worker machines in the cluster
All workers share the same user filestore
Worker node
Worker node
Worker node
Worker node
Worker node
Worker node
5
iceberg cluster specifications

AMD-based nodes containing
96 nodes each with 4 cores and 16 GB of memory
31 nodes each with 8 cores and 32 GB of memory
TOTAL AMD CORES 632, TOTAL MEMORY 2528
GBThe 8-core nodes are connected to each other
via 16 GBits/sec infiniband for MPI jobsThe
4-core nodes are connected via the much slower "1
Gbits/sec" ethernet connections for MPI jobs.
Scratch space on each node is 400 GBytes
Intel Westmere based nodes all infiniband
connected, containing
103 nodes each with 12 cores and 24 GB of memory
( i.e. 2 6-core Intel X5650 )
4 nodes each with 12 cores and 48 GB of memory
8 Nvidia Tesla Fermi M2070s GPU units for GPU
programming
TOTAL INTEL CPU CORES 1152 , TOTAL MEMORY
2400 GBScratch space on each node is 400
GBTotal GPU memory 48 GBEach GPU unit is
capable of about 1TFlop of single precision
floating point performance, or 0.5TFlops at
double precision. Hence yielding maximum GPU
processing power of upto 8 TFlops in total.

6
Introduction to Linux on Iceberg

What are UNIX and Linux?
The Shell
Getting Connected
Basic UNIX/Linux Commands
Working with Directories
Editing files
Help!
Running Programs
Exercise

7
What is UNIX/Linux ?

Multi-Tasking O/S
Multi-User O/S
Available on a range of Computers

8
History of UNIX/Linux?

Unix operating system was developed around 1969
in the Bell Labs
Originally written using C
Around 1990 Linus Torvalds of Helsinki University
started off a freely available academic version
of Unix
Linux is the Antidote to a microsoft dominated
future

9
Which UNIX?

SunOS Sun Microsystems now OpenIndiana,
OpenSolaris from Oracle
IRIX Silicon Graphics
HP-UX Hewlett Packard
Linux For IBM PC compatibles
There are a number of certification bodies with
published standards and test suites to ensure
quality of products. Such as
Posix Portable Operating System Interface ( IEEE
standard).

10
Linux and GNU

Linux is an implementation of Unix
Linux/Unix operating system is written in C
Linux is not part of the GNU project but uses the
same licensing agreements
Many of the linux utilities and tools are taken
from the GNU project.
There are many flavours of linux distributions.
The mix of the kernel (linux) with the utilities
(GNU and other) and the installation procedure
determine the flavour. Some of these are
Fedora
SuSE
Redhat
Debian
Mandrake
Knoppix

11
UNIX Internals (Simplified)

Kernel
System Internals
Shell
Command Interpreter
Programming language
File System
Process Management

12
UNIX Shells

sh Bourne Shell (Original Shell) (Steven Bourne
of ATT)
bash Bourne Again Shell (GNU Improved Bourne
Shell)
csh C-Shell (C-like Syntax)(Bill Joy of Univ. of
California)
ksh Korn-Shell (Bournesome C-shell)(David Korn
of ATT)
tcsh Turbo C-Shell (More User Friendly
C-Shell).
You can switch from one shell to another by just
typing the name of the shell. exit return you
back to previous shell.

13
Getting an account

All staff and research students are entitled to
use iceberg
For Registration See
http//www.shef.ac.uk/wrgrid/register
Staff can have an account by simply emailing
ucards-reg_at_sheffield.ac.uk

14
passwords

In normal linux environment the passwd command
can be used to change the user passwords.
However, because we manage passwords centrally
this command will not work on iceberg.
If you wish to change your iceberg password you
will have to do this via a web interface at the
following URL
http//www.shef.ac.uk/cics/password

15
Access iceberg Remote logging in

Terminal access is described at
http//www.shef.ac.uk/wrgrid/using/access
Recommended access is via any browser at
www.shef.ac.uk/wrgrid This uses Sun Global
Desktop ( All platforms, Graphics-capable)Also
possible
Using an X-Windows client ( MS Windows,
Graphics-capable)
Exceed 3D
Cygwin
Various ssh clients (MS Windows, Terminal-only )
putty, SSH
Note ssh clients can also be used in combination
with Exceed or Cygwin to enable graphics
capability. Above web page describes how this can
be achieved.

16
Access icebergfrom MAC or Linux platforms

The web browser method of access ( as for Windows
platforms) also works on these platforms.
More customary and efficient method of access is
by using the ssh command from a
command-shell.Example ssh X
iceberg_username_at_iceberg.shef.ac.ukNote1 -X
parameter is needed to make sure that you can use
the graphics or gui capabilities of the software
on iceberg.Note2 Depending on the configuration
of your workstation you may also have to issue
the command xhost before the ssh command.

17
Basic X Concepts

X Server runs on local machine
PC Exceed, Cygwin, Xming
UNIX Workstation Included in OS
Apple Mac Exodus
X Client runs on remote machine
Graphical Application
xterm
xcalc
Modelling and visualisation packages etc.

18
Multiple ssh or xterm shells

There are no limits to the number of ssh or xterm
windows one can start simultaneously by methods
described in the previous slides. You may also
start extra xterm windows from the host by simply
typing xterm
On iceberg we also have a local command named
Xterm that starts up an xterm window with nicer
to use parameters. On iceberg we strongly
recommend that you use Qsh rather than xterm or
Xterm command so as to make use of a free worker
node. Qsh will act like Xterm but will make use
of a worker node.
Typing exit will terminate an xterm or ssh
session neatly.
This will also close the xterm window but not the
ssh window. ssh windows can be closed via the
file? exit menu.

19
Web browser method of access Sun Global Desktop
Start session on headnode
Start an interactive session on a worker
Start Application on a Worker
Help
20
Operating system and utilities

Linux Operating System version is-
Scientific Linux, which is RedHat Enterprise 5
based
Default Shell is- bash
Available editors-Graphical Editors
gedit ( not available on headnode)
emacs
nedit
Text editors
vi
pico ( not available on headnode)
nano

21
Login Environment

Default shell is bash ( you can request to change
your default shell ).
On login into iceberg many environment variables
are setup automatically to provide a consistent
interface and access to software.
Each user has a .bashrc file in their directory
to help setup the user environment.
Type set to get a list of all the environment
variables.
Change all variables that are in CAPITALS with
extreme care.
Modify/enhance your environment by adding
commands to the end of your .bashrc file , such
as alias commands. (Again do this with care! )

22
Some basic rules

Unix is case sensitive.
Commands are in lower case.
Backspace and/or Del Keys correct typing errors.
If the terminal parameters are not correctly set
try CtrlH
CtrlC Aborts a program or command.
You can use the arrow keys to recall previous
commands, optionally edit and execute them.

23
Key combinations in bash
CtrlA Move cursor to beginning of line
CtrlC ltBreakgt Stop running program
CtrlD ltLogoutgt
CtrlE Move cursor to end of line
CtrlH Delete character to the left
CtrlZ Suspend the program
Arrow left/right Move pointer left/right to insert chars etc.
Arrow up/down Recall previous commands so as to edit them andsubmit them by pressing ENTER
Tab Tab tab Command or filename completion. Auto-complete the command line (to save typing) If tab does not work tab again to show possibilities
24
Format of Unix commands

command option ... filename ...
eg ls
ls -l tutorial
more tutorial

25
List Directory

ls list directory
ls -l list directory in long format
ls -a list all (inc. hidden) files
-rw------ l course01 57 Oct 18 1105
hello.c

Access Permissions
Number ofbytes in file
Date and time last modified
26
Directory Structure
/ (root)
home
usr
cs4un2
cs4un1
Home directory of user cs4un1 /home/cs4un1
When you log in you are positioned in your home
directory. The environment variable HOME is also
set to contain this directory name.
27
Working with Directories

pwd print working directory
cd change directory
cd move to home directory
cd .. move up one level
cd mydir move into a subdirectory
cd /var/adm move to an absolute directory
mkdir directory_name create a new directory
rmdir directory_name delete an empty directory

28
Filenames

Filenames can comprise of
a-z, A-Z alphabetic characters
0-9 digits
.-_ special characters
montue_01.06-03-96
Wildcards when referencing files
any character or sequence of characters
? any single character

29
Displaying contents of a text_file

more filename
This command will start listing the contents of
filename on screen and pause after a screenfull
of data. While pausing, use the following
characters to control the output.
Spacebar next screenful
n Spacebar next n lines
Enter next line
b back one screen
n b back n screens full
q quit
? or h list commands
where n is a whole number

30
Displaying contents of a text_file continued

cat options filename filename
This command will output the contents of
filenames to standard-output ( normally screen)
without pausing.
Following options are useful
-v display non-printing characters
-n display with lines numbered on the left
tail -n filename
This command lists the last 10 lines of a text
file.
If a number is specified (.eg. -20 ) lists the
last n (i.e 20) lines

31
Copying files

Copy files (optionally directories)
cp fromfile tofile
Some of the useful options are
-R or r Recursive copy fromfile is a
directory so the entire directory and its
contents are copied.
e.g. cp r mydir newdir
-p preserve. Preserves all attributes of
the file ,such as access rights and creation
date.
Copy and concatenate files by using cat
Cat command concatenates contents of list of
files and directs the output to standard output
(normally screen). When used with redirection gt
it can be used to join files together.
e.g. cat file1 file2 file3 gt new_big_file

32
Renaming and deleting files

mv This command will move a file or directory to
a new location. It can thus be used to rename
files/directories as well as change their
locations in the global directory structure.
Syntax
mv source destination
Example
mv myfile mynewfile
mv myfile subdirectory/myfile
mv mysubdir mynewsubdir
rm This command will delete a file (optionally
a directory if used with r option).
Syntax rm object_to_delete
Example
rm myfile
rm r mydirectory

33
Working with files

To copy a file cp my_file my_new_file
To move ( or rename ) a file mv my_file
my_new_file
To delete a file rm my_file
To list the contents of a file less
file_name
To make a new directory( i.e. folder) mkdir
new_directory
To copy a file into another directory cp my_file
other_directory
To move a file into another directory mv my_file
other_directory
To copy a directory to another location cp R
directory other_directory
To remove a directory and all its contents!!! rm
R directory
( use with care )
Wildcards means matching any sequence of
characters.
For example cp .dat my_directory

34
Problem Session

Attempt problem 1-5 on the handout

35
Transferring files to/from iceberg

Summary of file transfer methods as well as links
to downloadable tools for file transfers to
iceberg are published at http//www.sheffield.ac
.uk/wrgrid/using/access
Command line tools such as scp, sftp and gftp are
available on most platforms.
Can not use ftp ( non-secure ) to iceberg.
Graphical tools that transfer files by dragging
and dropping files between windows are available
winscp, coreftp, filezilla, cyberduck

36
Pitfalls when transferring files

ftp is not allowed to by iceberg. Only sftp is
accepted.
Do not use spaces in filenames. Linux do
not like it.
Secure file transfer programs sftp classify
all files to be transferred as either ASCII_TEXT
or BINARY.
All SFTP clients attempt to detect the type of a
file automatically before a transfer starts but
also provide advanced options to manually declare
the type of the file to be transferred.
Wrong classification can cause problems
particularly when transfers take place between
different operating systems such as between Linux
and Windows.
If you are transferring ASCII_TEXT files to/from
windows/Linux, to check that transfers worked
correctly while on iceberg, type cat v
journal_fileIf you see a M at the end of each
line you are in trouble !!!
CURE dos2unix wrong_file on iceberg

37
Your filestore

Three areas of filestore always available on
iceberg
These areas are available from all headnode and
worker nodes.
home directory
/home/username
5 GBytes allocations
Permanent, secure, backed up area ( deleted files
can be recovered )
data directory
/data/username
50 GBytes of ollocation
Not backed but mirrored on another server
/fastdata area
/fastdata
Much faster access from MPI jobs
No storage limits but no backup, or mirroring
Files older than 90 days gets deleted
automatically
Always make a directory under /fastdata and work
there.

38
Scratch area ( only available during a job)

Located at /scratch
Used as temporary data storage on the current
compute node alone.
File I/O to /scratch is faster than to NFS
mounted /home and /data areas
File I/O to small files is faster than to
/fastdata but
for large files /fastdata is faster than
/scratch
Data not visible to other worker nodes and
expected to exist only during the duration of the
job.
HOW TO USE SCRATCH
Create a directory using your username under
/scratch on a worker and work from that directory
Example mkdir /scratch/USER
cp mydata /scratch/USER
cd /scratch/USER
( then run your program )

39
Storage allocations

Storage allocations for each area are as follows
On /home 5 GBytes
On /data 50 GBytes
No limits on /fastdata
Check your usage and allocation often to avoid
exceeding the quota by typing quota
If you exceed your quota, you get frozen and the
only way out of it is by reducing your
filestorage usage by deleting unwanted files via
the RM command ( note this is in CAPITALS ).
Requesting more storage
Email iceberg-admins_at_sheffield.ac.uk to request
for more storage.
Excepting the /scratch areas on worker nodes, the
view of the filestore is the same on every worker.

40
Running tasks on iceberg

Two iceberg headnodes are gateways to the cluster
of worker nodes.
Headnodes main purpose is to allow access to the
worker nodes but NOT to run cpu intensive
programs.
All cpu intensive computations must be performed
on the worker nodes. This is achieved by
qsh command for interactive jobs and
qsub command for batch jobs.
Once you log into iceberg, taking advantage of
the power of a worker-node for interactive work
is done simply by typing qsh and working in the
new shell window that is opened. The next set of
slides assume that you are already working on one
of the worker nodes (qsh session).

41
Where on the cluster ?

Most of the application packages, compilers and
software libraries are only available on the
worker_nodes.
Iceberg headnodes are suitable for only
light-weight jobs such as editing files.
If you are on one of the headnodes, you will need
to type qsh to get an interactive session to the
worker nodes.
How do you know where you are ? The command
prompt will contain your userid_at_hostname
Example ch1abc_at_node-056
If you are on one of the headnodes, hostname will
be iceberg1 or iceberg2
If you are on an amd-based worker-node, it will
be amd-nodenn
where nn is the number of the amd-node.
If you are on an intel-based worker-node it will
be node-nnn
where nnn is the number of the intel node.
You can always type echo HOSTNAME to find out
the name of the machine you are currently using.

42
Running programs on iceberg

Iceberg is the gateway to the cluster of worker
nodes and the only one where direct logging in is
allowed.
Icebergs main purpose is to allow access to the
worker nodes but NOT to run cpu intensive
programs.
All cpu intensive computations must be performed
on the worker nodes. This is achieved by the qsh
command for the interactive jobs and qsub command
for the batch jobs.
Once you log into iceberg, taking advantage of
the power of a worker node for interactive work
is done simply by typing qsh and working in the
new shell window that is opened. This what
appears to be a trivial task has would in fact
have queried all the worker nodes for you and
started a session on the least loaded worker in
the cluster.
The next set of slides assume that you are
already working on one of the worker nodes (qsh
session).

43
Running programs

Two modes of operation foreground and background
Foreground Interact with program via
keyboard/screen
Background No connection with keyboard/screen
Submit to backbround by Appending
EG myprog gt myfile
The symbols gt redirect output and any errors
to the file myfile
Although the above method of running jobs on the
background is feasible, the prio we recommend
that you submit your background into the batch
queue via the qsub command.

44
Redirection

Most unix commands are not aware of the source of
their input or the destination of their output.
They simply read/write from/to stdin/stdout. The
shell takes care of these issues.
Standard Input (defaultgtkeyboard)
Standard Output (defaultgtscreen)
Redirection symbols lt,gt,gtgt can be used to specify
files as the source/destination of the read/write
operations to override the above defaults.

45
Redirection continued

Most unix commands are not aware of the source of
their input or the destination of their output.
They simply read/write from/to stdin/stdout. The
shell takes care of these issues.
To redirect the output to a file use the gt
symbol.
Example ls -l gt dirlist
The gt symbol should be used with care as it
may over-write an existing file. gtgt symbol can
be used instead if the output should be appended
to the end of an existing file rather than
over-writing it.
Example ls -l gtgt logfile
If nothing is directed to a file then a zero size
file is created, or if the file already existed
then the contents of the file is removed
Example gt afile
The file /dev/null is a special symbol to
indicate a black-hole
Example ls l gt /dev/null

46
Redirection continued

If any program expects any of its input from the
standard-input-channel , i.e. the keyboard, it
can also read the same information from a file by
redirection.
To read input from a file use the lt symbol.
Example write cs1xyz lt message.fil
Here any text input write program expects from
the keyboard will be simply read from a file
named message.fil.
Each end-of-line will be treated as an ltENTERgt
on the keyboard.

47
Piping

Feeding the output of one command into the input
of another command
The symbol is called a pipe
command command
eg ls -al more
ls -la grep Nov

48
Examples of re-direction and piping

ls l grep Jun
ls l grep Jun gt june_files
ls l grep Jun cut c 57-80 gt june_files
cut c 1-10 lt test_files
aspell l lt message.txt gt report.txt
grep fluent lt news.dat
grep fluent lt news.dat cut c 1-72
(grep fluent lt news.dat ) cut c 1-72
(grep fluent lt news.dat ) cut c 1-72 gt
fluent.news

49
Foreground Program Control

Kill a program Ctrl C
Stop a program Ctrl Z
Note a stopped program still exists in the system
and hence can be re-started.

50
Program control within current shell

jobs Lists jobs (programs)
bg job_id Place a job in the background
fg job_id Return a job to the foreground
stop job_id Stop a job
kill job_id Kill a job
Process_id can be used in place of job_id for
more definitive way of identification.
jobs
1 Running time.sh gt out
stop 1
1 Stopped (signal) time.sh gt out
bg 1
1 time.sh gt out
kill 1
Terminated

51
Program control using ps and kill

ps Report process status
ps -f -u username
UID PID PPID C STIME TTY TIME CMD
username 24816 24585 0 162304 pts/50 000
sleep 2
username 20169 19956 0 160545 pts/50 001
-csh
username 24585 20169 0 163507 pts/50 000
/bin/sh time.sh
kill Terminate process
Find its process ID (PID) using the ps command
Kill the process using the kill command
kill 24585
Sometimes kill on its own does not work so try
kill -KILL 24585 or kill -9 pid

52
Repeating Previous Commands

Operation
history List previous commands
!! re-run last command
!n re-run the nth command
!str last command starting with str
eg !vi
Setup
Add the following to your .cshrc file
set history40

53
Searching in files

grep string file
This command finds and prints out the lines in
the file(s) containing the specified string
string word or phrase
file file or list of files (wild_card can be
used)
Note We strongly advise that the string is
quoted.
Examples grep Green Man england.dat
grep Zodiac t.dat
grep Zonea-z security.fil

54
Manual Pages and Info system

Man Manual pages Give text-based help on
usage.
Manual pages are grouped into sections (1,2,3,4
..).
There is usually one manual page per command
which is located in one of the directories
defined by the MANPATH environment variable.
To access the Manual page for a command just
type
man command
To get a list of manual pages that contain a
word type
man k topic
Info Similar to man but can scroll with cursor
keys and link to other information.