Distributed File Systems

1
Distributed File Systems

• Objectives
• to understand Unix network file sharing
• Contents
• Installing NFS
• How To Get NFS Started
• The /etc/exports File
• Activating Modifications The Exports File
• NFS And DNS
• Configuring The NFS Client
• Other NFS Considerations
• Practical
• to share and mount NFS file systems
• Summary

2
NFS/DFS An Overview

• Unix distributed filesystems are used to
• centralise administration of disks
• provide transparent file sharing across a network
• Three main systems
• NFS Network File Systems developed by Sun
  Microsystems 1984
• AFS Andrew Filesystem developed by
  Carnegie-Mellon University
• Unix NFS packages usually include client and
  server components
• A DFS server shares local files on the network
• A DFS client mounts shared files locally
• a Unix system can be a client, server or both
  depending on which commands are executed
• Can be fast in comparasion to many other DFS
• Very little overhead
• Simple and stable protocols
• Based on RPC (The R family and S family)

3
General Overview of NFS

• Developed by Sun Microsystems 1984
• Independent of operating system, network, and
  transport protocols.
• Available on many platforms including
• Linux, Windows, OS/2, MVS, VMS, AIX, HP-UX.
• Restrictions of NFS
• stateless open architecture
• Unix filesystem semantics not guaranteed
• No access to remote special files (devices, etc.)
• Restricted locking
• file locking is implemented through a separate
  lock daemon
• Industry standard is currently nfsV3 as default
  in
• RedHat, SuSE, OpenBSD, FreeBSD, Slackware,
  Solaris, HP-UX, Gentoo
• Kernel NFS or UserSpace NFS

4
Three versions of NFS available

• Version 2
• Supports files up to 4GB long (most common
  2GByte)
• Requires an NFS server to successfully write data
  to its disks before the write request is
  considered successful
• Has a limit of 8KB per read or write request. (1
  TCP Window)
• Version 3 is the industry standard
• Supports extremely large file sizes of up to 264
  - 1 bytes
• Supports files up to 8 Exabyte
• Supports the NFS server data updates as being
  successful when the data is written to the
  server's cache
• Negotiates the data limit per read or write
  request between the client and server to a
  mutually decided optimal value.
• Version 4 is coming
• File locking and mounting are integrated in the
  NFS daemon and operate on a single, well known
  TCP port, making network security easier
• Support for the bundling of requests from each
  client provides more efficient processing by the
  NFS server.
• File locking is mandatory, whereas before it was
  optional

5
Important NFS Daemons

• Portmap The primary daemon upon which all the RPC
  rely
• Manages connections for applications that use the
  RPC specification
• Listens to TCP port 111 for initial connection
• negotiate a range of TCP ports, usually above
  port 1024, for further comms.
• You need to run portmap on both the NFS server
  and client.
• Nfs (rpc.nfsd)
• Starts the RPC processes needed to serve shared
  NFS file systems
• Listens to TCP or UDP port 2049 (port can vary)
• The nfs daemon needs to be run on the NFS server
  only.
• Nfslock (rpc.mountd)
• Used to allow NFS clients to lock files on the
  server via RPC processes.
• Neogated port UDP/TCP port
• The nfslock daemon needs to be run on both the
  NFS server and client
• netfs
• Allows RPC processes run on NFS clients to mount
  NFS filesystems on the server.
• The nfslock daemon needs to be run on the NFS
  client only.

6
The NFS Protocol Stack aka. VFS
MOUNT
NFS
server
client
XDR
RPC
TRANSPORT, NETWORK, LINK PHYSICAL LAYERS
RPC depend on PORTMAP which is on both client and
server
7
Installing kernelNFS, Linux

• Check if NFS is installed with rpm
• Check if RPC portmap package installed rpm
• If not Install them, allways begin with portmap
• If you are not running SuSE
• Install portmap, nfs-utils,nfs-server (should be
  implemented in kernel)

suse93 rpm -qa grep nfs nfs-utils-1.0.7-3 ya
st2-nfs-client-2.11.7-3 yast2-nfs-server-2.11.5-3
rpm -qa grep portmap portmap-5beta-733
rpm ivh http//ftp.sunet.se/pub/os/Linux/distri
butions/suse/suse/i386/9.3/suse/i586/portmap-5beta
-733.i586.rpm rpm ivh http//ftp.sunet.se/pub/o
s/Linux/distributions/suse/suse/i386/9.3/suse/i586
/nfs-utils-1.0.7-3.i586.rpm
8
How To Get kernelNFS server Started

• Activate the 3 nessesary servers for NFS at boot
• NFS server demon
• NFS file locking
• RPC portmap
• Start the PORTMAPPER and NFS server
• Which starts all dependent services
• Whatever you do allways start PORTMAP first
• Check that services for NFS is running with
  rpcinfo
• In some Unixes you need to separately start
• /etc/init.d/portmap start or shortly portmap(d)
• /etc/init.d/nfs start or shortly nfs(d)
• /etc/init.d/nfslock start or shortly nfslock(d)

insserv portmap insserv nfsserver
rcportmap start rcnfsserver start
rpcinfo -p localhost program vers proto
port 100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper 100003
2 udp 2049 nfs 100003 3 udp 2049
nfs 100227 3 udp 2049 nfs_acl
100003 2 tcp 2049 nfs 100003 3
tcp 2049 nfs 100227 3 tcp 2049
nfs_acl 100024 1 udp 1034 status
100021 1 udp 1034 nlockmgr
program vers proto port 100021 4 udp
1034 nlockmgr 100024 1 tcp 1029
status 100021 1 tcp 1029 nlockmgr
100021 3 tcp 1029 nlockmgr 100021
4 tcp 1029 nlockmgr 100005 1 udp
835 mountd 100005 1 tcp 838 mountd
100005 2 udp 835 mountd 100005
2 tcp 838 mountd 100005 3 udp
835 mountd 100005 3 tcp 838 mountd
9
How To Get NFS client Started

• Activate the 2 nessesary servers for NFS at boot
• NFS file locking nfslock
• RPC portmap
• Start the PORTMAPPER and NFS server
• With rc
• Check that services for NFS is running with
  rpcinfo
• Note! There can be more services running
  dependent on your system setup
• In some Unixes you need to separately start
• /etc/init.d/netfs start or shortly netfs(d)
• /etc/init.d/nfslock start or shortly nfslock(d)
• Allways start portmap first then netfs and last
  nfslock

insserv portmap
rcportmap start
rpcinfo -p localhost rpcinfo -p program vers
proto port 100000 2 tcp 111
portmapper 100000 2 udp 111
portmapper
10
NFS And DNS

• Check FORWARD resolution
• Check REVERSE resolution
• Both forward and reverse must be same
• If not, fix your DNS zonefiles (review netadmin
  chapter 3)
• Syncronized /etc/hosts in server and client will
  also do
• Some common error messages
• Lookup host resolution error
• Timeout firewall port setup
• Not registered portmap is not running

host 192.168.0.1 1.0.168.192.in-addr.arpa
domain name pointer a01.my-site.com.
host a01.my-site.com a01.my-site.com has
address 192.168.0.1
forward lookup doesn't exist
RPC Timeout
RPC Program not registered
failed server is down.
11
The NFS Server sharing directories

• The exportfs command is used to share directories
  on the network
• any directory can be exported
• subdirectories of an exported directory may not
  be exported unless they are on a different disk
• parents of an exported directory may not be
  exported unless they are on a different disk
• only local filesystems can be exported
• Some exportfs o sharing options
• We share the home directory in v verbose mode
• rw Read Write (default)
• squash_uids, squash_gids make user and group
  ids specified
• to be squashed to user with identity nobody
• directory is shared to host rosies only

ro read only access rw read and write
access sync write when requested wdelay wait for
sync hide dont show subdirs that is exported of
other export no_all_squash remote uids gids
become equal of client root_squash remote root
uid become anonymous on the client no_root_squash
remote root equals to local root user squash_uids
remote uids gids are threated as identity
nobody
exportfs v -o rw,squash_uids0-499,squash_gids
0-499 rosies/home exporting rosies/home
12
More on Shared Directories

• If someone is using the shared directory, you
  will not be able unshare.
• Check if someone is accessing RPC, using a share
• The first red line show that someone is using
• RPC against our server. The second red line
• show that someone have accessed /home
• Unshare a share in v verbose mode
• Check what the server is
• sharing

showmount a localhost All mount points on
server ,192.168.1.0/24/home /home /install/
suse9.3 rosies rosies,192.168.1.0/24
exportfs -v -u rosies/home unexporting
roseis/home
exportfs -v /home 192.168.1.0/24(rw,wd
elay,root_squash) /exports/network-install/SuSE/9.
3 ltworldgt(ro,wdelay,root_squash) /
install/suse9.3
ltworldgt(ro,wdelay,root_squash)
13
The /etc/exports File, static shares

• Sample exports file
• Some options in exports file (same as exportfs)
• Squash changes remote identity to selectable
  local identity
• Linux uses another format in /etc/exports than
  BSD systems

cat /etc/exports /data/files           (ro,sync
) /home                 192.168.0.0/24(rw,sync) /d
ata/test            .my-site.com(rw,sync) /data/d
atabase        192.168.0.203/32(rw,sync)
ro read only access rw read and write
access sync write when requested wdelay wait for
sync hide dont show subdirs that is exported of
other export no_all_squash remote uids gids
become equal of client root_squash remote root
uid become anonymous on the client no_root_squash
remote root equals to local root user squash_uids
remote uids gids are threated as identity
nobody
14
The /etc/exports File, Squashing

• Sample exports file using map_static
• Map_static file /etc/squash.map
• Squash changes remote identity to selectable
  local identity

cat /etc/exports /data/files           (ro,sync
) /home                 192.168.0.0/24(map_static
/etc/squash.map,rw,sync) /data/test            .m
y-site.com(rw,sync) /data/database        192.168.
0.203/32(rw,sync)
/etc/squash.map remote local comment uid
0-100 - squash to user nobody gid 0-100 -
squash to group nobody uid 1-200 1000 map to
uid 1000 - 1100 gid 1-200 500 map to gid 500 -
600 uid 0-100 2001 map individual user to uid
2001 gid 0-100 2001 map individual user to gid
2001
15
Activating Modifications in Exports File

• Re-reading all entries in /etc/exports file
• When no directories have been exported to NFS,
  then the "exportfs -a" command is used
• After adding share(s) to /etc/exports file
• When adding a share you can use the "exportfs -r"
  command to export only the new entries
• Deleting, Moving Or Modifying A Share
• In this case it is best to temporarily unmount
  the NFS directories using the "exportfs -ua"
  command followed by the "exportfs -a" command.
• Termporary export /usr/src to hosts on net
  192.168.0.0

exportfs -a
exportfs -r
exportfs -ua exportfs -a
exportfs 192.168.0.0/24/usr/src o rw
16
Exercise - Sharing Directories

• Write down the commands to do the following?

With one command share /usr/share readonly for
all clients in your net Permanently Share /etc
readonly for rosies and tokyo and read/write for
seoul list the file containing the permanent
shares two commands showing what your host has
shared check who has mounted your shared
directories check who has mounted directories
on rosies check the server nfs status From
the server, with one command check that the
nfs-client has portmapper running
17
The nfsstat Command
nfsstat -s

• Server statistics
• A large table arrives after command is issued
• Client statistics
• Server numbers of filehandlers
• Usage information on the server's file handle
  cache, including the total number of lookups,
  and the number of hits and misses.
• The server has a limited number of filehandlers
  that can be tuned

nfsstat -c
Server nfs v3 null getattr setattr
lookup access readlink 0 0 15
31 0 0 0 0 0 0 0
0 read write create mkdir
symlink mknod 0 0 0 0 0 0
0 0 0 0 0 0 remove rmdir
rename link readdir
readdirplus 0 0 0 0 0 0 0
0 0 0 0 0 fsstat fsinfo
pathconf commit 17 35 16 33 0
0 0 0
nfsstat -o fh Server file handle cache lookup
anon ncachedir ncachedir stale 0
0 0 0 0
18
Error Thresholds For The "nfsstat" Command
19
The NFS Client side

• Ensure Portmap Is Running
• Clients need portmap only to be running
• Also check that server is up
• If not, start portmap
• Show exported shares on a remote server
• Temporary mount nfs shares on client with default
  options
• umount temporaty mounted nfs shares on client

rpcinfo -p localhost
rpcinfo -p 192.168.0.10
rcportmap start
showmount -e 192.168.0.10 Export list for
192.168.0.10 /home
/exports/network-install/SuSE/9.3
mkdir /mnt/nethome mount t nfs
192.168.0.10/home /mnt/nethome
umount /mnt/nethome
20
To see what is mounted on client side

• Using the df command show disk usage
• The mount command is most detailed about mount
  options
• The showmount shows all exported shares on a
  remote server plus all mounts from client
• Client nfsstat will show statistics

df F NFS Filesystem 1k-blocks
Used Available Use Mounted on 192.168.0.10/insta
ll/suse9.3 79366688 58235488 21131200 74
/mnt/a
mount grep nfs 192.168.0.10/install/suse9.3
on /mnt/a type nfs (rw,addr192.168.0.10)
showmount -a 192.168.1.60 All mount points on
192.168.1.60 ,192.168.1.0/24/home /home /in
stall/suse9.3 192.168.0.2
nfsstat c Client rpc stats calls retrans
authrefrsh 129 0 0
21
mount o ltoptionsgt t nfs

• NFS clients access network shared directories
  using the mount command
• NFS mount o options
• rw/ro read-write (default) or read-only
• hard retry mount operation until server responds
  (default) or
• soft try mount once and allow to timeout
• retrans transmission and timeout parameters for
  soft mounted operationstimeout
• bg after first mount failure, retry mount in the
  background
• intr allow operations on filesystems to be
  interrupted with kill signals
• nfsversn The version of NFS the mount command
  should attempt to use
• Use /etc/fstab to make NFS mounts permanent
• a02/tmp  /mnt/nethome   nfs    soft,intr,nfsver
  s3      0      0
• Manually mounting /tmp as /mnt/nethome on local
  host from a02

hostname a01 mount o rw,soft -t nfs a02/tmp
/mnt/nethome
22
Mount nfs-shares at boot in client

• Make entries in /etc/fstab
• Some /etc/fstab mount options
• Mount all unmounted
• If you made changes on live system in fstab, you
  can mount all unmounted filesystem with

/etc/fstab Directory
MountPoint Type   Options   Dump  
FSCK 192.168.0.10/data/files  /mnt/nfs   nfs    s
oft,nfsvers3     0      0
auto mount this when mount a is used defaults
(rw suid dev exec auto nouser async) user allow
regular users to mount/umount sync use syncron
I/O most safe soft skip mount if server not
responding hard try until server
responds retryminutes bg/fg retry mounting in
background or foreground
mount a
23
Possible NFS Mount options
24
Exercise - Using mount with NFS

• What command will mount /usr/share from mash4077
  on the local mount point /usr/share?
• How do I check what filesystems are mounted
  locally?
• Make a static mount in a01 /mnt/nethome of
  exported a02/tmp in /etc/fstab
• Manually mount exported a02/usr/share as read
  only on a01
• How can I show what is nfs exported on the server

25
NFS security

• NFS is inherently insecure
• NFS can be run in encrypted mode which encrypts
  data over the network
• AFS more appropriate for security conscious sites
• User IDs must be co-ordinated across all
  platforms
• UIDs and not user names are used to control file
  access (use LDAP or NIS)
• mismatched user id's cause access and security
  problems
• Fortunately root access is denied by default
• over NFS root is mapped to user nobody

mount grep "/share" mail/share on /share
id uid318(hawkeye) gid318(hawkeye) touch
/share/hawkeye ssh mail ls -l
/share/hawkeye -rwxr-xr-x 2 soonlee sonlee 0
Jan 11 1121 /share/hawkeye
26
NFS Hanging

• Run NFS on a reliable network
• Avoid having NFS servers that NFS mount each
  other's filesystems or directories
• Always use the sync option whenever possible
• Mission critical computers shouldn't rely on an
  NFS server to operate
• Dont have NFS shares in search path

27
NFS Hanging continued

• File Locking
• Known issues exist, test your applications
  carefullý
• Nesting Exports
• NFS doesn't allow you to export directories that
  are subdirectories of directories that have
  already been exported unless they are on
  different partitions.
• Limiting "root" Access
• no_root_squash
• Restricting Access to the NFS server
• You can add user named "nfsuser" on the NFS
  client to let this user squash access for all
  other users on that client
• Use nfsV3 if possible

28
NFS Firewall considerations

• NFS uses many ports
• RPC uses TCP port 111
• NFS server itself uses port 2049
• MOUNTD listens on neogated UDP/TCP ports
• NLOCKMGR listens on neogated UDP / TCP ports
• Expect almost any TCP/UDP port over 1023 can be
  allocated for NFS
• NFS need a STATEFUL firewall
• A stateful firewall will be able dealing with
  traffic that originates from inside a network and
  block traffic from outside
• SPI can demolish NFS
• Stateful packet inspection on cheaper
  routers/firewalls can missinteprete NFS traffic
  as DOS attacks and start drop packages
• NFSSHELL
• This is a hacker tool, it can hack some NFS
• Invented by Leendert van Doom
• Use VPN and IPSEC tunnels
• With complex services like NFS IPSEC or some kind
  of VPN should be considered if used in untrusted
  networks.

29
Common NFS error messages
30
NFS Automounter for clients or servers

• Automatically mount directories from server when
  needed
• To activate automount manually and at boot
• Management of shares centralized on server
• Increases security and reduces lockup problems
  with static shares
• Main configuration sit in /etc/auto.master
• Simple format is MOUNT-KEY MOUNT-OPTIONS LOCATION
  
• MOUNT-KEY is local mountpoint, here /doc, /-
  (from root) and /home
• MOUNT-OPTIONS is the standard mount options
  previously described, here -ro
• LOCATION can be a direct share on a server like
  server and map file auto.direct and indirect like
  /etc/auto.home.
• Common configuration /etc/auto.misc is for
  floppy/cd/dvd.
• Centralized administration need to set
  /etc/nsswitch.conf

rcautofs start
insserv autofs
/doc -ro server/usr/doc /- /etc/auto.direct /h
ome /etc/auto.home
automount files nis ldap
31
Direct And Indirect Map Files structure

• File /etc/auto.master sets the mandatory
  automount config
• map files always try to mount in auto.master
  mount key
• Direct map file /etc/auto.direct
• Direct Maps are used to define NFS filesystems
  that are mounted on different servers or that
  all don't start with the same prefix.
• Indirect map file /etc/auto.home
• Indirect Maps define directories that can be
  mounted under the same mount point. Like users
  home directories.

/data/sales -rw server/disk1/data/sales
/sql/database -ro,soft snail/var/mysql/database
peter server/home/peter kalle akvarius/home/b
ob walker iss/home/bunny
32
Wildcards In Map Files

• Wildcards In Map Files
• The asterisk (), which means all
• the ampersand (), which instructs automounter to
  substitute the value of the key for the
  character.
• Using the Ampersand Wildcard /etc/auto.home
• the key is peter, so the ampersand wildcard is
  interpreted to mean peter too. This means you'll
  be mounting the server/home/peter directory.
• Using the Asterisk Wildcard /etc/auto.home
• In the example below, the key is , meaning that
  automounter will attempt to mount any attempt to
  enter the /home directory. But what's the value
  of the ampersand? It is actually assigned the
  value of the key that triggered the access to the
  /etc/auto.home file. If the access was for
  /home/peter, then the ampersand is interpreted to
  mean peter, and server/home/peter is mounted. If
  access was for /home/kalle, then
  akvarius/home/kalle would be mounted.

peter server/home/
bigboy/home/
33
Other DFS Systems

• RFS Remote File Sharing
• developed by ATT to address problems with NFS
• stateful system supporting Unix filesystem
  semantics
• uses same SVR4 commands as NFS, just use rfs as
  file type
• standard in SVR4 but not found in many other
  systems
• AFS Andrew Filesystem
• developed as a research project at
  Carnegie-Mellon University
• now distributed by a third party (Transarc
  Corporation)
• available for most Unix platforms and PCs running
  DOS, OS/2, Windows
• uses its own set of commands
• remote systems access through a common interface
  (the /afs directory)
• supports local data caching and enhanced security
  using Kerberos
• fast gaining popularity in the Unix community

34
Summary

• Unix supports file sharing across a network
• NFS is the most popular system and allows Unix to
  share files with other O/S
• Servers share directories across the network
  using the share command
• Permanent shared drives can be configured into
  /etc/fstab
• Clients use mount to access shared drives
• Use mount and exportfs to look at distributed
  files/catalogs