Distributed File Systems: RPC, NFS, and AFS

1
Distributed File SystemsRPC, NFS, and AFS
2
Announements

• Homework 6 available later tonight
• Due next Tuesday, December 2nd
• See me after class to pick up prelim
• Upcoming Agenda
• No class on ThursdayHappy Thanksgiving!
• Next week last week of classesDecember 2nd and
  4th
• FinalThursday, December 18th at 2pm
• Room 131 Warren Hall
• Length is 2hrs

3
Goals for Today

• Distributed file systems (DFS)
• Network file system (NFS)
• Remote Procedure Calls (RPC)
• Andrew file system (AFS)

4
Distributed File Systems (DFS)
5
Distributed File Systems

• Goal view a distributed system as a file system
• Storage is distributed
• Web tries to make world a collection of
  hyperlinked documents
• Issues not common to usual file systems
• Naming transparency
• Load balancing
• Scalability
• Location and network transparency
• Fault tolerance
• We will look at some of these today

6
Transfer Model

• Upload/download Model
• Client downloads file, works on it, and writes it
  back on server
• Simple and good performance
• Remote Access Model
• File only on server client sends commands to get
  work done

7
Naming transparency

• Naming is a mapping from logical to physical
  objects
• Ideally client interface should be transparent
• Not distinguish between remote and local files
• /machine/path or mounting remote FS in local
  hierarchy are not transparent
• A transparent DFS hides the location of files in
  system
• 2 forms of transparency
• Location transparency path gives no hint of file
  location
• /server1/dir1/dir2/x tells x is on server1, but
  not where server1 is
• Location independence move files without
  changing names
• Separate naming hierarchy from storage devices
  hierarchy

8
File Sharing Semantics

• Sequential consistency reads see previous writes
• Ordering on all system calls seen by all
  processors
• Maintained in single processor systems
• Can be achieved in DFS with one file server and
  no caching

9
Caching

• Keep repeatedly accessed blocks in cache
• Improves performance of further accesses
• How it works
• If needed block not in cache, it is fetched and
  cached
• Accesses performed on local copy
• One master file copy on server, other copies
  distributed in DFS
• Cache consistency problem how to keep cached
  copy consistent with master file copy
• Where to cache?
• Disk Pros more reliable, data present locally
  on recovery
• Memory Pros diskless workstations, quicker data
  access,
• Servers maintain cache in memory

10
File Sharing Semantics

• Other approaches
• Write through caches
• immediately propagate changes in cache files to
  server
• Reliable but poor performance
• Delayed write
• Writes are not propagated immediately, probably
  on file close
• Session semantics (AFS) write file back on close
• Alternative (NFS) scan cache periodically and
  flush modified blocks
• Better performance but poor reliability
• File Locking
• The upload/download model locks a downloaded file
• Other processes wait for file lock to be released

11
Network File System (NFS)
12
Network File System (NFS)

• Developed by Sun Microsystems in 1984
• Used to join FSes on multiple computers as one
  logical whole
• Used commonly today with UNIX systems
• Assumptions
• Allows arbitrary collection of users to share a
  file system
• Clients and servers might be on different LANs
• Machines can be clients and servers at the same
  time
• Architecture
• A server exports one or more of its directories
  to remote clients
• Clients access exported directories by mounting
  them
• The contents are then accessed as if they were
  local

13
Example
14
NFS Mount Protocol

• Client sends path name to server with request to
  mount
• Not required to specify where to mount
• If path is legal and exported, server returns
  file handle
• Contains FS type, disk, i-node number of
  directory, security info
• Subsequent accesses from client use file handle
• Mount can be either at boot or automount
• Using automount, directories are not mounted
  during boot
• OS sends a message to servers on first remote
  file access
• Automount is helpful since remote dir might not
  be used at all
• Mount only affects the client view!

15
NFS Protocol

• Supports directory and file access via remote
  procedure calls (RPCs)
• All UNIX system calls supported other than open
  close
• Open and close are intentionally not supported
• For a read, client sends lookup message to server
• Server looks up file and returns handle
• Unlike open, lookup does not copy info in
  internal system tables
• Subsequently, read contains file handle, offset
  and num bytes
• Each message is self-contained
• Pros server is stateless, i.e. no state about
  open files
• Cons Locking is difficult, no concurrency control

16
NFS Implementation

• Three main layers
• System call layer
• Handles calls like open, read and close
• Virtual File System Layer
• Maintains table with one entry (v-node) for each
  open file
• v-nodes indicate if file is local or remote
• If remote it has enough info to access them
• For local files, FS and i-node are recorded
• NFS Service Layer
• This lowest layer implements the NFS protocol

17
NFS Layer Structure
18
How NFS works?

• Mount
• Sys ad calls mount program with remote dir, local
  dir
• Mount program parses for name of NFS server
• Contacts server asking for file handle for remote
  dir
• If directory exists for remote mounting, server
  returns handle
• Client kernel constructs v-node for remote dir
• Asks NFS client code to construct r-node for file
  handle
• Open
• Kernel realizes that file is on remotely mounted
  directory
• Finds r-node in v-node for the directory
• NFS client code then opens file, enters r-node
  for file in VFS, and returns file descriptor for
  remote node

19
Cache coherency

• Clients cache file attributes and data
• If two clients cache the same data, cache
  coherency is lost
• Solutions
• Each cache block has a timer (3 sec for data, 30
  sec for dir)
• Entry is discarded when timer expires
• On open of cached file, its last modify time on
  server is checked
• If cached copy is old, it is discarded
• Every 30 sec, cache time expires
• All dirty blocks are written back to the server

20
Remote Procedure Call (RPC)
21
Procedure Call

• More natural way is to communicate using
  procedure calls
• every language supports it
• semantics are well defined and understood
• natural for programmers to use
• Basic idea define server as a module that
  exports a set of procedures callable by client
  programs.
• To use the server, the client just does a
  procedure call, as if it were linked with the
  server

call
Client
Server
return
22
(Remote) Procedure Call

• So, we would like to use procedure call as a
  model for distributed communication.
• Lots of issues
• how do we make this invisible to the programmer?
• what are the semantics of parameter passing?
• how is binding done (locating the server)?
• how do we support heterogeneity (OS, arch.,
  language)
• etc.

23
Remote Procedure Call

• The basic model for Remote Procedure Call (RPC)
  was described by Birrell and Nelson in 1980,
  based on work done at Xerox PARC.
• Goal to make RPC as much like local PC as
  possible.
• Used computer/language support.
• There are 3 components on each side
• a user program (client or server)
• a set of stub procedures
• RPC runtime support

24
RPC

• Basic process for building a server
• Server program defines the servers interface
  using an interface definition language (IDL)
• The IDL specifies the names, parameters, and
  types for all client-callable server procedures
• A stub compiler reads the IDL and produces two
  stub procedures for each server procedure a
  client-side stub and a server-side stub
• The server writer writes the server and links it
  with the server-side stubs the client writes
  her program and links it with the client-side
  stubs.
• The stubs are responsible for managing all
  details of the remote communication between
  client and server.

25
RPC Stubs

• Client-side stub is a procedure that looks to the
  client as if it were a callable server procedure.
• Server-side stub looks like a calling client to
  the server
• The client program thinks it is calling the
  server
• in fact, its calling the client stub.
• The server program thinks its called by the
  client
• in fact, its called by the server stub.
• The stubs send messages to each other to make RPC
  happen.

26
RPC Call Structure
proc foo(a,b) begin foo... end foo
client program
client makes local call to stub proc.
server is called by its stub
server program
call foo(x,y)
call foo
call foo
stub unpacks params and makes call
proc foo(a,b)
call foo(x,y)
client stub
stub builds msg packet, inserts params
server stub
send msg
msg received
runtime sends msg to remote node
runtime receives msg and calls stub
RPC runtime
RPC runtime
Call
27
RPC Return Structure
proc foo(a,b) begin foo... end foo
client program
server program
server proc returns
call foo(x,y)
client continues
return
return
stub builds result msg with output args
proc foo(a,b)
call foo(x,y)
client stub
stub unpacks msg, returns to caller
server stub
msg received
send msg
runtime responds to original msg
runtime receives msg, calls stub
RPC runtime
RPC runtime
return
28
RPC Information Flow
Client (caller)
RPC Runtime
unmarshal ret vals
mbox2
Machine A
Machine B
marshal return values
mbox1
Server (callee)
RPC Runtime
29
RPC Binding

• Binding is the process of connecting the client
  and server
• The server, when it starts up, exports its
  interface,
• identifying itself to a network name server and
• telling the local runtime its dispatcher address.
• The client, before issuing any calls, imports the
  server,
• which causes the RPC runtime to lookup the server
  through the name service and
• contact the requested server to setup a
  connection.
• The import and export are explicit calls in the
  code.

30
RPC Marshalling

• Marshalling is packing of procedure params into
  message packet.
• RPC stubs call type-specific procedures to
  marshall (or unmarshall) all of the parameters to
  the call.
• On client side, client stub marshalls parameters
  into call packet
• On the server side the server stub unmarshalls
  the parameters to call the servers procedure.
• On return, server stub marshalls return
  parameters into return packet
• Client stub unmarshalls return params and returns
  to the client.

31
Problems with RPC

• Non-Atomic failures
• Different failure modes in distributed system
  than on a single machine
• Consider many different types of failures
• User-level bug causes address space to crash
• Machine failure, kernel bug causes all processes
  on same machine to fail
• Some machine is compromised by malicious party
• Before RPC whole system would crash/die
• After RPC One machine crashes/compromised while
  others keep working
• Can easily result in inconsistent view of the
  world
• Did my cached data get written back or not?
• Did server do what I requested or not?
• Answer? Distributed transactions/Byzantine Commit
• Performance
• Cost of Procedure call same-machine RPC
  network RPC
• Means programmers must be aware that RPC is not
  free
• Caching can help, but may make failure handling
  complex

32
Cross-Domain Comm./Location Transparency

• How do address spaces communicate with one
  another?
• Shared Memory with Semaphores, monitors, etc
• File System
• Pipes (1-way communication)
• Remote procedure call (2-way communication)
• RPCs can be used to communicate between address
  spaces on different machines or the same machine
• Services can be run wherever its most
  appropriate
• Access to local and remote services looks the
  same
• Examples of modern RPC systems
• CORBA (Common Object Request Broker Architecture)
• DCOM (Distributed COM)
• RMI (Java Remote Method Invocation)

33
Microkernel operating systems

• Example split kernel into application-level
  servers.
• File system looks remote, even though on same
  machine
• Why split the OS into separate domains?
• Fault isolation bugs are more isolated (build a
  firewall)
• Enforces modularity allows incremental upgrades
  of pieces of software (client or server)
• Location transparent service can be local or
  remote
• For example in the X windowing system Each X
  client can be on a separate machine from X
  server Neither has to run on the machine with
  the frame buffer.

34
Andrew File System (AFS)
35
Andrew File System (AFS)

• Named after Andrew Carnegie and Andrew Mellon
• Transarc Corp. and then IBM took development of
  AFS
• In 2000 IBM made OpenAFS available as open source
• Features
• Uniform name space
• Location independent file sharing
• Client side caching with cache consistency
• Secure authentication via Kerberos
• Server-side caching in form of replicas
• High availability through automatic switchover of
  replicas
• Scalability to span 5000 workstations

36
AFS Overview

• Based on the upload/download model
• Clients download and cache files
• Server keeps track of clients that cache the file
• Clients upload files at end of session
• Whole file caching is central idea behind AFS
• Later amended to block operations
• Simple, effective
• AFS servers are stateful
• Keep track of clients that have cached files
• Recall files that have been modified

37
AFS Details

• Has dedicated server machines
• Clients have partitioned name space
• Local name space and shared name space
• Cluster of dedicated servers (Vice) present
  shared name space
• Clients run Virtue protocol to communicate with
  Vice
• Clients and servers are grouped into clusters
• Clusters connected through the WAN
• Other issues
• Scalability, client mobility, security,
  protection, heterogeneity

38
AFS Shared Name Space

• AFSs storage is arranged in volumes
• Usually associated with files of a particular
  client
• AFS dir entry maps vice files/dirs to a 96-bit
  fid
• Volume number
• Vnode number index into i-node array of a volume
• Uniquifier allows reuse of vnode numbers
• Fids are location transparent
• File movements do not invalidate fids
• Location information kept in volume-location
  database
• Volumes migrated to balance available disk space,
  utilization
• Volume movement is atomic operation aborted on
  server crash

39
AFS Operations and Consistency

• AFS caches entire files from servers
• Client interacts with servers only during open
  and close
• OS on client intercepts calls, and passes it to
  Venus
• Venus is a client process that caches files from
  servers
• Venus contacts Vice only on open and close
• Does not contact if file is already in the cache,
  and not invalidated
• Reads and writes bypass Venus
• Works due to callback
• Server updates state to record caching
• Server notifies client before allowing another
  client to modify
• Clients lose their callback when someone writes
  the file
• Venus caches dirs and symbolic links for path
  translation

40
AFS Implementation

• Client cache is a local directory on UNIX FS
• Venus and server processes access file directly
  by UNIX i-node
• Venus has 2 caches, one for status one for data
• Uses LRU to keep them bounded in size

41
Summary

• RPC
• Call procedure on remote machine
• Provides same interface as procedure
• Automatic packing and unpacking of arguments
  without user programming (in stub)
• NFS
• Simple distributed file system protocol. No
  open/close
• Stateless server
• Has problems with cache consistency, locking
  protocol
• AFS
• More complicated distributed file system protocol
• Stateful server
• session semantics consistency on close

42
Prelim II

• Prelims graded
• Mean 73 (Median 76), Stddev 13.3, High 98 out of
  100!
• Good job!
• Re-grade policy
• Submit written re-grade request to Nazrul.
• Entire prelim will be re-graded.
• We were generous the first time
• If still unhappy, submit another re-grade
  request.
• Nazrul will re-grade herself
• If still unhappy, submit a third re-grade
  request.
• I will re-grade. Final grade is law.

43
Grade distribution
44
Question 3

• Hardlinks
• Need a count in inode/fileheader
• Remove decrements count and removes file if count
  is 0
• New syscall Link(char src, char dst)
• Softlinks
• Need to file type indicated in inode/fileheader
• Also, need path to target file
• Open needs to change to perform recursive lookup
• New syscall SymLink(char src, char dst)

45
Question 4

• Concurrent writers
• RAID 0, 10, 5, 6
• Not 1 or 4 because cannot perform independent
  writes
• Concurrent readers (but not concurrent writers)
• RAID 1 and 4
• 2k-1 disks and want concurrent readers
• Unavailable RAID 1 requires 2k disks
• Undesirable RAID 4, 5, and 6 requires complex
  controllers

46
Happy Thanksgiving!!!