Outline

1
Outline

• Announcements
• File Management
• Structured files
• Low-level file implementations
• Directories
• File Systems

2
Announcements

• Final exam will be cumulative
• The split between the before-midterm and after
  will be around 30 (before-midterm) and 70
  (after-midterm)
• Lab3
• I made some of the codes I have available to you
• http//www.cs.fsu.edu/liux/courses/cop4610/assign
  ments/lab3_code
• or on program or linprog at liux/public_html/cour
  ses/cop4610/assignments/lab3_code
• A demo program that is only implemented the
  functionality partially
• liux/public_html/courses/cop4610/assignments/lab3
  on on program or linprog

3
Announcements cont.

• If you have not signed to demonstrate your lab2,
  please do so
• If none of the available slots works for you,
  please contact Yong Chen for an appointment
• You have to do the demonstration on or before
  November 26, 2003
• After that, it will be graded solely based on
  source code and test results you turned in and
  the worst cases will be assumed if there are any
  questions

4
Announcements cont.

• Scheduling for the remaining semester
• Homework 5 will be due on Nov. 25, 2003
• Lab3 will be due on Dec. 3, 2003
• Quiz 3 will be on Nov. 25, 2003 (About deadlocks
  and memory management)
• Finish file system today
• Finish memory management in three lectures
• One lecture on protection and security
• Dec. 2 will cover some advanced topics in
  distributed operating systems
• Dec. 4 will have a final review

5
Announcements cont.

• There will be no class on Nov. 11 and no
  recitation on Nov. 12
• There will be no class on Nov. 27 (thanksgiving)
  and no recitation class on Nov. 26
• I plan also NOT to have the recitation class on
  Nov. 19
• If there is a need, we will have

6
Operating System Components
7
Why Programmers Need Files
8
File system context
9
Fig 13-2 The External View of the File Manager
Application Program
mount()
write()
open()
close()
read()
lseek()
File Mgr
Device Mgr
Memory Mgr
File Mgr
Device Mgr
Memory Mgr
Process Mgr
Process Mgr
UNIX
Windows
Hardware
10
Levels in a file system
11
Information Structure
12
Logical structures in a file
13
Low-Level Files
14
File systems

• File system
• A data structure on a disk that holds files
• actually a file system is in a disk partition
• a technical term different from a file system
  as the part of the OS that implements files
• File systems in different OSs have different
  internal structures

15
A file system layout
16
File system descriptor

• The data structure that defines the file system
• Typical fields
• size of the file system (in blocks)
• size of the file descriptor area
• first block in the free block list
• location of the file descriptor of the root
  directory of the file system
• times the file system was created, last modified,
  and last used

17
File system layout variations

• MS/DOS uses a FAT (file allocation table) file
  system
• so does the Macintosh OS (although the MacOS
  layout is different)
• New UNIX file systems use cylinder groups
  (mini-file systems) to achieve better locality of
  file data

18
Locating file data

• The logical file is divided into logical blocks
• Each logical block is mapped to a physical disk
  block
• The file descriptor contains data on how to
  perform this mapping
• there are many methods for performing this
  mapping
• we will look at several of them

19
Dividing a file into blocks
20
Contiguous Allocation

• Each file occupies a set of contiguous blocks on
  the disk
• Simple only starting location and length are
  required
• Random access
• Wasteful of space (dynamic storage-allocation
  problem)
• Files cannot grow
• Mapping from logical to physical
• Block to be accessed Q starting address
• Displacement into block R

21
A contiguous file
22
A contiguous file cont.
23
Allocation Strategies

• Best fit
• Chooses the minimum contiguous block that is
  large enough
• First fit
• Chooses the contiguous block that is large enough
• Worst fit
• Chooses the maximum contiguous block that is
  large enough

24
Keeping a file in pieces

• We need a block pointer for each logical block,
  an array of block pointers
• block mapping indexes into this array
• Each file is a linked list of disk blocks
• But where do we keep this array?
• usually it is not kept as contiguous array
• the array of disk pointers is like a second
  related file (that is 1/1024 as big)

25
Block pointers in the file descriptor
26
Block pointers in contiguous disk blocks
27
Block pointers in the blocks
28
Block pointers in the blocks cont.
29
Block pointers in an index block
30
Block pointers in an index block cont.
31
Chained index blocks
32
Two-level index blocks
33
Two-level index blocks cont.
34
UNIX Hybrid Method
inode
35
Inverted disk block index (FAT)
36
DOS FAT Files
37
Free-Space Management

• Bit vector (n blocks)

0
1
2
n-1

1 ? blocki free 0 ? blocki occupied
biti
???

• First free block number

(number of bits per word) (number of 0-value
words) offset of first 1 bit
38
Free-Space Management - cont.

• Bit map requires extra space. Example
• block size 212 bytes
• disk size 230 bytes (1 gigabyte)
• n 230/212 218 bits (or 32K bytes)
• Easy to get contiguous files
• Linked list (free list)
• Cannot get contiguous space easily
• No waste of space

39
Free list organization
40
Free-Space Management - cont.

• Need to protect
• Pointer to free list
• Bit map
• Must be kept on disk
• Copy in memory and disk may differ.
• Cannot allow for blocki to have a situation
  where biti 0 in memory and biti 1 on
  disk.
• Solution
• Set biti 0 in disk.
• Allocate blocki
• Set biti 0 in memory

41
Implementing Low Level Files

• Secondary storage device contains
• Volume directory (sometimes a root directory for
  a file system)
• External file descriptor for each file
• The file contents
• Manages blocks
• Assigns blocks to files (descriptor keeps track)
• Keeps track of available blocks
• Maps to/from byte stream

42
Disk Organization
Boot Sector
Volume Directory
Blk0
Blk1
Blkk-1

Track 0, Cylinder 0

Blkk
Blkk1
Blk2k-1
Track 0, Cylinder 1


Blk
Blk
Blk
Track 1, Cylinder 0


Blk
Blk
Blk
Track N-1, Cylinder 0


Blk
Blk
Blk
Track N-1, Cylinder M-1
43
Low-level File System Architecture
Block 0


b0 b1 b2 b3
bn-1
.
.
.
Randomly Accessed Device
Sequential Device
44
File Descriptors

• External name
• Current state
• Sharable
• Owner
• User
• Locks
• Protection settings
• Length
• Time of creation
• Time of last modification
• Time of last access
• Reference count
• Storage device details

45
An open() Operation

• Locate the on-device (external) file descriptor
• Extract info needed to read/write file
• Authenticate that process can access the file
• Create an internal file descriptor in primary
  memory
• Create an entry in a per process open file
  status table
• Allocate resources, e.g., buffers, to support
  file usage

46
File Manager Data Structures
Keep the state of the process-file session
2
Copy info from external to the open file
descriptor
1
Open File Descriptor
Process-File Session
Return a reference to the data structure
3
External File Descriptor
47
Opening a UNIX File
48
Reading and Writing the Byte Stream

• Two stages
• Reading bytes into or writing bytes out of the
  memory copy of the block
• Reading the physical blocks into or writing them
  out of memory from/to storage devices
• Packing or unmarshalling procedure converts
  secondary storage blocks into a byte stream
• Unpacking or marshalling procedure converts a
  byte stream into blocks

49
Marshalling the Byte Stream

• Must read at least one buffer ahead on input
• Must write at least one buffer behind on output
• Seek ? flushing the current buffer and finding
  the correct one to load into memory
• Inserting/deleting bytes in the interior of the
  stream

50
File Block Buffering

• Storage devices use block I/O and files place an
  explicit order on the bytes
• Therefore, it is possible to predict what is
  likely to be read after a byte
• When file is opened, manager reads as many blocks
  ahead as feasible
• After a block is logically written, it is queued
  for writing behind, whenever the disk is
  available
• Buffering has an enormous effect on the overall
  performance of the system
• Buffer pool usually variably sized, depending
  on virtual memory needs
• Interaction with the device manager and memory
  manager

51
Supporting Other Storage Abstractions

• Low-level file systems avoid encoding
  record-level functionality
• If applications use very large or very small
  records, a generic file manager may not be
  efficient
• Some operating systems provide a higher-layer
  file system to support applications with large or
  small files
• Database management systems and multimedia
  documents are examples

52
Structured Files
53
Record-Oriented Sequential Files
54
Electronic Mail Example
55
Indexed Sequential Files
56
Database Management Systems

• A database is a very highly structured set of
  information
• Stored across different files
• Optimized to minimize access time
• DBMSs implementation
• Some DBMSs use the normal files provided by the
  OS for generic use
• Some use their own storage device block

57
Disk compaction
58
Memory-mapped Files

• A files contents are mapped directly into the
  virtual address space
• Files can be read from or written to by
  referencing the corresponding virtual addresses
• Memory-mapped files are very useful when a file
  is shared or accessed repeatedly

59
Memory-mapped Files cont.
60
Directories

• A directory is a set of logically associated
  files and other directories of files
• Directories are the mechanism we use to organize
  files
• The file manager provides a set of commands to
  manage directories
• Traverse a directory
• Enumerate a list of all files and nested
  directories

61
Directory Structures

• How should files be organized within directory?
• Flat name space
• All files appear in a single directory
• Hierarchical name space
• Directory contains files and subdirectories
• Each file/directory appears as an entry in
  exactly one other directory -- a tree
• Popular variant All directories form a tree,
  but a file can have multiple parents.

62
Directory Structures
63
Directory Structures cont.
64
A directory tree
65
Directory Implementation

• Device Directory
• A device can contain a collection of files
• Easier to manage if there is a root for every
  file on the device -- the device root directory
• File Directory
• Typical implementations have directories
  implemented as a file with a special format
• Entries in a file directory are handles for other
  files (which can be files or subdirectories)

66
Directory Implementation

• Linear list of file names with pointer to the
  data blocks.
• simple to program
• time-consuming to execute
• Hash Table linear list with hash data
  structure.
• decreases directory search time
• collisions situations where two file names hash
  to the same location
• fixed size

67
Mounting file systems

• Each file system has a root directory
• We can combine file systems by mounting
• that is, link a directory in one file system to
  the root directory of another file system
• This allows us to build a single tree out of
  several file systems
• This can also be done across a network, mounting
  file systems on other machines

68
UNIX mount Command
69
Mounting a file system
70
VFS-based File Manager
71
NFS Architecture
72
Summary

• File storage methods
• Contiguous files
• File pointers in the file descriptor
• Contiguous file pointers
• Chained data blocks
• Chained single index blocks
• Double index blocks
• Triple index blocks
• Hybrid solutions
• Directories
• File Systems