CS 241 Section Week

1
CS 241 Section Week 11(11/06/08)
2
Outline

• LMP1 Overview
• Files Overview
• File I/O
• UNIX File Systems
• inodes
• Directories
• UNIX File Operations
• Directory
• File Status
• Links
• UNIX File Systems vs. Windows File Systems

3
LMP1 Overview
4
LMP1 Overview

• LMP1 attempts to encode or decode a number of
  files the following way
• encode gt ./mmap -e -b16 file1 file2 ...
• decode gt ./mmap -d -b8 file1 file2 ...
• It has the following parameters
• It reads whether it has to encode (-e) or
  decode(-d)
• the number of bytes (rw_units) for each
  read/write from the file

5
LMP1 Overview

• You have TWO weeks to complete and submit LMP1.
  We have divided LMP1 into two stages
• Stage 1
• Implement a simple virtual memory.
• It is recommended you implement the my_mmap()
  function during this week.
• You will need to complete various data structures
  to deal with the file mapping table, the page
  table, the physical memory, etc.

6
LMP1 Overview

• You have TWO weeks to complete and submit LMP1.
  We have divided LMP1 into two stages
• Stage 2
• Implement various functions for memory mapped
  files including
• my_mread() , my_mwrite() and my_munmap()
• Handle page faults in your my_mread() and
  my_mwrite() functions
• Implement two simple manipulations on files
• encoding
• decoding

7
Files Overview
8
Files Overview

• How do you get data in/out of a program?

9
Files Overview

• How do you get data in/out of a program?
• Unix solution
• Files.

10
Files Overview

• How do you get data in/out of a program?
• Unix solution
• Files.
• Many different kinds of I/O are viewed, to some
  degree, as accessing a file stream
• stdin, stdout, stderr, /dev/audio,

11
Files Overview

• How do you get data in/out of a program?
• Unix solution
• Files.
• Many different kinds of I/O are viewed, to some
  degree, as accessing a file stream
• stdin, stdout, stderr, /dev/audio,
• pipes (cat file grep text), network socket, ...

12
Unix file system tree

• /

13
Unix file system tree

• / /bin/

14
Unix file system tree

• / /bin/ /home/

15
Unix file system tree

• / /bin/ /home/
• /home/someuser/

16
Unix file system tree

• / /bin/ /home/
• /home/someuser/
• /home/someuser/somefile.txt

17
Unix file system tree

• / /bin/ /home/
• /home/someuser/
• /home/someuser/somefile.txt /usr/
• /usr/bin/
• /usr/lib/

18
Files

• A file is structured as a sequence of bytes,
  modeled after the notion of a tape

19
Files

• A file is structured as a sequence of bytes,
  modeled after the notion of a tape
• Text contents of some file go here.

start of file
file offset (similar to tape head)
20
Files

• A file is structured as a sequence of bytes,
  modeled after the notion of a tape
• Text contents of some file go here.

start of file
file offset (similar to tape head)
21
Internal File Structure

• A file is just a series of bytes

T
h
i
s
w
n
i
x
f
i
l
e
.
22
Internal File Structure

• A file is just a series of bytes

T
h
i
s
w
n
i
x
f
i
l
e
.
Start of File
End of File
Current Offset
23
I/O Libraries in C
stdio fopen, fread, fwrite, fclose,
... (buffered I/O)
User Process
open, read, write, close, select, poll,
... (direct to kernel I/O)
kernel system call handler
Kernel
file I/O
terminal I/O
pipe I/O
network I/O
audio I/O
24
I/O libraries in C
User process
(buffered I/O)

stdio
fopen,
fread,
fwrite,
fclose,
(direct to kernel I/O)
open,
read,
write,
close,

kernel system call handler
file I/O
terminal I/O
pipe I/O
network I/O
audio I/O
Kernel
25
File I/O
26
Buffered I/O stdio.h

• We've previously used stdio.h lightly
• printf(3)
• fprintf(3)
• etc.

27
Buffered I/O stdio.h

• We've previously used stdio.h lightly
• printf(3)
• fprintf(3)
• etc.
• The stdio functions performing buffering
• Read a large chunk from a file

28
Buffered I/O stdio.h

• We've previously used stdio.h lightly
• printf(3)
• fprintf(3)
• etc.
• The stdio functions performing buffering
• Read a large chunk from a file
• When the user requests a few bytes, just read
  them out of the buffer lower overhead

29
Buffered I/O Advantages

• Weve previously used
• printf()
• fprintf()
• In MP5, you had to call fflush() to flush the
  buffer
• fflush(stdout)

30
Buffered I/O Advantages

• The reason you needed to was the bytes were
  buffered until there was one of two conditions
• Enough time had passed to write the output
• Enough output had been received to write the
  output

31
Buffered I/O Advantages

• Why use buffers?
• I/O operations are SLOW!
• Every time you write just one byte, you dont
  want to have to access your hard drive.

32
open()

• int open(const char pathname, int
  flags)

33
open()

• int open(const char pathname, int
  flags)
• open() returns an int, which is your file
  descriptor

34
open()

• int open(const char pathname, int
  flags)
• open() takes in either a relative or full path
  name
• ../../../../../usr/importantfile.txt

35
open()

• int open(const char pathname, int
  flags)
• Various flag options to allow a file to only be
  appended to (O_APPEND), opened as write only
  (O_WRONLY), and more.

36
open()

• To open a file for reading
• int ifd open(./input.txt,
  O_RDONLY)
• To open OR create a file for writing, with given
  permissions
• int ofd open(output.txt,
  O_WRONLY O_CREAT, S_IRUSR
  S_IWUSR)

37
fopen()

• FILE fopen( const char filename,
  const char mode)
• Rather than an int (file descriptor), fopen
  returns a FILE stream.

38
File Permissions

• In UNIX, the file permissions system is
  relatively basic.
• Each file has a single owner and a single group
  associated with it.
• Each file also has permissions associated with
  itself for the owner, members of the group the
  file is in, and for everyone else.

39
File Permissions

• These permissions are stored as a
  three-octal-digit number (000 to 777).

7
5
5
40
File Permissions

• The most-significant number is the owners
  permission.

Owner
7
5
5
41
File Permissions

• The middle number is the groups permission.

Group
7
5
5
42
File Permissions

• The least-significant number is everyone elses
  permission.

Other
7
5
5
43
File Permissions

• Each octal number is simply three bits a read
  bit, a write bit, and an execute bit.

7
5
5
Read
1
1
1
Write
1
0
0
Execute
1
1
1
44
File Permissions

• Thus
• 755 means everyone can read and execute by file,
  but only the owner can write to (edit) my file
• 644 means everyone can read my file, only the
  owner can write to my file, and no one can
  execute it
• 660 means only members of the files group and
  the files owner may read or edit the file
  others cannot even read it

45
Other C file commands!

• close(int fd)
• Close the file associated with the given file
  descriptor number.
• Can you close stdout? Try it.
• fclose(FILE stream)
• Just like close(), fclose can close stdout.

46
Other C file commands!

• ssize_t read(int fd, void buf,
  size_t count)
• Read up to count bytes from a file descriptor
  into the buffer buf.
• ssize_t write(int fd, void buf,
  size_t count)
• Write count bytes to a file descriptor from the
  buffer buf.

47
Buffered I/O versions

• size_t fread(void ptr, size_t size,
  size_t count, FILE
  stream)
• Read up to countsize bytes from a file
  descriptor into the buffer ptr.
• size_t fwrite(void ptr, size_t size,
  size_t count, FILE
  stream)
• Write countsize bytes to a file descriptor from
  the buffer ptr.

48
Other C file commands!

• off_t lseek(int fd, off_t offset,
  int whence)
• Seek to a different point in the file.
• lseek(fd, 4, SEEK_SET)
• Seek four bytes after the beginning of the file.
• lseek(fd, -4, SEEK_END)
• Seek four bytes before the end of the file.
• lseek(fd, 16, SEEK_CUR)
• Seek sixteen bytes ahead of the current position.

49
Other C file commands!

• int fseek(FILE stream, long int
  offset, int origin)
  
• fseek(stream, 4, SEEK_SET)
• Seek four bytes after the beginning of the file.
• fseek(stream, -4, SEEK_END)
• Seek four bytes before the end of the file.
• fseek(stream, 16, SEEK_CUR)
• Seek sixteen bytes ahead of the current position.

50
File descriptors

• A file acts like a tape.

51
File descriptors

• A file acts like a tape.

...
Array in kernel
52
File descriptors

• A file acts like a tape.

fd 1
...
Array in kernel
53
File descriptors

• A file acts like a tape.

fd 1
...
Array in kernel
file
Text contents of somefile go here...
54
File descriptors

• A file acts like a tape.

fd 1
...
Array in kernel
file
offset
Text contents of somefile go here...
55
Example show file contents

• We'll write a program to print the contents of a
  file to the terminal (similar to cat, but for one
  file).
• ./show-file somefile.txtText contents of
  somefile go here.
• Uses open(2), read(2), write(2), close(2).

56
lseek(2)

• include ltsys/types.hgtinclude ltunistd.hgtoff_t
  lseek(int fd, off_t offset,int whence)
• Moves the file offset. whence is one of
  SEEK_SET, SEEK_CUR, or SEEK_END.
• Returns the new offset.

57
Files

• A file is structured as a sequence of bytes,
  modeled after the notion of a tape
• Text contents of some file go here.

start of file
file offset (use lseek() to move or read
position)
58
Example get byte at offset

• Now we'll write a program to print the byte at a
  given offset in a file, or by default, the file
  length.
• ./get-byte somefile.txt35 ./get-byte
  somefile.txt 3t
• Uses lseek(2) plus functions from before.

59
UNIX File Systems
60
UNIX File Systems

• inode per-file data structure

61
UNIX File Systems

• inode per-file data structure
• Advantage

62
UNIX File Systems

• inode per-file data structure
• Advantage
• Efficient for small files
• Flexible if the size changes

63
UNIX File Systems

• inode per-file data structure
• Advantage
• Efficient for small files
• Flexible if the size changes
• Disadvantage

64
UNIX File Systems

• inode per-file data structure
• Advantage
• Efficient for small files
• Flexible if the size changes
• Disadvantage
• File must fit in a single disk partition

65
UNIX File Systems

• inode per-file data structure
• Advantage
• Efficient for small files
• Flexible if the size changes
• Disadvantage
• File must fit in a single disk partition

66
UNIX File Systems

• inode (continued)
• Storing Large Files

67
Directories are files too!

• Directories, like files, have inodes with
  attributes and pointers to disk blocks

68
Directories are files too!

• Directories, like files, have inodes with
  attributes and pointers to disk blocks
• Each directory contains the name and i-node for
  each file in the directory.

69
Directories are files too!

• Directories, like files, have inodes with
  attributes and pointers to disk blocks
• Each directory contains the name and i-node for
  each file in the directory.

70
UNIX File Operations
71
Directory functions

• include ltunistd.hgt
• Change the directory
• int chdir(const char path)

72
Directory functions

• include ltunistd.hgt
• Change the directory
• int chdir(const char path)
• Get the current working directory
• char getcwd(char buf, size_t size)

73
Directory functions

• include ltunistd.hgt
• Change the directory
• int chdir(const char path)
• Get the current working directory
• char getcwd(char buf, size_t size)
• Get the maximum path length
• long fpathconf(int fildes, int name)
• long pathconf(const char path, int name)
• long sysconf(int name)

74
Directory reading functions

• include ltdirent.hgt
• Open the directory
• DIR opendir(const char dirname)

75
Directory reading functions

• include ltdirent.hgt
• Open the directory
• DIR opendir(const char dirname)
• Close the directory
• int closedir(DIR dirp)

76
Directory reading functions

• include ltdirent.hgt
• Open the directory
• DIR opendir(const char dirname)
• Close the directory
• int closedir(DIR dirp)
• Read the directory
• struct dirent readdir(DIR dirp)

77
Example 1

• Use opendir and readdir to print all the
  filenames in the current directory
• include ltdirent.hgt
• DIR dir
• struct dirent entry
• dir opendir(.)
• while(entry readdir(dir))
• printf(s\n,entry-gtd_name)
• closedir(dir)
• Remember to include error checking!!

78
Whats in a directory entry?

• struct dirent
• Member Fields
• char d_name
• Null-terminated file name
• ino_t d_fileno
• inode number
• unsigned char d_namlen
• Length of file name
• unsigned char d_type
• Type of file
• DT_REG, DT_DIR, DT_FIFO, DT_SOCK, DT_CHR, DT_BLK,
  DT_UNKNOWN

79
Example 2

• Modify Example 1 to use the member fields of
  struct dirent to display the inode for each file,
  as well as whether the file is a directory or a
  regular file.
• include ltdirent.hgt
• DIR dir
• struct dirent entry
• dir opendir(.)
• while(entry readdir(dir))
• printf(s\n,entry-gtd_name)
• if(entry-gtd_type DT_DIR)
• printf(Directory )
• else
• printf(File )
• closedir(dir)
• Remember to include error checking!!

80
More Directory Functions

• include ltdirent.hgt
• Set the position of next readdir
• void seekdir(DIR dir, off_t offset)
• Set the position back to the start of the
  directory
• void rewinddir(DIR dirp)
• Get the current location of directory stream
• off_t telldir (DIR dir)

81
Warning! Warning!

• opendir and readdir are NOT thread-safe.
• DO NOT open two directories at the same time!

82
How to recursively traverse a directory tree

• Open the directory (opendir)
• Read each entry (readdir)
• If the file is a directory (d_type), store it
  (e.g. in an array of strings).
• Close the directory (closedir)
• Traverse each saved subdirectory, EXCEPT '.' and
  '..'

83
File information stat

• Use the stat functions to view the files inodes
  attributes.
• include ltsys/stat.hgt
• include ltsys/types.hgt
• include ltunistd.hgt
• For a file
• int stat(const char restrict path, struct stat
  restrict buf)
• For a link
• int lstat(const char restrict path, struct stat
  restrict buf)
• For a file descriptor
• int fstat(int fildes, struct stat buf)

84
Example 3

• Modify Example 2 to also give file information
  about each file.
• How large is each file?
• Which files are world-readable?
• Which files have been modified in the last 24
  hours?
• Hint man 2 stat

85

• include ltstdio.hgt
• include ltdirent.hgt
• include lttime.hgt
• include ltsys/types.hgt
• include ltsys/stat.hgt
• include ltunistd.hgt
• include lterrno.hgt
• int main(int argc, char argv)
• DIR dir
• struct dirent entry
• time_t now time(NULL)
• if((dir opendir(".")) NULL)
• perror("Can't open directory")
• exit(-1)

86
Useful fields and macros in struct stat

• stat.st_size
• File size
• stat.st_mode
• File type
• User permissions
• Etc.
• stat.st_mtime
• Time of last modification
• S_ISDIR(stat.st_mode)
• Is this a directory?

87
Links

• Hard Link
• Directory Entry
• e.g. all regular files
• Symbolic Link
• Also called a Soft Link
• A special file that serves as a reference to
  another file

88
Link Functions

• include ltunistd.hgt
• To create a new link
• int link(const char oldpath, const char
  newpath)
• Same as ln
• To removes an entry from the directory
• int unlink(const char path)
• Same as rm
• Returns 0 if successful, -1 with errno set if
  unsuccessful

89
Hard Link Example

• Command Line
• ln /dirA/name1 /dirB/name2
• C Code Segments
• if (link("/dirA/name1", "/dirB/name2") -1)
• perror("Failed to make a new link in /dirB")

90
Hard Link Example (contd)

• Q What happens if /dirA/name1 is deleted and
  recreated?

91
Hard Link Example (contd)

• A /dirA/name1 and /dirB/name2 are now two
  distinct files.

92
Symbolic Link Function

• include ltunistd.hgt
• To create a symbolic link
• int symlink(const char oldpath,
• const char newpath)
• Same function as command ln s
• Returns 0 if successful, -1 with errno set if
  unsuccessful

93
Soft Link Example

• Command Line
• ln s /dirA/name1 /dirB/name2
• C Code Segments
• if (symlink("/dirA/name1", "/dirB/name2") -1)
• perror("Failed to create a symbolic link in
  /dirB")

94
Soft Link Example (contd)

• Q What happens if /dirA/name1 to is deleted and
  recreated?

95
Soft Link Example (contd)

• A /dirA/name1 has a different inode, but
  /dir/name2 still links to it.

96
Link number

• The link number (the st_nlink field in stat)
  tells how many directory entries link to this
  inode. The link number is
• Set to 1 when a file is created
• Incremented when link is called
• Decremented when unlink is called
• The link number appears in the second column of
  the output of ls l. Try it!
• The link number only counts hard links, not soft
  links.

97
Summary

• UNIX File Systems
• i-nodes
• directories
• File operations
• Directory
• opendir, readdir, closedir
• File Status
• stat, fstat, lstat
• Links
• link, symlink, unlink

98
UNIX File Systems vs. Windows File Systems
99
UNIX File Systems

• I-node per-file data structure
• Advantage
• Efficient for small files
• Flexible if the size changes
• Disadvantage
• File must fit in a single disk partition

100
UNIX File Systems

• I-node (continued)
• Storing Large Files

101
Windows File Systems

• FAT File Allocation Table
• Advantage
• Random access is faster
• Disadvantage
• FAT should be in memory
• FAT16, FAT32
• Number of bits to identify blocks on a disk

102
Windows File Systems

• NTFS
• 64-bit index
• MFT Master File Table
• MFT record example
• Run represents one or multiple consecutive
  blocks

103
Windows File Systems

• NTFS (continued)
• Storing Large Files