CS110 Lecture 3: Filesystem Design, Part 2

CS110: Principles of Computer Systems

Winter 2021-2022

Stanford University

Instructors: Nick Troccoli and Jerry Cain

PDF of this presentation

This is a major diagram that shows how a Unix v6 file system is laid out. As with prior diagrams, it has an array-format, with sectors that go from 0, 1, 2, ..., (skip), ... 1024, 1025, etc. Sector 0 is labeled as the 'boot block' and sector 1 is labeled as the 'Superblock'. Sector 2 has been divided into 16 individual 'inodes', and all sectors up to 1024 are designated similarly. Sectors 0 - 1023 are labeled as the 'Filesystem metadata'. Sectors 1024 to the end of the disk are labeled as 'File contents'. In another part of the diagram, there are two inodes, labeled 'inode 1 (stored in sector 2, offset 0)' and 'inode 2 (stored in sector 2, offset 32). Inode 1 also has inside it: 'Type: directory, Filesize: 32 bytes, Contents: block 1024'. Inode 2 has the following inside: 'Type: regular file, Filesize: 1028 bytes, Contents: blocks 1027, 1028, 1025). Finally, the diagram has a block shown as 'Block 1024' which is divided into two rows. Row 1 says 'Bytes 0-15: a.mp3 2' and row 2 says 'Bytes 16-31: b.txt 3'

Asking Questions

Feel free to raise your hand at any time with a question
If you are more comfortable, you can post a question in the Ed forum thread for each day’s lecture (optionally anonymously)
We will monitor the thread throughout the lecture for questions

Visit Ed (or access via Canvas):

https://edstem.org/us/courses/16701/discussion/

Today's thread:

https://edstem.org/us/courses/16701/discussion/985437

CS110 Topic 1: How can we design filesystems to store and manipulate files on disk, and how can we interact with the filesystem in our programs?

Learning About Filesystems

Unix v6 Filesystem design, part 1 (files)

Unix v6 Filesystem design, part 2 (large files + directories)

Interacting with the filesystem from our programs

Lecture 2

This Lecture

Lecture 4

assign2: implement portions of a filesystem!

Learning Goals

Learn how to use indirect blocks to support even larger files
Understand the design of the Unix v6 filesystem in how it represents directories
Practice with the full process of going from file path to file data
Understand the tradeoffs and limitations in filesystem design

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Recap: Unix V6 Filesystem

We are imagining that we are filesystem implementers
We have a hard disk that supports only two operations, and need to layer complex filesystem operations (like reading/writing/locating entire files) on top:

The Unix V6 filesystem is one example of a filesystem design - there are many alternatives!

void readSector(size_t sectorNumber, void *data);
void writeSector(size_t sectorNumber, const void *data);

This is a diagram of the sector layout on a hard disk or solid state disk. It looks like an array of boxes, with each box labeled 'sector 0, bytes 0-511', 'sector 1, bytes 512-1023', 'sector 2, bytes 1024-1535', ... , 'sector 6, bytes 3072-3583'.

Recap: Unix V6 Filesystem

The inode table at the start of the disk stores one inode per file.

An inode is a structure containing information about a file such as its size and which blocks elsewhere on disk store its contents.

Recap: Large Files So Far

Problem: an inode stores only 8 block numbers. Therefore, the largest a file can be is 512 * 8 = 4096 bytes (~4KB). That definitely isn't realistic!

Solution: let's store them in a block, and then store that block's number in the inode! This approach is called indirect addressing.

inode

filesize: 5000

blocknums: 450

...

block 450

451,42,15,67,

125,665,467,

231,162,136

block 450

451,42,15,67,

125,665,467,

231,162,136

block 451

The quick brown fox jumped over the...

Recap: Singly-Indirect Addressing

The Unix V6 filesystem uses singly-indirect addressing (blocks that store payload block numbers) just for large files.

check flag or size in inode to know whether it is a small file (direct addressing) or large one (indirect addressing)
- If small, each block number in the inode stores payload data
- If large, first 7 block numbers in the inode stores block numbers for payload data
- 8th block number? we'll get to that :)

inode

filesize: 5000

blocknums: 450

...

block 450

451,42,15,67,

125,665,467,

231,162,136

block 450

451,42,15,67,

125,665,467,

231,162,136

block 451

The quick brown fox jumped over the...

Recap: Singly-Indirect Addressing

Let's assume for now that an inode for a large file uses all 8 block numbers for singly-indirect addressing. What is the largest file size this supports? Each block number is 2 bytes big.

inode

filesize: 5000

blocknums: 450

...

block 450

451,42,15,67,

125,665,467,

231,162,136

block 450

451,42,15,67,

125,665,467,

231,162,136

block 451

The quick brown fox jumped over the...

8 block numbers in an inode x

256 block numbers per singly-indirect block x

512 bytes per block

= ~1MB

The Missing Pieces

After our discussion so far, you may still have key questions like:

where are filenames stored?
how do you find a file given its name?
how are directories represented?

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Even Larger Files

Problem: even with singly-indirect addressing, the largest a file can be is 8 * 256 * 512 = 1,048,576 bytes (~1MB). That still isn't realistic!

Solution: let's use doubly-indirect addressing; store a block number for a block that contains singly-indirect block numbers.

Doubly-Indirect Blocks

Solution: let's use doubly-indirect addressing; store a block number for a block that contains singly-indirect block numbers.

inode

filesize: 5000

blocknums: 450

...

block 450

451,42,15,67,

125,665,467,

231,162,136

block 450

451,42,15,67,

125,665,467,

231,162,136

block 451

55,34,12,44,...

block 55

The quick brown fox jumped over the...

Allows even larger files, but data takes even more steps to access. How do we employ this idea?

Indirect Addressing

The Unix V6 filesystem uses indirect addressing (blocks that store payload block numbers) just for large files.

check flag or size in inode to know whether it is a small file (direct addressing) or large one (indirect addressing)
- If small, each block number in the inode stores payload data
- If large, first 7 block numbers are singly-indirect
- NEW: If large (and if needed), 8th block number is doubly-indirect (it refers to a block that stores singly-indirect block numbers)

In other words; a file can be represented using at most 256 + 7 = 263 singly-indirect blocks. The first seven are stored in the inode. The remaining 256 are stored in a block whose block number is stored in the inode.

Indirect Addressing

An inode for a large file stores 7 singly-indirect block numbers and 1 doubly-indirect block number. What is the largest file size this supports? Each block number is 2 bytes big.

263 singly-indirect block numbers total x

256 block numbers per singly-indirect block x

512 bytes per block

= ~34MB

Indirect Addressing

An inode for a large file stores 7 singly-indirect block numbers and 1 doubly-indirect block number. What is the largest file size this supports? Each block number is 2 bytes big.

OR:

(7 * 256 * 512) + (256 * 256 * 512) ~ 34MB

(singly indirect) + ( doubly indirect )

Better! still not sufficient for today's standards, but perhaps in 1975. Moreover, since block numbers are 2 bytes, we can number at most 2^16 - 1 = 65,535 blocks, meaning the entire filesystem can be at most 65,535 * 512 ~ 32MB.

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Unix V6 Filesystem Practice

Assume we have a large file with inumber 16. How do we find the block containing the start of its payload data? How about the remainder of its payload data?

Unix V6 Filesystem Practice

Go to block 26, and start reading block numbers. For the first number, 80, go to block 80 and read the beginning of the file (the first 512 bytes). Then go to block 41 for the next 512 bytes, etc.
After 256 blocks, go to block 35, repeat the process. Do this a total of 7 times, for blocks 26, 35, 32, 50, 58, 22, and 59, reading 1792 blocks.
Go to block 30, which is a doubly-indirect block. From there, go to block 87, which is an indirect block. From there, go to block 89, which is the 1793rd block.

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Now we understand how files are stored. But how do we find them?

The Directory Hierarchy

Filesystems usually support directories ("folders")
A directory can contain files and more directories
A directory is a file container. It needs to store what files/folders are contained within it. It also has associated metadata.
All files live within the root directory, "/"
We can specify the location of a file via the path to it from the root directory:

Common filesystem task: given a filepath, get the file's contents.

/classes/cs110/index.html

Key Idea: let's model a directory as a file. We have already designed support for storing payloads and metadata. Why not use it?

Directories as Files

A directory is a file container. It needs to store what files/folders are contained within it. It also has associated metadata.

Have an inode for each directory
A directory's "payload data" is a list of info about the files it contains
A directory's "metadata" is information about it such as its owner
Inodes can store a field telling us whether something is a directory or file

We can layer support for directories right on top of our implementation for files!

Representing Directories

Design decision: the Unix V6 filesystem makes directory payloads contain a 16 byte entry for each file/folder that is in that directory.

The first 14 bytes are the name (not necessarily null-terminated!)
The last two bytes are the inumber

Representing Directories

Given the inode for a directory, how could we find the inumber for a file it contains called "b.txt"?

The Lookup Process

/classes/cs110/index.html

Start at the root directory

The Lookup Process

/classes/cs110/index.html

In the root directory, find the entry named "classes".

The Lookup Process

/classes/cs110/index.html

In the "classes" directory, find the entry named "cs110".

The Lookup Process

/classes/cs110/index.html

In the "cs110" directory, find the entry named "index.html". Then read its contents.

The Lookup Process

The root directory ("/") is set to have inumber 1. That way we always know where to go to start traversing. (0 is reserved to mean "NULL" or "no inode").

The Lookup Process

/classes/cs110/index.html

Go to inode with inumber 1 (root directory).

The Lookup Process

/classes/cs110/index.html

In its payload data, look for the entry "classes" and get its inumber. Go to that inode.

The Lookup Process

/classes/cs110/index.html

In its payload data, look for the entry "cs110" and get its inumber. Go to that inode.

The Lookup Process

/classes/cs110/index.html

In its payload data, look for the entry "index.html" and get its inumber. Go to that inode and read in its payload data.

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

We want to find a file called "/local/files/fairytale.txt", which is a small file.

This diagram is to show how to find the file "/local/files/fairtale.txt" See the tables below.

Filename Lookup Practice #1 of 3

go to inode 1. It's small. We need to look in block 25 for the list of its entries.
Look in block 25 for "local" -> inode 16.
Go to inode 16. It's small. We need to look in blocks 27/54 for the list of its entries.
Look in block 27 for "files" (and then 54 if necessary) -> inode 31.
Go to inode 31. It's small. We need to look in block 32 for the list of its entries.
Look in block 32 for "fairytale.txt" -> inode 47.
go to inode 47. It's small. We need to look in blocks 80,89,87 in order for its 1,057 bytes of payload data.

We want to find a file called "/medfile", which is a large file.

Filename Lookup Practice #2 of 3

go to inode 1. It's small. We need to look in block 25 for the list of its entries.
Look in block 25 for "medfile" -> inode 16.
Go to inode 16. It's large. We need to look in block 26 for the first 256 payload block numbers.
Read through numbers in block 26. First, go to block 80 for the first 512 payload bytes. Then, go to block 87 for the second 512 payload bytes.
After doing this 256 times, go to block 30 and repeat.
Continue with all remaining singly-indirect blocks in the inode.

This image is to show how to find a file called "/medfile". See the tables below.

We want to find a file called "/largefile", which is a very large file.

Filename Lookup Practice #3 of 3

go to inode 1. It's small. We need to look in block 25 for the list of its entries.
Look in block 25 for "largefile" -> inode 16.
Go to inode 16. It's large. For the first seven block numbers, go to those blocks and read their 256 block numbers to get payload blocks.
For the eighth block, go to block 30. For each block number, go to that block and read in its block numbers to get payload blocks.

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Extra: hard links and soft links
Recap

Hard Links

It's possible for two different filenames to resolve to the same inumber.
Useful if you want multiple copies of a file without having to duplicate its contents.
If you change the contents of one, the contents of all of them change.
The inode stores the number of names that resolve to that inode, i_nlink. When that number is 0, and no programs are reading it, the file is deleted.
What we are describing here is called a hard link. All normal files in Unix (and Linux) are hard links, and two hard links are indistinguishable as far as the file they point to is concerned. In other words, there is no "real" filename, as both file names point to the same inode.
In Unix, you can create a link using the ln command.

In addition to hard links, the Unix filesystem has the ability to create soft links. A soft link is a special file that contains the path of another file, and has no reference to the inumber.
Soft links can "break" in the sense that if the path they refer to is gone (e.g., the file is actually removed from the disk), then the link will no longer work.
To create a soft link in Unix, use the -s flag with ln.
Example:

cgregg@myth66:/tmp$ echo "This is some text in a file" > file1
cgregg@myth66:/tmp$ ls -l file*
-rw------- 1 cgregg operator 28 Sep 27 09:57 file1
cgregg@myth66:/tmp$ ln -s file1 file2
cgregg@myth66:/tmp$ ls -l file*
-rw------- 1 cgregg operator 28 Sep 27 09:57 file1
lrwxrwxrwx 1 cgregg operator  5 Sep 27 09:58 file2 -> file1
cgregg@myth66:/tmp$ echo "Here is some more text." >> file2
cgregg@myth66:/tmp$ cat file1
This is some text in a file
Here is some more text.
cgregg@myth66:/tmp$ rm file1
rm: remove regular file 'file1'? y
cgregg@myth66:/tmp$ ls -l file*
lrwxrwxrwx 1 cgregg operator 5 Sep 27 09:58 file2 -> file1
cgregg@myth66:/tmp$ cat file2
cat: file2: No such file or directory
cgregg@myth66:/tmp$

When we create a soft link, ls gives us the path to the original file
But, the reference count for the original file remains unchanged
Again, changing the contents of the file via either filename changes the file.
If we delete the original file, the soft link breaks! The soft link remains, but the path it refers to is no longer valid. If we had removed the soft link before the file, the original file would still remain.

Soft Links

Lecture Plan

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Recap

The Unix V6 Filesystem is a well-engineered filesystem that makes many design decisions for how to represent files and directories.
Every file/directory has an inode containing information about it and what blocks store its data.
If the file size is small, an inode stores up to 8 direct block numbers
If the file size is large, an inode stores 7 singly-indirect and 1 doubly-indirect block number
A directory is "treated as a file" - its payload data stores the name and inumber of each of its entries in 16 byte chunks.
To find files, we can hop between directory inodes until we reach the inode for our file. We can start at inumber 1 for the root directory.

Design Principles: Modularity & Layering

We built layers on top of the low-level readSector and writeSector to implement a higher-level filesystem:

sectors -> blocks
blocks -> files
files -> inodes
inodes -> file names
file names -> path names
path names -> absolute path names

Design Principles: Modularity & Layering

Modularity: subdivision of a larger system into a collection of smaller subsystems, which themselves may be further subdivided into even smaller sub-subsystems.

Layering: the organization of several modules that interact in some hierarchical manner, where each layer typically only opens its interface to the module above it.

These ideas aren't specific to filesystems! Eg. networking systems also rely on layering.

Design Principles: Modularity & Layering

These ideas aren't specific to filesystems! Eg. networking systems also rely on layering.

Design Principles: Name Resolution

Our filesystem resolves human-friendly names (like "/usr/bin/program") to machine-friendly names (inumbers). This is called name resolution. Names let us refer to system resources.

This idea isn't specific to filesystems! E.g. when we visit a website URL like google.com, DNS ("domain name service") converts google.com (human-friendly) to an IP address like 74.125.239.51 (machine-friendly).

Recap

Recap: The Unix v6 Filesystem so far
Large Files
Practice: Indirect Blocks
Directories
Practice: Filename lookup
Recap

Next time: how do we interface with the filesystem in our programs?