📘 File Systems and Storage

A file system turns a device full of numbered blocks into named files and directories that survive crashes. This page covers how that works, why "write returned successfully" does not mean "the data is on disk", and how storage hardware shapes the design.

Table of Contents

1. Files and Directories
2. File Descriptors
3. Inodes
4. Hard Links and Soft Links
5. On-Disk Layout and Allocation
6. Crash Consistency: Journaling and Copy-on-Write
7. The VFS and Common File Systems
8. The Page Cache and Durability
9. Storage Hardware
10. Disk Scheduling
11. RAID
12. Questions

1. Files and Directories

A file is a named sequence of bytes plus metadata. The OS does not care about the format; that is the application's job.

A directory is a special file that maps names to inode numbers. Unix uses a single tree rooted at /, and other file systems are mounted onto directories in it.

Permissions

-rwxr-x---  1 alice devs  4096 Sep 24 10:00 deploy.sh
 |  |  |
 |  |  +-- others: no access
 |  +----- group: read, execute
 +-------- owner: read, write, execute

Octal: r = 4, w = 2, x = 1, so rwxr-x--- = 750. For directories, x means "may enter or traverse", and r means "may list names". Special bits: setuid (run as the file owner, like passwd), setgid, and sticky (in /tmp, only the owner can delete their files). umask sets default permissions for new files; ACLs and capabilities add finer control.

2. File Descriptors

A file descriptor (FD) is a small integer that a process uses to refer to an open file, socket, pipe, or device.

The kernel keeps three levels of tables:

• The open file entry holds the current offset and flags.
• After fork, parent and child share open file entries, so they share the offset.
• dup2(fd, 1) makes FD 1 point to the same entry as fd; this is how shells implement > redirection.
• Each process has an FD limit (ulimit -n, often 1,024 soft by default); servers with many connections must raise it, or they fail with EMFILE: Too many open files.
• lsof -p <pid> or ls -l /proc/<pid>/fd lists a process's open FDs.

3. Inodes

An inode stores everything about a file except its name.

stat notes.txt          # show inode number and metadata
ls -i                   # inode numbers
df -i                   # inode usage; a disk can be "full" with free space if inodes run out

Classic ext2/ext3 block pointers: 12 direct pointers, then single, double, and triple indirect blocks. With 4 KB blocks and 4-byte pointers, one indirect block holds 1,024 pointers, so single indirect covers 4 MB, double 4 GB, and triple 4 TB. Modern ext4 and XFS use extents instead: (start block, length) ranges, which describe large contiguous files compactly.

Deleting a file removes a directory entry and decrements the link count. The data is freed only when the link count reaches zero and no process has the file open. That is why deleting a huge log file that a process still writes to does not free space; lsof +L1 finds such deleted-but-open files.

4. Hard Links and Soft Links

Symlinks are used for version switching (current -> releases/2026-09-24) and atomic deploys (create a new symlink, then rename it over the old one).

5. On-Disk Layout and Allocation

A typical Unix file system layout:

| boot block | superblock | inode bitmap | data bitmap | inode table | data blocks ... |

ext4 groups these into block groups so a file's inode and data stay close together.

6. Crash Consistency: Journaling and Copy-on-Write

Appending a block to a file updates several structures: the data block, the inode, and the data bitmap. A crash between those writes leaves the file system inconsistent (a block marked used but owned by nobody, or an inode pointing at garbage).

Journaling modes in ext4:

This is the same write-ahead logging idea databases use; see Database Internals.

7. The VFS and Common File Systems

The Virtual File System layer gives one API (open, read, write, stat) over many file system implementations, including virtual ones like /proc, network ones like NFS, and FUSE file systems in user space.

Checksums matter: hard drives and SSDs occasionally return silently corrupted data (bit rot), and only checksumming file systems like ZFS and Btrfs detect and, with redundancy, repair it.

8. The Page Cache and Durability

Reads and writes go through the page cache in RAM.

• write() returning success means the data is in the kernel's memory, not on disk.
• A power loss before writeback loses it.
• fsync(fd) blocks until the file's data and metadata reach stable storage, including flushing the device's own write cache.
• fdatasync skips non-essential metadata like timestamps.
• O_DIRECT bypasses the page cache (databases with their own buffer pools use it).
• O_SYNC makes every write synchronous.

Writing a file safely

// Atomic file replacement
int fd = open("config.json.tmp", O_WRONLY | O_CREAT | O_TRUNC, 0644);
write(fd, data, len);
fsync(fd);                                   // data is durable
close(fd);
rename("config.json.tmp", "config.json");   // atomic on POSIX file systems
int dir = open(".", O_RDONLY);
fsync(dir);                                  // make the rename itself durable
close(dir);

Readers see either the old or the new file, never a partial one. Skipping the directory fsync can lose the rename after a crash.

Durability is expensive: an fsync costs from tens of microseconds on enterprise NVMe with power-loss protection to many milliseconds on consumer drives and HDDs. Databases amortize it with group commit (one fsync for many transactions).

A famous lesson: in 2018 PostgreSQL discovered that after a failed fsync Linux may drop the dirty pages and report success on a retry, so the data is silently lost. PostgreSQL now panics and recovers from its WAL on fsync failure ("fsyncgate").

9. Storage Hardware

HDD access time = seek time + rotational latency + transfer time. At 7,200 RPM a rotation takes 8.3 ms, so average rotational latency is about 4.2 ms.

Flash quirks the OS and databases care about:

The Linux block layer uses the multi-queue design (blk-mq) to match NVMe's parallel queues.

10. Disk Scheduling

For HDDs, the order of requests determines how far the head moves. Example: head at cylinder 53, queue 98, 183, 37, 122, 14, 124, 65, 67, disk cylinders 0 to 199.

SSTF can starve far requests; SCAN and C-SCAN give more uniform wait times.

Linux I/O schedulers today:

Check with cat /sys/block/nvme0n1/queue/scheduler.

11. RAID

RAID combines disks for performance, redundancy, or both.

Parity uses XOR: if P = A xor B xor C, then any lost block equals the XOR of the others.

RAID is not a backup. It protects against disk failure, not against deletion, corruption, ransomware, or losing the whole machine. Cloud block storage (EBS, Persistent Disk) is replicated behind the scenes; object storage (S3) uses erasure coding, a generalization of parity across many machines.

12. Questions

Next: I/O and Linux Internals.