📘 I/O and Linux Internals

This page connects OS theory to the systems you run every day. It explains how one Node.js or nginx process handles tens of thousands of connections, and how a Docker container is just a normal Linux process with a few kernel features switched on.

Table of Contents

1. I/O Models
2. I/O Multiplexing: select, poll, epoll
3. The C10K Problem and Event Loops
4. io_uring
5. Zero-Copy
6. Namespaces
7. cgroups
8. Capabilities, seccomp, and LSMs
9. What a Container Really Is
10. eBPF
11. Linux Toolbox
12. Questions

1. I/O Models

The five classic Unix I/O models (from Stevens' Unix Network Programming):

Synchronous vs asynchronous is about whether the app does the copy step. Blocking vs non-blocking is about whether a call waits. Readiness-based APIs (epoll) say "you can read now without blocking"; completion-based APIs (io_uring, IOCP) say "your read is done, here is the data".

2. I/O Multiplexing: select, poll, epoll

int ep = epoll_create1(0);
struct epoll_event ev = { .events = EPOLLIN, .data.fd = listen_fd };
epoll_ctl(ep, EPOLL_CTL_ADD, listen_fd, &ev);

struct epoll_event events[64];
for (;;) {
    int n = epoll_wait(ep, events, 64, -1);        // sleep until something is ready
    for (int i = 0; i < n; i++) {
        if (events[i].data.fd == listen_fd) {
            int c = accept4(listen_fd, NULL, NULL, SOCK_NONBLOCK);
            struct epoll_event cev = { .events = EPOLLIN, .data.fd = c };
            epoll_ctl(ep, EPOLL_CTL_ADD, c, &cev);
        } else {
            handle_readable(events[i].data.fd);    // read until EAGAIN
        }
    }
}

EPOLLEXCLUSIVE and SO_REUSEPORT spread accepts across worker processes without the thundering herd (all workers waking for one connection).

3. The C10K Problem and Event Loops

Around 1999 the question was how to serve 10,000 concurrent connections on one machine. Thread-per-connection needed 10,000 threads, each with a stack (often 1 to 8 MB reserved) and context-switch costs.

Node.js: libuv runs the event loop on epoll / kqueue / IOCP. Network I/O is truly non-blocking; file system operations, DNS lookup, and crypto run on a thread pool (4 threads by default, UV_THREADPOOL_SIZE) because regular file I/O cannot be made non-blocking with epoll. See Advanced JavaScript for the event loop phases.

Today C10M (ten million) is the benchmark, reached with kernel bypass (DPDK), io_uring, or careful epoll tuning.

4. io_uring

io_uring (Linux 5.1, 2019) is a completion-based async I/O interface built on two ring buffers shared between the application and the kernel.

Used by high-performance databases and storage engines (ScyllaDB via Seastar, TigerBeetle, PostgreSQL 18's new io_method = io_uring option), Rust runtimes (tokio-uring, glommio), and QEMU. Because of a history of security bugs, some environments (Google production, ChromeOS, Android apps, many container runtimes' default seccomp profiles) restrict or disable it.

5. Zero-Copy

Sending a file over a socket the naive way copies data four times and makes two system calls per chunk:

read():  disk -> page cache (DMA) -> user buffer (CPU copy)
write(): user buffer -> socket buffer (CPU copy) -> NIC (DMA)

Kafka's throughput story is largely sequential disk I/O, the page cache, batching, and sendfile.

6. Namespaces

Namespaces give a process its own view of a global resource.

sudo unshare --pid --fork --mount-proc --uts --net bash   # a crude "container"
hostname sandbox; ps aux                                   # only a few processes visible
lsns                                                       # list namespaces
nsenter -t <pid> -n ip addr                                # run a command in another process's net namespace

7. cgroups

Control groups limit and account for the resources a group of processes may use. cgroups v2 (a single unified hierarchy) is the default on current distributions and Kubernetes.

cat /sys/fs/cgroup/<group>/memory.max
cat /sys/fs/cgroup/<group>/cpu.stat     # nr_throttled, throttled_usec
systemd-cgls                            # tree of cgroups
systemd-cgtop                           # live usage per cgroup

Namespaces decide what a process can see; cgroups decide how much it can use.

8. Capabilities, seccomp, and LSMs

9. What a Container Really Is

A container is an ordinary process that:

1. Runs in new namespaces (its own view of PIDs, mounts, network, hostname).
2. Is placed in a cgroup with resource limits.
3. Has a root file system built from image layers with overlayfs (the old chroot idea, done with pivot_root).
4. Has reduced privileges via capabilities, seccomp, and an LSM profile.

The stack: Docker or Kubernetes (via CRI) call containerd or CRI-O, which call an OCI runtime like runc (or crun, or gVisor / Kata Containers for stronger isolation) to make the kernel calls.

Consequences worth knowing:

• ps on the host shows container processes; they are not hidden inside a VM.
• A kernel exploit in one container can compromise the host, which is why multi-tenant platforms use gVisor, Kata, or Firecracker microVMs.
• The container sees the host's total CPU and memory in /proc/cpuinfo and /proc/meminfo unless tools like LXCFS are used, so runtimes must read cgroup limits to size themselves.
• PID 1 inside the container must handle signals and reap zombies (see Processes and Threads).

10. eBPF

eBPF lets you load small, verified programs into the kernel and attach them to events: system calls, function entries, network packets, tracepoints. The verifier guarantees they terminate and cannot crash the kernel.

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_openat { printf("%s %s\n", comm, str(args->filename)); }'

11. Linux Toolbox

Processes and CPU

Memory and disk

Useful /proc and /sys files

The 60-second performance checklist

Adapted from Brendan Gregg's well-known checklist, run in order on a slow Linux box:

uptime                 # load trend
dmesg -T | tail        # OOM kills, errors
vmstat 1               # run queue, swapping, CPU split
mpstat -P ALL 1        # one hot core?
pidstat 1              # which process
iostat -xz 1           # disk latency and saturation
free -m                # memory and page cache
sar -n DEV 1           # network throughput
sar -n TCP,ETCP 1      # connections and retransmits
top                    # overview

Think in USE: for every resource, check Utilization, Saturation (queueing), and Errors.

12. Questions

Next: OS Interview Playbook.