The Filesystem Hierarchy & Everything Is a File
Linux's most elegant idea: the entire system state — devices, processes, kernel internals — is reachable through a single unified namespace of files you can read, write, and stat with the same five system calls.
Overview
When Ken Thompson and Dennis Ritchie designed Unix in the late 1960s, they made a radical bet: instead of giving every kind of resource — disk files, terminals, network sockets, hardware devices — its own bespoke API, they would expose everything through a single abstraction: the file. Open it, read it, write it, close it. This bet paid off so spectacularly that forty years later the Linux kernel, the most complex piece of software humans have ever built, still honors it.
The Filesystem Hierarchy Standard (FHS) is the map of this unified namespace. It tells you where configuration lives, where programs are installed, where the kernel exports its own state as browsable text files, and where hardware devices appear as nodes you can cat. If you can read a file, you can interrogate a running process. If you can write a file, you can reconfigure the kernel without rebooting. This is not a metaphor — it is literally how Linux works.
Understanding the FHS and the "everything is a file" philosophy is not just trivia for a shell quiz. It is the conceptual foundation for understanding how system calls work, how device drivers expose interfaces, how the kernel communicates with userspace, and how tools like strace, lsof, perf, and systemd are built. Every serious piece of systems software you will ever read or write depends on this model.
In this module you will go from "I know there are directories on Linux" to being able to explain why /proc/1/fd/0 is a symlink to your terminal, how writing a number to /proc/sys/net/ipv4/ip_forward enables kernel IP forwarding with zero recompilation, and what a file descriptor actually is in kernel memory. By the end, you will see the elegance that experienced kernel hackers see when they navigate these trees — and you will have the tools to start reading the kernel source that implements them.
Every kernel subsystem you will eventually contribute to — networking, storage, memory management, scheduling — exposes its state and configuration through the virtual filesystems covered in this module. Before you can read a device driver, you must understand how it registers entries under /sys. Before you can write a patch to the process scheduler, you must understand how /proc/PID/status gets populated. Before you can debug a system call, you must understand what a file descriptor is in kernel memory and how copy_to_user fills the buffer your read() call receives.
More practically: every kernel contributor spends hours every day navigating /proc and /sys. When a patch reviewer asks "does this expose a new sysfs attribute?", you need to know what that means and what the conventions are. When a bug report says "the value in /proc/interrupts is wrong after the patch", you need to be able to find the kernel code that writes that file and trace the data path. This module gives you the vocabulary and the mental model to work fluently at that level — which makes everything that comes after it land faster and stick harder.
The VFS Dispatch Path: One read() Reaching Three Backends
A single read() system call fans out to completely different kernel backends depending on what the file descriptor points to. The VFS layer (vfs_read and the file_operations vtable) provides the uniform interface; ext4, procfs, and device drivers each implement their own backend. The user program is oblivious to which path is taken.
The concepts
01The Filesystem Hierarchy Standard: A Tour of the Tree
The FHS is a specification maintained by the Linux Foundation that defines what belongs where on a Linux system. It exists so that software written for one distribution works on another, and so that humans can navigate an unfamiliar system without a map.
The key directories and what they guarantee:
/— the root. Everything hangs from here. On a running system, this is the VFS (Virtual File System) root, not necessarily a single physical disk./bin,/sbin— essential user/system binaries needed before/usris mounted (historically on a separate partition). On modern systemd systems these are often symlinks to/usr/bin,/usr/sbin./usr— the bulk of the OS: binaries, libraries, headers, man pages. Think of it as "Unix System Resources"./etc— editable text configuration. All system-wide config lives here:/etc/fstab(filesystems to mount),/etc/passwd(user database),/etc/hosts(static DNS),/etc/ssh/sshd_config. The name predates this meaning — it was literally "et cetera" — but the modern convention is clear./var— variable data that grows over time: log files (/var/log), mail spools, package manager databases, container image layers./tmp— ephemeral scratch space, typically backed bytmpfs(RAM), wiped on boot./home— user home directories./rootis root's home, kept separate by convention./lib,/lib64— shared libraries needed to run the binaries in/binand/sbin./lib/modules/$(uname -r)/is where kernel modules live./boot— the bootloader (grub), the compressed kernel image (vmlinuz), and the initial RAM disk (initrd/initramfs)./dev— device nodes. Not a real filesystem on disk — populated at boot byudevfrom the kernel's sysfs events./proc— a window into the running kernel. Entirely in memory, generated on demand./sys— the structured kernel object tree, one directory per kernel object./mnt,/media— conventional mount points for temporary mounts and removable media./run— runtime data for the current boot: PID files, sockets, lock files. Replaces the old/var/runand is alwaystmpfs.
The tree is navigated with the same tools whether you are reading a file on an NVMe SSD, interrogating a kernel data structure, or talking to a hardware device. That uniformity is the point.
# See the FHS in action: check which /bin is real vs. symlink
ls -la /bin /sbin /usr/bin | head -5
# On a modern Fedora/Ubuntu:
# /bin -> usr/bin
# /sbin -> usr/sbin
# The kernel version string is stored in /proc
cat /proc/version
# Linux version 6.8.0 (gcc version 13.2.0) #1 SMP ...
# Every mounted filesystem
cat /proc/mounts | column -t | head -10
02The "Everything Is a File" Philosophy: What It Really Means
The phrase "everything is a file" is often taught as trivia. It is actually a profound architectural decision with consequences that ripple through every layer of the system.
What it really means: The kernel exposes a uniform interface — open, read, write, close, ioctl, mmap — and arranges for every kind of resource to be reachable through a path in a filesystem namespace. The actual storage behind that path can be:
- Bytes on an ext4 filesystem on a disk (a regular file)
- Bytes from a network (an NFS mount)
- A character stream from a terminal (a char device)
- A block device (an HDD, SSD, partition)
- A kernel data structure rendered as text on demand (procfs)
- A kernel object's attributes in sysfs
- A network socket (exposed as a file descriptor, and accessible as
/proc/PID/fd/N) - A named pipe (FIFO), a Unix domain socket, an eventfd, a timerfd...
The power comes from composability. Because everything speaks the same interface, you can:
cat /dev/urandom | head -c 16 | xxd— pipe random bytes from a device through a filterstrace -e openat ls— see exactly which "files" a program openslsof -p $$— list every file descriptor held by the current shelldd if=/dev/sda of=/dev/null bs=4M— benchmark a block device using the file copy tool
The kernel side: Every filesystem (ext4, tmpfs, procfs, sysfs...) registers a set of operations structs (file_operations, inode_operations, super_operations). When you call read() on a file, the VFS layer looks up the file's inode, finds its file_operations->read function pointer, and calls it. For a regular file that function reads disk blocks. For /proc/uptime that function calls into the scheduler to compute elapsed time and formats it as ASCII. The caller never knows which backend it is talking to.
This is the interface vs. implementation separation taken to its logical extreme.
/* Both of these use the exact same read() system call.
The kernel dispatches to different backends. */
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
int main(void) {
char buf[128];
ssize_t n;
/* Backend: ext4 on NVMe — reads real disk blocks */
int fd1 = open("/etc/hostname", O_RDONLY);
n = read(fd1, buf, sizeof(buf) - 1);
buf[n] = '\0';
printf("hostname file: %s", buf);
close(fd1);
/* Backend: procfs — kernel generates string on demand */
int fd2 = open("/proc/uptime", O_RDONLY);
n = read(fd2, buf, sizeof(buf) - 1);
buf[n] = '\0';
printf("uptime: %s", buf);
close(fd2);
/* Backend: /dev/urandom — kernel CSPRNG */
int fd3 = open("/dev/urandom", O_RDONLY);
unsigned char rand_byte;
read(fd3, &rand_byte, 1);
printf("random byte: 0x%02x\n", rand_byte);
close(fd3);
return 0;
}
03File Descriptors: The Kernel's Bookmarks
When you call open(), the kernel doesn't give you a pointer to the file. It gives you a small non-negative integer: a file descriptor (fd). This integer is an index into a per-process file descriptor table maintained in the kernel.
Three levels of indirection in the kernel:
- Per-process fd table (
task_struct -> files -> fdt -> fd[]): an array indexed by the fd number. Each slot holds a pointer to astruct file. struct file(the "open file description"): holds the current file offset, open flags, and a pointer tofile_operations. Two processes can share the samestruct file(e.g., afterfork()) — their reads advance the same offset.struct inode: the actual file metadata (size, permissions, timestamps, data block pointers). Multiplestruct fileobjects can point at the same inode (the file opened twice).
Why integers? Because they cross the kernel/userspace boundary cheaply. You pass an int in a register. The kernel validates it (is it in range? is that slot non-null?) and then dereferences the table internally. Userspace never touches kernel pointers.
Standard fds: By convention (and POSIX mandate), every process starts with three open file descriptors:
0— stdin (standard input)1— stdout (standard output)2— stderr (standard error)
These are just entries in the fd table. They can be closed and replaced. When a shell runs ls > out.txt, it calls open("out.txt", ...) to get fd N, then dup2(N, 1) to copy that into slot 1, then close(N). The process that execs ls inherits an fd 1 that points at the file. ls calls write(1, ...) without knowing or caring.
Seeing fds live: /proc/PID/fd/ is a directory of symlinks, one per open fd. The link target shows what the fd points to. This is how lsof works — it just reads those symlinks.
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
int main(void) {
/* Open a file — gets fd 3 (0,1,2 already taken) */
int fd = open("/etc/hostname", O_RDONLY);
printf("fd = %d\n", fd); /* prints: fd = 3 */
/* Inspect our own fd table via /proc */
/* In a real program you'd readlink(), but let's use system() */
char cmd[64];
snprintf(cmd, sizeof(cmd), "ls -la /proc/%d/fd", getpid());
system(cmd);
/* Output:
lrwx------ 1 user user 64 ... 0 -> /dev/pts/0 (terminal)
lrwx------ 1 user user 64 ... 1 -> /dev/pts/0 (terminal)
lrwx------ 1 user user 64 ... 2 -> /dev/pts/0 (terminal)
lr-x------ 1 user user 64 ... 3 -> /etc/hostname
lr-x------ 1 user user 64 ... 4 -> /proc/27381/fd (ls itself)
*/
/* dup2: redirect stdout to a file */
int out = open("/tmp/captured.txt", O_WRONLY | O_CREAT | O_TRUNC, 0644);
dup2(out, 1); /* fd 1 now points at the file */
close(out);
write(1, "this goes to the file\n", 22);
close(fd);
return 0;
}
04/proc: The Kernel's Live Dashboard
/proc is a virtual filesystem — it exists entirely in RAM, has no backing storage, and its contents are generated on demand when you read them. The kernel registers it as type proc at boot.
Process entries — /proc/PID/: For every running process, the kernel creates a directory /proc/PID/ (where PID is the process ID). Inside:
cmdline— the command line, argv, null-separatedstatus— human-readable process state, memory, UID/GIDmaps— virtual memory map: every mapping, its permissions and backing filesmaps— per-mapping memory statistics (RSS, PSS, swap)fd/— symlinks to open file descriptors (as covered above)fdinfo/N— details on fd N: offset, flagsmem— the process's virtual address space as a file (readable withptracecapability)exe— symlink to the executable on diskcwd— symlink to the current working directoryenviron— the environment variables, null-separatedsched— scheduling statisticswchan— the kernel function the process is sleeping innet/— per-process-namespace network state
System-wide entries: At the top level of /proc, outside PID directories:
cpuinfo— one block per CPU, from the CPU's CPUID instructionmeminfo— detailed memory accounting (MemTotal, MemFree, Buffers, Cached, SwapUsed...)interrupts— per-CPU interrupt counts, one row per IRQ lineloadavg— the 1/5/15-minute load averages and runqueue depthuptime— seconds since boot, idle timeversion— kernel version stringfilesystems— filesystems compiled in or loaded as modulesmounts— currently mounted filesystems (alias:/proc/self/mounts)net/— network statistics:dev(per-interface),tcp,udp,route...sys/— kernel tunables (see the sysctl section below)self/— a magic symlink that always resolves to the current process's PID directory
How it works internally: The procfs code is in fs/proc/ in the kernel source. For each file, a proc_ops struct provides a proc_read (or seq_file) callback. When you read() the file, the kernel calls that callback, which walks kernel data structures and writes formatted text into your buffer. The file has zero size on disk (because it doesn't exist on disk) — ls -l /proc/cpuinfo shows 0 bytes, but cat works fine because the size is generated dynamically.
#!/bin/bash
# Tour of /proc — run this and read the output
echo '=== Your process =='
cat /proc/self/status | grep -E '^(Name|Pid|VmRSS|Threads)'
echo '=== Open file descriptors of this shell =='
ls -la /proc/$$/fd
echo '=== Memory map of this shell (first 5 lines) =='
cat /proc/$$/maps | head -5
# Each line: addr_start-addr_end perms offset dev inode path
# e.g.: 55a3c2f00000-55a3c2f28000 r-xp 00000000 08:01 1234567 /bin/bash
echo '=== System memory =='
grep -E '^(MemTotal|MemFree|MemAvailable|Cached|SwapTotal)' /proc/meminfo
echo '=== Load average =='
cat /proc/loadavg
# 0.42 0.55 0.61 2/847 29183
# ^1min ^5min ^15min running/total last_pid
echo '=== What syscall is PID 1 sleeping in? =='
cat /proc/1/wchan
05/proc/sys and sysctl: Writing to the Running Kernel
/proc/sys/ is a subtree of procfs where each file corresponds to a kernel tunable — a parameter you can read and write to change kernel behavior on a live, running system. No reboot required.
The sysctl(8) command is just a friendlier interface to reading and writing these files. sysctl -w net.ipv4.ip_forward=1 is literally the same as echo 1 > /proc/sys/net/ipv4/ip_forward.
Key subtrees:
/proc/sys/kernel/— core kernel settings:hostname,pid_max,panic(seconds before reboot on kernel panic),printk(kernel log verbosity levels),randomize_va_space(ASLR)/proc/sys/net/— networking:ipv4/ip_forward(enable routing),ipv4/tcp_*(TCP tuning),core/rmem_max(socket receive buffer max)/proc/sys/vm/— virtual memory:swappiness(0–100, how aggressively to swap),dirty_ratio(% of RAM to allow dirty pages before writeback),overcommit_memory/proc/sys/fs/— filesystem:file-max(system-wide fd limit),inotify/max_user_watches
How it works: Each sysctl is registered in the kernel with a ctl_table struct that holds the variable's memory address, type, and min/max constraints. When you write to the file, the kernel validates the value and stores it directly into the kernel variable. Reading reads from that variable. No parsing, no config file — you are directly poking kernel memory through a controlled interface.
Persistence: Changes made via /proc/sys/ or sysctl -w are not persistent across reboots. To make them permanent, add lines to /etc/sysctl.conf or drop a file into /etc/sysctl.d/. The sysctl -p command (or systemd on boot) re-applies them.
Security: Writing to /proc/sys/ requires CAP_SYS_ADMIN (effectively root). Within containers and namespaces, some sysctls are namespaced (networking), others are system-global and cannot be changed inside a container.
#!/bin/bash
# Read and write kernel tunables
# Read: current hostname as seen by kernel
cat /proc/sys/kernel/hostname
# Read: is IP forwarding enabled? (0=no, 1=yes)
cat /proc/sys/net/ipv4/ip_forward
# Write: enable IP forwarding (requires root)
sudo sysctl -w net.ipv4.ip_forward=1
# kernel.net.ipv4.ip_forward = 1
# ...equivalent to:
# echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward
# Read: ASLR setting (0=off, 1=conservative, 2=full)
cat /proc/sys/kernel/randomize_va_space
# Read vm.swappiness (default 60 on most distros)
sysctl vm.swappiness
# List ALL current sysctl values:
sysctl -a 2>/dev/null | head -20
# Persist a setting across reboots:
echo 'net.ipv4.ip_forward = 1' | sudo tee /etc/sysctl.d/99-forwarding.conf
sudo sysctl -p /etc/sysctl.d/99-forwarding.conf
06/sys: The Kernel Object Model
/sys (sysfs) was introduced in Linux 2.6 to solve a problem with /proc: procfs had become a dumping ground for anything kernel developers wanted to expose, with no consistent structure. Sysfs provides a structured, hierarchical view of the kernel's internal object model — specifically, the kobject subsystem.
Every object in the kernel that registers with the driver model (every device, every bus, every driver) gets a directory in /sys. The hierarchy mirrors real hardware topology.
Key subtrees:
/sys/bus/— buses (pci, usb, i2c, platform...). Each bus hasdevices/anddrivers/subdirectories./sys/devices/— the canonical device tree, organized by physical topology:platform/,pci0000:00/,usb1/.../sys/class/— devices organized by type regardless of bus./sys/class/net/has one directory per network interface./sys/class/block/has one per block device. Each is a symlink into/sys/devices/./sys/module/— loaded kernel modules and their parameters./sys/kernel/— miscellaneous kernel state:debug/,mm/,tracing/./sys/power/— power management:/sys/power/stateacceptsfreeze,mem,diskto trigger system sleep states.
Attributes: Each directory contains attribute files — small text files (usually a single number or string) that expose one property of the kernel object. For a network interface /sys/class/net/eth0/ you get:
operstate—upordownspeed— link speed in Mbpsaddress— MAC addressmtu— current MTUstatistics/rx_bytes,statistics/tx_bytes— traffic counters
The rule: sysfs attributes should be one value per file. This is enforced by convention and by the ATTRIBUTE_GROUPS macros in driver code. This is the opposite of procfs files that can contain arbitrarily formatted multi-field output.
Writing to sysfs: Many attributes are writable. Writing to /sys/class/net/eth0/mtu changes the interface MTU. Writing 1 to /sys/bus/pci/devices/0000:00:1f.2/remove asks the kernel to hot-remove a PCI device. This is how udev rules, NetworkManager, and many system tools configure hardware without any kernel recompilation.
#!/bin/bash
# Explore /sys — the kernel object model
# List all network interfaces
ls /sys/class/net/
# lo eth0 wlan0 docker0 ...
# Inspect eth0
cat /sys/class/net/eth0/operstate # up
cat /sys/class/net/eth0/speed # 1000 (Mbps)
cat /sys/class/net/eth0/address # 00:11:22:33:44:55
cat /sys/class/net/eth0/mtu # 1500
# Traffic statistics
cat /sys/class/net/eth0/statistics/rx_bytes
cat /sys/class/net/eth0/statistics/tx_bytes
# List all block devices
ls /sys/class/block/
# sda sda1 sda2 nvme0n1 nvme0n1p1 ...
# See where /sys/class symlinks point (physical path)
ls -la /sys/class/net/eth0
# lrwxrwxrwx ... eth0 -> ../../devices/pci.../net/eth0
# List all PCI devices
ls /sys/bus/pci/devices/ | head -5
# Read a module parameter
cat /sys/module/usbcore/parameters/autosuspend
# Trigger a power management state (DANGEROUS — suspends machine!)
# echo mem > /sys/power/state
07/dev: Device Nodes and the mknod System Call
/dev contains device nodes — special files that are entry points into kernel device drivers. When you open /dev/sda, the kernel looks up the device's major and minor numbers encoded in the inode, finds the registered driver for that major number, and routes your read()/write() calls to that driver's file_operations.
Types of device nodes:
- Character devices (
cinls -l): byte-stream devices. Data is read/written sequentially, like a terminal or a serial port. Examples:/dev/tty(current terminal),/dev/null,/dev/zero,/dev/urandom,/dev/mem. - Block devices (
binls -l): random-access, block-oriented storage. The kernel caches block device I/O in the page cache. Examples:/dev/sda(whole disk),/dev/sda1(first partition),/dev/nvme0n1. - Pseudo-devices: These have no hardware counterpart but use the same interface:
/dev/null— swallows all writes, reads return EOF immediately/dev/zero— reads return infinite zero bytes, writes discarded/dev/urandom— reads return cryptographically secure random bytes from the kernel's CSPRNG/dev/full— reads return zeros, writes always fail withENOSPC(useful for testing error paths)
Major and minor numbers: The major number identifies the driver. The minor number identifies the specific device instance handled by that driver. For example, SCSI disk drives use major 8; minor 0 is /dev/sda, minor 16 is /dev/sdb, minor 1 is /dev/sda1.
udev and dynamic population: In early Linux, /dev was a static directory with hundreds of pre-created nodes. Today, udev (part of systemd) listens for uevent netlink messages from the kernel whenever hardware is added or removed. It dynamically creates or removes device nodes in /dev, runs rules (/etc/udev/rules.d/), sets permissions, creates symlinks, and runs scripts. The entire /dev directory is a devtmpfs — a RAM filesystem.
Named pipes and Unix sockets also appear in the filesystem as special files (p and s in ls -l), further extending the uniform namespace.
# Inspect device nodes
ls -la /dev/sda /dev/tty /dev/null /dev/urandom
# brw-rw---- ... 8, 0 ... /dev/sda (block, major=8, minor=0)
# crw-rw-rw- ... 5, 0 ... /dev/tty (char, major=5, minor=0)
# crw-rw-rw- ... 1, 3 ... /dev/null (char, major=1, minor=3)
# crw-rw-rw- ... 1, 9 ... /dev/urandom (char, major=1, minor=9)
# Read 16 random bytes and hex-dump them
dd if=/dev/urandom bs=16 count=1 2>/dev/null | xxd
# /dev/null: the bit bucket
echo "disappear" > /dev/null
cat /dev/null # prints nothing
# /dev/zero: source of zeros (useful for wiping, creating sparse files)
dd if=/dev/zero of=/tmp/zeroes bs=1M count=1 2>&1
# 1+0 records in/out, 1048576 bytes copied
# See udev rules for a device
udevadm info --query=all --name=/dev/sda | head -20
# Create a device node manually (requires root, rarely needed)
# mknod /tmp/my_null c 1 3 # same major/minor as /dev/null
08Special Files: Pipes, Sockets, and Symbolic Links
The "everything is a file" philosophy extends to several other types of filesystem objects that go beyond regular files and device nodes.
Named pipes (FIFOs): A FIFO (mkfifo) creates a persistent, named rendezvous point for one-directional communication between two processes. Unlike anonymous pipes (|), FIFOs have a name in the filesystem and persist until deleted. A writer opens the FIFO for writing; a reader opens it for reading. Data written by the writer is read by the reader in FIFO order. If no reader is present, the writer blocks at open() — the kernel will not allow data to pile up without a consumer.
Unix domain sockets: Created by bind() on a AF_UNIX socket, these appear as s files in the filesystem. They provide bidirectional IPC between processes on the same machine, much faster than loopback TCP because there is no network stack overhead. systemd's socket-activation mechanism, Docker's daemon socket (/var/run/docker.sock), and X11's display socket (/tmp/.X11-unix/X0) are all Unix domain sockets. You can see them with ls -la and interact with them using socat or nc -U.
Symbolic links: A symlink (ln -s) is a special file whose content is a path string. The kernel dereferences symlinks automatically on most path lookups (namei). Symlinks can cross filesystem boundaries (unlike hard links), can point to non-existent targets, and are used heavily in the FHS (e.g., /lib -> usr/lib, /proc/self always resolving to the current process). /proc/PID/fd/N symlinks are special: they are "magic" symlinks whose target is only valid while the fd is open — readlink() works, but the path might not be openable by path if the original file has been unlinked (you'd need to open the fd symlink directly via open("/proc/self/fd/3").
Hard links: Multiple directory entries pointing to the same inode. They must be on the same filesystem, cannot span to directories (on most filesystems), and the file's data is only deleted when the link count drops to zero and no process has the file open. This is why deleting a running executable or a currently-open log file doesn't destroy the data immediately — the kernel keeps the inode alive as long as any fd or directory entry references it.
#!/bin/bash
# Named pipe (FIFO) demo
mkfifo /tmp/myfifo
# In terminal 1:
echo "hello from writer" > /tmp/myfifo # BLOCKS until reader opens
# In terminal 2 (run this first, or use &):
cat /tmp/myfifo # reads and prints "hello from writer"
rm /tmp/myfifo
# Unix domain socket — talk to docker
echo '{"method":"GET","path":"/version"}' | \
socat - UNIX-CONNECT:/var/run/docker.sock 2>/dev/null | head -c 200
# Symlink — /proc/self magic
ls -la /proc/self # symlink -> current PID
readlink /proc/self # prints the PID of the subshell running this
# Hard link vs symlink
echo original > /tmp/original
ln /tmp/original /tmp/hardlink # hard link, same inode
ln -s /tmp/original /tmp/symlink # symlink, different inode
stat /tmp/original /tmp/hardlink | grep 'Inode:'
# Both show same inode number!
stat /tmp/symlink | grep 'Inode:' # different inode
# Removing the original doesn't affect the hard link
rm /tmp/original
cat /tmp/hardlink # still works!
09The Virtual File System (VFS) Layer: How the Kernel Unifies Everything
The "everything is a file" abstraction is implemented by the VFS (Virtual File System), a layer of indirection in the kernel that sits between system calls and concrete filesystem implementations.
The key VFS data structures (all in include/linux/fs.h):
struct super_block: Represents a mounted filesystem instance. Holds the filesystem type, mount flags, and a pointer tosuper_operations(how to read/write superblocks, allocate inodes, etc.).struct inode: Represents a file's metadata: mode, owner, timestamps, size, and — critically —i_fop, a pointer to afile_operationsstruct, andi_op, a pointer toinode_operations. This is where polymorphism lives.struct dentry: Represents a path component — a name-to-inode mapping in a directory. The dentry cache (dcache) speeds up path lookups by caching recently resolved paths.struct file: Represents an open file — created atopen(), destroyed atclose(). Holds the current offset, open flags, and a pointer to the inode (via dentry).struct file_operations: A table of function pointers:open,read,write,llseek,ioctl,mmap,poll,release... Each filesystem or device driver fills in the ones it supports. This is classic C polymorphism via function pointer tables.
The read() system call path (simplified):
- User calls
read(fd, buf, count)— software interrupt / syscall entry - Kernel looks up fd in
current->files->fdt->fd[fd]→struct file *f - Calls
f->f_op->read(f, buf, count, &f->f_pos) - For ext4: reads pages from the page cache, filling from disk if needed
- For procfs
/proc/uptime: calls the registered seq_file callback, formats the string - For
/dev/urandom: calls the CSPRNG to generate bytes - Kernel copies data to userspace buffer with
copy_to_user() - Returns number of bytes read
The elegance: steps 1, 2, and 7 are identical for all files. Only step 3 varies — and it varies via a function pointer, not a giant switch statement. This is why adding a new filesystem or device driver doesn't require changing the VFS.
/* Simplified view of the VFS structures — from include/linux/fs.h */
/* Function pointer table — the "vtable" for a file type */
struct file_operations {
ssize_t (*read)(struct file *, char __user *, size_t, loff_t *);
ssize_t (*write)(struct file *, const char __user *, size_t, loff_t *);
int (*open)(struct inode *, struct file *);
int (*release)(struct inode *, struct file *);
long (*unlocked_ioctl)(struct file *, unsigned int, unsigned long);
/* ... many more ... */
};
/* Every file's metadata + the pointer to its operations */
struct inode {
umode_t i_mode; /* permissions + file type bits */
kuid_t i_uid;
kgid_t i_gid;
loff_t i_size;
const struct file_operations *i_fop; /* THIS is the magic */
/* ... */
};
/* The read() syscall, conceptually */
SYSCALL_DEFINE3(read, unsigned int, fd, char __user *, buf, size_t, count)
{
struct file *f = fdget(fd); /* look up fd table */
if (!f) return -EBADF;
/* calls f->f_op->read — dispatches to the right backend */
return vfs_read(f, buf, count, &f->f_pos);
}
10/etc: Configuration as Plain Text
/etc is where every piece of system-wide configuration lives, and it follows a principle that reinforces "everything is a file": configuration is plain text, human-readable, and editable with any text editor.
This is not accidental. Plain text files can be version-controlled (git, etckeeper), diffed, backed up, distributed, templated, and audited. Compare this to a binary registry (Windows) where you need a special tool to read or modify configuration.
Critical files every systems programmer must know:
/etc/fstab— the filesystem table. Each line specifies a device/UUID, mount point, filesystem type, mount options, and fsck parameters. Read at boot bymount -a; also read bysystemd-fstab-generatorto create mount units./etc/passwdand/etc/shadow— user database.passwdis world-readable (programs need to look up usernames);shadowholds the password hashes and is root-only. Format: colon-separated fields, one user per line./etc/group— group membership database./etc/hosts— static hostname-to-IP mappings, consulted before DNS. The line127.0.0.1 localhostis here./etc/resolv.conf— DNS resolver configuration:nameserver 8.8.8.8,search example.com./etc/ssh/sshd_config— SSH daemon configuration. Knowing how to read this is essential for understanding security hardening./etc/sudoers— who can run what as root. Always edit withvisudo(validates syntax before saving)./etc/systemd/system/— local systemd unit files overriding or adding to the system defaults./etc/ld.so.conf— tells the dynamic linker where to find shared libraries. Changes take effect afterldconfig./etc/sysctl.d/— persistent kernel parameter configuration, applied at boot.
The convention of *.d/ directories (e.g., /etc/sysctl.d/, /etc/cron.d/, /etc/ssh/sshd_config.d/) deserves special mention. Instead of editing a monolithic config file, each package or administrator drops a file into the .d/ directory. Files are processed in lexicographic order. This makes it trivial to add, remove, or override configuration without merge conflicts — a lesson in composable system design that appears throughout Linux.
# Decode /etc/fstab structure
cat /etc/fstab
# UUID=abc123 / ext4 defaults 0 1
# UUID=def456 /boot vfat umask=0077 0 2
# UUID=ghi789 swap swap sw 0 0
# tmpfs /tmp tmpfs nodev,nosuid,size=2G 0 0
# ^device ^mntpnt ^fstype ^options ^dump ^fsck-order
# Decode a passwd line
# root:x:0:0:root:/root:/bin/bash
# ^usr ^pw(x=shadow) ^uid ^gid ^gecos ^home ^shell
grep ^root /etc/passwd
# List all users with UID >= 1000 (real human users)
awk -F: '$3 >= 1000 {print $1, $3, $7}' /etc/passwd
# See what libraries ldconfig knows about
ldconfig -p | head -10
# Which process owns /etc/resolv.conf? (systemd-resolved rewrites it)
ls -la /etc/resolv.conf # might be a symlink!
readlink -f /etc/resolv.conf
Key insights
- ✦The VFS's file_operations struct is C's version of a vtable — it is how the kernel achieves polymorphism across dozens of filesystem types and device drivers without any runtime type checking.
- ✦A file descriptor is not a file — it is a kernel-maintained index into a table of open file descriptions. Two file descriptors in two different processes can share the same open file description (and thus the same offset), which is how parent/child I/O coordination works after fork().
- ✦/proc and /sys have zero bytes on disk because they have no disk backing — they are generated in kernel memory at read() time. The 0-byte size shown by ls -l is accurate; the content is not.
- ✦Writing to /proc/sys/ directly modifies a live kernel variable. There is no config-file parsing, no daemon restart. You are writing to kernel memory through a controlled interface.
- ✦The .d/ directory convention (sysctl.d, cron.d, udev/rules.d) is a packaging and composability pattern: it lets N independent contributors each own their own file rather than fighting over one monolithic config.
- ✦When a file is unlinked while still open, its inode remains alive until all file descriptors to it are closed. This is why you can delete a log file while a daemon is writing to it — the daemon's fd still works, but the directory entry is gone.
- ✦Every device driver in the kernel is just a struct with function pointers. /dev nodes, sysfs attributes, and procfs files are all just different ways of routing system calls to those function pointers.
Common pitfalls
- ⚠Treating /proc file sizes as meaningful — ls -l shows 0 bytes for all procfs files because they have no disk backing, but cat and read() work fine. Never use file size to decide whether to read a procfs entry.
- ⚠Forgetting that sysctl changes via /proc/sys/ or sysctl -w are ephemeral — they vanish on reboot. To persist, add to /etc/sysctl.d/. A classic ops mistake: enabling ip_forward manually, then wondering why it's gone after the next reboot.
- ⚠Confusing file descriptors with file descriptions — after fork(), parent and child share the same open file description (same offset). If both read() from the same fd, they will each get different chunks of the file, not the same chunk.
- ⚠Opening /dev/sda directly while it has mounted partitions — you can do it for reading (useful for forensics), but writing directly to the device while a filesystem is mounted can corrupt the filesystem silently.
- ⚠Conflating /sys/class/net/eth0 with /proc/net — both expose network information, but sysfs (per-device attributes) and procfs (aggregated tables) serve different purposes and are maintained by different kernel subsystems.
- ⚠Editing /etc/sudoers with a regular editor — if you introduce a syntax error, sudo stops working and you can be locked out of root access. Always use visudo, which validates syntax before saving.
- ⚠Assuming /tmp is on disk — on most modern systems /tmp is a tmpfs (RAM disk). Files there vanish on reboot, and writing large files there competes with your RAM. Use /var/tmp for data that should survive a reboot.
Practice
- 01medium
Write a C program that reads /proc/self/maps and parses it to print a summary: for each mapping, print whether it is readable, writable, executable, and whether it is backed by a file. Count how many mappings of each permission combination exist.
- 02easy
Write a shell script that monitors a process (given as PID argument) every second and logs to a file: its RSS memory (from /proc/PID/status), number of open file descriptors (count entries in /proc/PID/fd/), and the system call it is sleeping in (/proc/PID/wchan). Run it against a long-running process like a web server or sleep.
- 03hard
Without using any external tools (no lsof, no ss, no netstat), write a C or Python program that reads /proc/net/tcp and decodes it to print all listening TCP ports. The addresses in /proc/net/tcp are hex-encoded little-endian; you must convert them to dotted-decimal IP and decimal port.
- 04easy
Use /sys/class/net/ to write a script that prints a network interface report: for each interface, show its state, MAC address, MTU, and the total bytes transmitted and received since boot. Format it as a table.
- 05hard
Implement a minimal 'strace-lite' in C: use /proc/self/syscall to read the current system call number of another process, and cross-reference it against a hardcoded table of syscall numbers and names (at least the top 20 common ones). Poll a target PID every 100ms and print any new syscall you see.
- 06medium
Create a named pipe (FIFO) and write a producer/consumer pair of C programs that communicate through it. The producer should write 100 timestamped messages and then close the pipe. The consumer should read and print each message. Verify that the producer blocks at open() until the consumer is ready by adding a deliberate sleep in the consumer before open().
Resources
The authoritative reference for every syscall and special filesystem. Read proc(5) cover to cover — it documents every /proc entry.
The clearest single-author tour of kernel internals at the right altitude for a newcomer. VFS chapter alone is worth the price.
The bible for Linux system programming. Covers file descriptors, /proc, device files, FIFOs, and sockets with extraordinary depth and accuracy.
The actual specification. Reading it once gives you a canonical mental map of the Linux directory tree.
Goes deeper than Love's book on internal data structures. Dense but rewarding for the VFS chapters.
LWN's curated index of articles on procfs — covers design decisions, security concerns, and evolution over kernel versions.
The authoritative kernel docs for sysfs rules, attribute naming conventions, and the kobject model.
Free, brilliant OS textbook. The file system chapters build the conceptual foundation before you dig into Linux specifics.
Checkpoints — you can now…
- ✓You can navigate /proc/PID/ and explain what each major subdirectory (fd/, maps, status, wchan, environ) contains and how the kernel generates it on demand.
- ✓You can write a C program that opens a regular file, a procfs file, and a device node, and explain why all three use the same read() call and what happens inside the kernel for each.
- ✓You can read and write /proc/sys/ tunables, explain the difference between a volatile sysctl change and a persistent one, and know where to put persistent settings.
- ✓You can use /sys/class/net/ and /sys/class/block/ to extract live hardware attributes without any external tools, and explain the one-value-per-file sysfs convention.
- ✓You can explain the three levels of the fd table (per-process table, struct file, inode) and why sharing a struct file across fork() shares the file offset.
- ✓You can distinguish between character devices, block devices, named pipes, and Unix sockets as they appear in the filesystem, and name a real-world use case for each.
- ✓You can describe the VFS dispatch path: how a read() system call reaches the right backend (ext4 vs. procfs vs. chardev) through the file_operations function pointer table.
- ✓You can explain why /tmp on a modern system should not be used for persistent data, what tmpfs is, and how to check whether a mount is backed by RAM or disk.