The Page Cache & Memory Reclaim

The page cache is a region of the kernel's address space that holds physical memory pages used to cache the contents of files and block devices. It is the primary consumer of 'free' memory on a busy Linux system — and intentionally so.

Every file on disk is divided into chunks the size of one memory page (typically 4 KiB on x86). Each of those chunks can be loaded into a page in the page cache and indexed by an address_space object. The address_space structure (defined in include/linux/fs.h) binds a set of pages to an inode (a file) and carries a pointer to address_space_operations — the method table that tells the kernel how to read and write pages for this particular filesystem.

When you call read(fd, buf, n), the kernel computes which page-aligned offsets within the file you need, looks them up in the address_space's xarray (a radix-tree-like data structure), and either finds them already cached (a page cache hit) or must load them from disk (a page cache miss). On a hit, the kernel copies bytes from the cached page into your user-space buffer. No disk I/O. No syscall overhead beyond the copy itself.

You can observe the page cache live:

# Before reading a large file:
free -m
# available: ~8000 MB

cat /dev/null > /proc/sys/vm/drop_caches   # won't work — need:
echo 3 | sudo tee /proc/sys/vm/drop_caches  # drop page cache, dentries, inodes

# Read a 2 GB file:
time cat bigfile.bin > /dev/null
# First run: slow (disk-bound)

time cat bigfile.bin > /dev/null
# Second run: blazing fast (served from page cache)

free -m
# available: ~6000 MB  — page cache consumed ~2 GB

This 'stealing' of free memory is not a leak — it is correct behavior. The kernel tracks exactly how much of that memory is page cache (shown as Cached: in /proc/meminfo) and will reclaim it instantly when applications need RAM.

The key data structures:

• struct page (or struct folio since 5.16): one physical page of memory
• struct address_space: maps (inode, file_offset) → page
• struct inode: file metadata; carries a pointer to its address_space
• xarray inside address_space: the lookup index, keyed by page index (file offset / PAGE_SIZE)

/* Simplified view of the relationship — kernel include/linux/fs.h */
struct address_space {
    struct inode        *host;        /* owner: inode or block device */
    struct xarray        i_pages;     /* cached pages, keyed by index */
    const struct address_space_operations *a_ops; /* read/write methods */
    unsigned long        nrpages;     /* total pages in cache */
    /* ... writeback fields, flags ... */
};

struct address_space_operations {
    int (*read_folio)(struct file *, struct folio *);
    int (*writepage)(struct page *, struct writeback_control *);
    void (*readahead)(struct readahead_control *);  /* batch read */
    /* ... many more ... */
};

When you write to a file, the kernel does not immediately send bytes to disk. It marks the corresponding page cache page as dirty — meaning the in-memory copy is newer than the on-disk copy — and returns to your program immediately. This is write-back caching, and it is why write() feels instantaneous even for large files.

This is a deliberate trade-off: disk I/O is orders of magnitude slower than RAM writes, so batching many small writes into one larger disk write improves throughput enormously. The danger is that if the machine loses power before the dirty page reaches disk, the data is lost. That is why databases call fsync() after a commit: fsync forces all dirty pages for a given file descriptor to disk and waits for the drive to confirm the write.

How dirty pages accumulate:

1. Your process calls write() or modifies an mmap-ed page.
2. The kernel sets PG_dirty in the page's flags.
3. The page is added to the inode's dirty_pages list and the system-wide writeback accounting updates.

Writeback triggers:

• Time-based: dirty_expire_centisecs (default 3000 = 30 seconds). Pages older than this are eligible for writeback.
• Ratio-based: dirty_ratio (default ~20% of RAM) and dirty_background_ratio (default ~10%). When dirty pages exceed the background ratio, kworker writeback threads begin writing. When they hit the hard ratio, processes are throttled — writes block until dirty pages drain.
• Explicit: fsync(), fdatasync(), sync(), or msync() from user space.
• Memory pressure: The reclaim path (kswapd / direct reclaim) will write back dirty pages before freeing them.

The writeback threads are bdi-default worker threads (one per backing device interface). They call into address_space_operations->writepages() which the filesystem implements.

# Watch dirty/writeback pages live:
watch -n1 'grep -E "Dirty|Writeback" /proc/meminfo'

# Tune background writeback threshold (% of RAM):
cat /proc/sys/vm/dirty_background_ratio   # typically 10
sysctl vm.dirty_background_ratio=5

# Force all pending writeback:
sync

The kernel tracks per-inode and per-BDI (block device interface) writeback state in struct bdi_writeback. The key file is mm/page-writeback.c. Understanding this file is essential for anyone working on filesystem performance.

/* How the kernel marks a page dirty after a write — simplified from mm/filemap.c */
int generic_write_end(struct file *file, struct address_space *mapping,
                      loff_t pos, unsigned len, unsigned copied,
                      struct folio *folio, void *fsdata)
{
    /* Mark the folio (group of pages) as dirty */
    folio_mark_dirty(folio);  /* sets PG_dirty flag */

    /* Account for written bytes, update file size */
    if (pos + copied > inode->i_size)
        i_size_write(inode, pos + copied);

    folio_unlock(folio);
    folio_put(folio);
    return copied;
}

/* folio_mark_dirty eventually calls:
 *   __folio_mark_dirty() -> sets PG_dirty
 *   __mark_inode_dirty()  -> adds inode to writeback list
 */

When memory runs low, the kernel must choose which pages to evict. The ideal eviction candidate is the page that will not be used again for the longest time — the classic Optimal algorithm. Since the future is unknown, Linux approximates it with an LRU (Least Recently Used) policy.

Linux maintains two LRU lists per NUMA node (and per memory zone within that node):

1. Active list: pages that have been accessed recently. "This page is hot, keep it."
2. Inactive list: pages that have not been accessed recently. "This page is a candidate for eviction."

Every page starts its life on the inactive list when first loaded into the page cache. If it is accessed again while still inactive, it is promoted to the active list (the kernel sets the PG_referenced flag on first access; if it is still referenced when the page is next examined by the scanner, it gets promoted).

The page reclaim scanner scans pages from the tail of the inactive list. It will:

• Free anonymous pages by swapping them out (writing to swap space).
• Free file-backed pages by either dropping them (if clean) or writing them back (if dirty) and then dropping them.
• Demote pages from the active list to the inactive list if the active list grows too large.

Why two lists? A single LRU can be "polluted" by a large sequential read that fills the cache with pages accessed exactly once, pushing out truly hot pages. The two-list design (originally from 2Q / CLOCK-Pro concepts) gives recently-accessed pages a second chance before eviction.

Since Linux 5.18+, the LRU machinery has been refactored into multi-gen LRU (MGLRU), which uses multiple generations (like a clock hand over a generational array) for better performance. The core intuition is the same but the implementation is more cache-friendly.

# See LRU page counts per zone:
cat /proc/zoneinfo | grep -A20 'Node 0, zone.*Normal' | grep -E 'nr_inactive|nr_active'

# Or via vmstat:
vmstat -m | head   # slab info
cat /proc/vmstat | grep -E 'pgpgin|pgpgout|pgscan|pgsteal|pgactivate|pgdeactivate'

Key fields in /proc/vmstat:

• pgactivate: pages moved from inactive → active
• pgdeactivate: pages moved from active → inactive
• pgscan_kswapd_*: pages scanned by kswapd
• pgsteal_kswapd_*: pages actually freed by kswapd

/* Kernel representation — include/linux/mmzone.h (simplified) */
struct lruvec {
    struct list_head    lists[NR_LRU_LISTS];
    /* NR_LRU_LISTS = 5:
     *  LRU_INACTIVE_ANON  -- anonymous pages, not recently used
     *  LRU_ACTIVE_ANON    -- anonymous pages, recently used
     *  LRU_INACTIVE_FILE  -- file-backed pages, not recently used
     *  LRU_ACTIVE_FILE    -- file-backed pages, recently used
     *  LRU_UNEVICTABLE    -- pages locked in RAM (mlock, huge pages)
     */
    struct pglist_data  *pgdat;     /* NUMA node this belongs to */
};

/* A page is placed on a list via: */
void lru_cache_add(struct page *page)  /* -> adds to inactive list */
void mark_page_accessed(struct page *page) /* -> may promote to active */

kswapd (kernel swap daemon) is a per-NUMA-node kernel thread whose job is to keep enough free pages available so that normal page allocations never have to wait for reclaim. Think of it as a janitor who empties the trash before the bins overflow, rather than after.

Watermarks: Each memory zone has three watermarks (defined in include/linux/mmzone.h):

• pages_min (MIN): If free pages fall here, the system is in serious trouble. Direct reclaim is triggered for every allocation.
• pages_low (LOW): kswapd wakes up and starts reclaiming.
• pages_high (HIGH): kswapd stops and goes back to sleep.

These watermarks are computed from min_free_kbytes (tunable via /proc/sys/vm/min_free_kbytes). The default is around 1/32 of RAM.

kswapd's loop:

1. Sleep waiting on kswapd_wait waitqueue.
2. Wake when free pages in any zone drop below LOW watermark.
3. Call balance_pgdat() → kswapd_shrink_node() → shrink_lruvec() → scan and free pages.
4. Continue until all zones are above HIGH watermark.
5. Go back to sleep.

What kswapd reclaims:

• File-backed clean pages: Simply removed from the page cache. No I/O needed. These are the cheapest to reclaim.
• File-backed dirty pages: Written back to disk first, then freed. Requires I/O.
• Anonymous pages: Written to the swap partition/file, then freed. Requires I/O and swap space.
• Slab caches: kswapd also calls shrink_slab() to reclaim kernel object caches (dentry cache, inode cache) via shrinker callbacks registered by subsystems.

The swappiness sysctl (default 60, range 0-200 in modern kernels) controls the relative priority of reclaiming anonymous pages vs. file pages. Higher swappiness = more willing to swap anonymous pages; lower = prefer to reclaim file cache first. Setting vm.swappiness=0 does not disable swap; it just makes the kernel very reluctant to use it.

# Watch kswapd activity:
watch -n1 'cat /proc/vmstat | grep -E "kswapd|pgfault|pgmajfault"'

# See current watermarks:
cat /proc/zoneinfo | grep -E 'min|low|high|free' | head -20

# Tune swappiness:
cat /proc/sys/vm/swappiness   # default 60
sudo sysctl vm.swappiness=10  # prefer file cache reclaim

/* kswapd main loop — mm/vmscan.c (heavily simplified) */
static int kswapd(void *p)
{
    pg_data_t *pgdat = (pg_data_t *)p;

    for ( ; ; ) {
        /* Sleep until woken by an allocator that found zone below LOW */
        wait_event_freezable(pgdat->kswapd_wait,
                             pgdat->kswapd_order >= 0);

        /* Reclaim until all zones reach HIGH watermark */
        balance_pgdat(pgdat, pgdat->kswapd_order,
                      pgdat->kswapd_highest_zoneidx);

        pgdat->kswapd_order = -1;   /* mark as done */
    }
    return 0;
}

/* balance_pgdat calls kswapd_shrink_node which calls shrink_lruvec:
 *
 * shrink_lruvec
 *   -> shrink_inactive_list  (scan tail of inactive LRU)
 *       -> shrink_page_list  (decide fate of each page)
 *           -> pageout()     (write dirty page back)
 *           -> or drop from cache (clean page)
 */

Direct reclaim happens when a memory allocation is attempted and the zone's free pages are already below the MIN watermark — so bad that kswapd cannot save it in time. The allocating process itself is forced to do the reclaim work synchronously, blocking its own allocation until enough pages are freed.

This is the kernel's equivalent of "I need memory right now, I'll do the cleaning myself."

The allocation path:

kmalloc / alloc_pages
  → __alloc_pages_nodemask
      → get_page_from_freelist   (fast path: zone has free pages)
      → __alloc_pages_slowpath   (slow path: reclaim needed)
          → wake_all_kswapd()    (ask kswapd to help)
          → __alloc_pages_direct_reclaim
              → try_to_free_pages
                  → shrink_zones
                      → shrink_lruvec (same code as kswapd)

Direct reclaim is expensive because:

1. It introduces latency into the allocating process (your app stalls).
2. If the pages are dirty, the process must wait for writeback I/O to complete.
3. It can cascade: many processes blocking simultaneously, all waiting on the same disk.

Reclaim priority: The scanner starts with a low priority (scans few pages per LRU pass) and escalates if it cannot free enough. Higher priority = more pages scanned per pass. This is the sc->priority field in struct scan_control.

GFP flags (Get Free Pages flags) control whether reclaim is allowed and how urgent it is:

• GFP_KERNEL: normal allocation, reclaim allowed, can sleep
• GFP_ATOMIC: interrupt context, no sleeping, no reclaim — fail immediately if no free pages
• GFP_NOIO: reclaim allowed but no I/O (to avoid deadlock in some filesystem contexts)
• GFP_NOFS: reclaim allowed but no filesystem calls

/* A driver allocating a DMA buffer — can reclaim, can sleep */
void *buf = kmalloc(PAGE_SIZE, GFP_KERNEL);

/* An interrupt handler — cannot sleep, cannot reclaim */
void *buf = kmalloc(64, GFP_ATOMIC);
if (!buf) {
    /* must handle NULL — there's no retry */
    return -ENOMEM;
}

Monitoring direct reclaim:

cat /proc/vmstat | grep -E 'allocstall|pgalloc|pgfault'
# allocstall_dma / allocstall_normal: times a process stalled in direct reclaim
# High values indicate memory pressure — investigate with perf or eBPF

/* Simplified call chain for a failing GFP_KERNEL allocation */
struct page *alloc_pages(gfp_t gfp_mask, unsigned int order)
{
    /* Fast path */
    page = get_page_from_freelist(gfp_mask, order, ...);
    if (page)
        return page;

    /* Slow path — we need to reclaim */
    if (!(gfp_mask & __GFP_DIRECT_RECLAIM))
        return NULL;   /* GFP_ATOMIC: fail fast */

    /* Wake kswapd in case it can help asynchronously */
    wake_all_kswapd(order, gfp_mask, ...);

    /* Try to free pages synchronously, in THIS process context */
    did_some_progress = __alloc_pages_direct_reclaim(gfp_mask, order, ...);

    if (did_some_progress)
        goto retry;

    /* Last resort: OOM killer */
    page = __alloc_pages_may_oom(gfp_mask, order, ...);
    return page;
}

When all reclaim has failed — kswapd has been running, direct reclaim has been tried, swap is full, there is literally no memory to free — the kernel invokes the Out-Of-Memory (OOM) killer. Its job is brutal but necessary: select one process, kill it, and use the freed memory to make progress.

How the OOM killer selects a victim: The kernel scores every process with oom_badness(), which computes:

badness = (RSS + swap_used + children_RSS) / total_RAM * 1000

Adjusted by the process's oom_score_adj (range -1000 to +1000, in /proc/<pid>/oom_score_adj). A value of -1000 makes the process immune; +1000 makes it the first target.

The process with the highest score is sent SIGKILL. The kernel logs a detailed report to dmesg/kernel ring buffer — this is the famous "OOM kill" message that every sysadmin has seen.

OOM groups (cgroup v2): With cgroups, you can configure the OOM killer to kill an entire cgroup (e.g., a container) when any member OOMs, rather than killing individual processes. This is critical for container runtimes.

Diagnosing an OOM kill:

# In dmesg, look for:
dmesg | grep -i 'oom\|killed process'
# You'll see the oom_score of every process, then:
# "Killed process 1234 (some_app) total-vm:..., anon-rss:..., file-rss:..."

# Protect your important service:
echo -500 > /proc/$(pgrep my_service)/oom_score_adj

# Make a throwaway process the first target:
echo 1000 > /proc/$(pgrep bloat)/oom_score_adj

# See current scores:
cat /proc/$(pgrep my_service)/oom_score

Notifier chain: The kernel has an OOM notifier (register_oom_notifier) that subsystems can register callbacks on. This lets, for example, a memory allocator flush its own caches before the OOM killer fires.

The overcommit model is deeply related. Linux by default allows processes to malloc() more virtual memory than physically exists, betting that not all of it will be used at once (overcommit_ratio, overcommit_memory sysctls). When that bet goes wrong — when processes try to use the memory they were promised — the OOM killer is the consequence. Understanding this tension (overcommit enables efficient memory use; OOM killer is the safety valve) is key to reasoning about production memory issues.

/* oom_badness() simplified — mm/oom_kill.c */
long oom_badness(struct task_struct *p, unsigned long totalpages)
{
    long points;
    long adj;

    adj = (long)p->signal->oom_score_adj;
    if (adj == OOM_SCORE_ADJ_MIN) /* -1000: protected */
        return LONG_MIN;

    /* Count pages: resident set + swap usage */
    points = get_mm_rss(p->mm) + get_mm_counter(p->mm, MM_SWAPENTS)
             + mm_pgtables_bytes(p->mm) / PAGE_SIZE;

    /* Scale by oom_score_adj */
    adj *= totalpages / 1000;
    points += adj;

    return points;  /* higher = more likely to be killed */
}

mmap() is the system call that bridges the page cache and virtual memory. Instead of copying bytes through a kernel buffer into user space (as read() does), mmap maps the page cache pages directly into the process's virtual address space. The process accesses file data using ordinary load/store instructions — no system call after the initial mmap.

How it works:

1. mmap(NULL, length, PROT_READ, MAP_SHARED, fd, offset) creates a VMA (Virtual Memory Area, struct vm_area_struct) in the process's address space.
2. The VMA's vm_ops->fault handler is set to filemap_fault.
3. Initially, no physical pages are mapped. The page table entries are empty.
4. When the process first reads address ptr[0], the CPU raises a page fault (hardware exception).
5. The kernel's fault handler (handle_mm_fault → filemap_fault) checks the page cache for the corresponding file page.
   • If found: maps it into the process's page table. The access resumes.
   • If not found: loads it from disk first (major fault = pgmajfault in vmstat), then maps it.
6. Subsequent accesses to the same page go through TLB and page table — no kernel involvement.

MAP_SHARED vs MAP_PRIVATE:

• MAP_SHARED: writes to the mapped region modify the page cache page directly (and mark it dirty). Other processes mapping the same file see the changes. This is how inter-process shared memory via files works.
• MAP_PRIVATE: writes trigger copy-on-write (COW). A private copy of the page is made, the page table remapped to the copy. The original page cache page is unmodified.

Why mmap beats read() for random access:

• read() copies data: disk → page cache → kernel buffer → user buffer. Two copies.
• mmap gives you the page cache page directly: disk → page cache. One copy.
• For random access patterns, mmap avoids the overhead of repeated lseek + read syscalls.

Why mmap loses for sequential I/O:

• read() with large buffers amortizes syscall cost and plays nicely with readahead (the kernel can detect sequential read() calls and prefetch). mmap page faults are harder for the kernel to predict and each fault has overhead.

#include <sys/mman.h>
#include <fcntl.h>
#include <unistd.h>

int fd = open("data.bin", O_RDONLY);
size_t len = 1024 * 1024;  /* 1 MiB */

/* Map 1 MiB of file into virtual address space */
uint8_t *p = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0);

/* This line triggers a page fault -> filemap_fault -> disk I/O (if not cached) */
uint8_t first_byte = p[0];

/* This line hits the TLB/page table — no kernel involvement */
uint8_t second_byte = p[1];

munmap(p, len);
close(fd);

/* Kernel VMA fault handler for file-backed mappings — mm/filemap.c */
vm_fault_t filemap_fault(struct vm_fault *vmf)
{
    struct file *file = vmf->vma->vm_file;
    struct address_space *mapping = file->f_mapping;
    pgoff_t offset = vmf->pgoff;  /* page index within file */
    struct folio *folio;
    vm_fault_t ret = 0;

    /* Look up page in cache */
    folio = filemap_get_folio(mapping, offset);
    if (!folio) {
        /* Cache miss: must read from disk */
        ret = VM_FAULT_MAJOR;   /* counted as major fault */
        folio = do_read_fault(vmf);   /* triggers filesystem read */
    }

    /* Install the physical page into the process page table */
    vmf->page = folio_file_page(folio, offset);
    return ret | VM_FAULT_LOCKED;
}

Every time you call read() on a file sequentially, the kernel notices the pattern and speculatively loads the next pages from disk before you ask for them. This is readahead (also called prefetching), and it is what makes cat bigfile | gzip run at near-disk-bandwidth rather than one-page-at-a-time latency.

The readahead state machine (in mm/readahead.c) tracks:

• ra->start: where the last readahead window began
• ra->size: size of the current readahead window (in pages)
• ra->async_size: how many pages to issue asynchronously ahead of current position

When the kernel detects that reads are sequential, it doubles the readahead window each time the process catches up with the prefetched pages, up to max_readahead_kb (default 128 KiB = 32 pages).

How readahead detects sequential access:

1. Compare the offset of the new read() call with ra->prev_pos (the end of the last read).
2. If they match (or are close), it is sequential — grow the window.
3. If they jump around randomly, readahead is disabled or kept small.

Asynchronous readahead: The kernel issues readahead I/O with READA priority (non-blocking). If a page is needed before the I/O completes, the read blocks — but for sequential workloads, the I/O typically completes well before the data is needed.

Manual hints:

/* Tell the kernel you're about to read this range sequentially */
posix_fadvise(fd, 0, file_size, POSIX_FADV_SEQUENTIAL);
/* Kernel sets readahead window 2x larger */

/* Tell the kernel you'll access randomly — disable readahead */
posix_fadvise(fd, 0, file_size, POSIX_FADV_RANDOM);

/* Prefetch a specific range explicitly */
readahead(fd, offset, count);  /* Linux-specific syscall */

Tuning readahead:

# Per block device readahead (in 512-byte sectors):
blockdev --getra /dev/sda    # get current setting
blockdev --setra 4096 /dev/sda  # set to 2 MiB (4096 * 512 bytes)

# Per mount:
# In /etc/fstab, add: ra=size (in KiB) as a mount option

# kernel's default per-backing-device:
cat /sys/block/sda/queue/read_ahead_kb
echo 256 > /sys/block/sda/queue/read_ahead_kb

Readahead dramatically changes the character of I/O: without it, a sequential file read is a chain of dependent page-fault waits. With it, the disk bandwidth is fully utilized and the process streams data at wire speed. This is why database bulk scans and log processing are so much faster than their single-page IOPS numbers would suggest.

/* ondemand_readahead() — core of the readahead engine (mm/readahead.c, simplified) */
static void ondemand_readahead(struct readahead_control *ractl,
                               bool hit_readahead_marker,
                               unsigned long req_size)
{
    struct file_ra_state *ra = ractl->ra;
    unsigned long max_pages = ra->ra_pages;  /* device readahead limit */

    if (sequential_access(ra, ractl->_index)) {
        /* Sequential: double the window, up to max_pages */
        ra->size = min(ra->size * 2, max_pages);
        ra->start = ractl->_index;
        ra->async_size = ra->size / 2;  /* issue half async, half sync */
    } else {
        /* Random: small readahead, just serve the request */
        ra->size = req_size;
        ra->async_size = 0;
    }

    /* Issue the actual read_folio calls */
    ractl->_nr_pages = ra->size;
    read_pages(ractl);
}

Physical memory is not uniform from the kernel's perspective. It is divided into zones based on hardware constraints and NUMA nodes based on physical topology.

Memory Zones (on x86-64):

• ZONE_DMA: memory below 16 MiB. Ancient ISA DMA devices could only address this range. Rarely relevant on modern hardware.
• ZONE_DMA32: memory below 4 GiB. PCI devices with 32-bit DMA addresses need pages from here.
• ZONE_NORMAL: the main zone — all memory above 4 GiB on 64-bit systems. Most allocations come from here.
• ZONE_MOVABLE: pages that can be migrated (for memory hotplug and huge page compaction).
• ZONE_HIGHMEM: 32-bit x86 only — memory above 896 MiB that couldn't be permanently mapped in kernel space. Obsolete on 64-bit.

Each zone has its own free list (the buddy allocator), its own LRU lists, and its own watermarks. kswapd runs per NUMA node, balancing all zones within that node.

NUMA (Non-Uniform Memory Access): In multi-socket servers, each CPU socket has its own bank of RAM. A CPU can access its local memory faster (lower latency) than a remote socket's memory. The kernel tracks which NUMA node each page belongs to and tries to allocate memory local to the requesting CPU (NUMA_LOCAL policy).

# Check NUMA topology:
numactl --hardware
# Shows nodes, memory per node, distances between nodes

# See page allocation stats per zone:
cat /proc/zoneinfo | head -60

# NUMA memory stats:
numastat -m   # per-node memory usage

For the page cache, NUMA matters because: file pages are allocated on whichever node the reading process runs on. On a NUMA system, two processes on different nodes reading the same file will have separate page cache copies, which can double memory use. The numactl command can pin processes and their memory to specific nodes.

Understanding zones is essential for driver development: a DMA buffer for a 32-bit PCI device must be allocated with GFP_DMA32, not GFP_KERNEL, or it might land above the 4 GiB boundary and be unreachable by the device.

/* Requesting a DMA-capable page for a 32-bit PCI device */
#include <linux/gfp.h>

/* This page is guaranteed to be below 4 GiB physical address */
struct page *dma_page = alloc_page(GFP_KERNEL | GFP_DMA32);
void *vaddr = page_address(dma_page);
dma_addr_t bus_addr = dma_map_page(dev, dma_page, 0, PAGE_SIZE,
                                    DMA_FROM_DEVICE);

/* Later, unmap and free */
dma_unmap_page(dev, bus_addr, PAGE_SIZE, DMA_FROM_DEVICE);
__free_page(dma_page);

/* For normal kernel allocations — use GFP_KERNEL, zone chosen automatically */
void *buf = kmalloc(1024, GFP_KERNEL);

Swap space is disk space (a partition or file) that acts as a spillover for physical RAM. When the kernel needs to reclaim an anonymous page (a page not backed by a file — heap, stack, private mappings), it cannot simply discard it because there is no file to reload it from. Instead, it writes the page to swap and frees the physical page. When the process accesses that address again, a major page fault fires, and the kernel reads the page back from swap.

Swap is not the page cache, but it uses a very similar architecture:

• Swap is managed through struct swap_info_struct, with each swap slot indexed like a file offset.
• Swapped-out pages are tracked by swap entries in the page table: instead of a physical page, the PTE contains a swap type and offset (the kernel recognizes this by the PTE's !present + !none bits).
• When a swapped page is read back, it goes through the swap cache (a temporary in-memory cache) before being placed in the process's page table.

Swap priority and multiple swap devices:

# Add a swap file:
fallocate -l 4G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

# See swap usage:
swapon --show
free -h

# With two swap devices, set priority (higher number = used first):
swapon -p 100 /dev/sdb1   # fast SSD swap, use first
swapon -p 10  /dev/sdc1   # slow HDD swap, overflow

zswap and zram: Modern systems increasingly use zswap (a compressed in-DRAM swap cache that stores compressed pages instead of going to disk) and zram (a block device backed by compressed RAM, used as the swap device itself). On low-memory devices (phones, embedded), zram is the only swap and avoids disk I/O entirely.

# Check if zswap is active:
cat /sys/module/zswap/parameters/enabled

# Enable zswap:
echo 1 > /sys/module/zswap/parameters/enabled

# zswap statistics:
cat /sys/kernel/debug/zswap/pool_total_size
cat /sys/kernel/debug/zswap/stored_pages

The interaction between swap and the page cache creates an interesting nuance: a page that is in swap AND has been read back into memory but not yet freed from the swap slot is said to be in the swap cache. This prevents races where two processes fault on the same swap page simultaneously and try to read it twice.

/* How the kernel handles a swap fault — mm/memory.c (simplified) */
vm_fault_t do_swap_page(struct vm_fault *vmf)
{
    struct vm_area_struct *vma = vmf->vma;
    swp_entry_t entry = pte_to_swp_entry(*vmf->pte); /* decode swap slot */
    struct folio *folio;

    /* Check swap cache first (avoid double-read) */
    folio = swap_cache_get_folio(entry, vma, vmf->address);
    if (!folio) {
        /* Not in swap cache: read from disk */
        folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
        /* This issues disk I/O — this is a major fault */
    }

    /* Install the page into the process page table */
    set_pte_at(vma->vm_mm, vmf->address, vmf->pte,
               mk_pte(folio_page(folio, 0), vma->vm_page_prot));

    /* Increment RSS, decrement swap usage */
    update_rss_counter(vma->vm_mm, folio);

    return 0;
}