R0KernelPath
module 074 / 262

Kernel Memory: Buddy, Slab, kmalloc/vmalloc

The kernel's memory allocators are a masterpiece of layered engineering — from raw pages to object caches — and understanding them is the foundation of all kernel programming.

~18h focused study9 concepts

Overview

Every program running on your machine — every browser tab, every shell command, every kernel driver — needs memory. But memory is finite, and the kernel is the ultimate arbiter of who gets what. Unlike a userspace allocator like malloc, the kernel's memory management must handle wildly different allocation sizes (from a 4-byte integer to gigabytes of device buffers), must work under interrupt context where sleeping is forbidden, must never deadlock, and must do all of this in microseconds.

This module takes you inside the Linux kernel's memory management subsystem — a layered architecture of elegant algorithms built on top of one another. At the bottom sits the buddy allocator, which carves physical RAM into power-of-two page blocks with a deceptively simple bookkeeping trick. On top of that sits the slab allocator (and its modern replacement, SLUB), which solves the "lots of tiny objects" problem that would make the buddy system hideously wasteful. Finally, the high-level interfaces — kmalloc, vmalloc, alloc_pages — give kernel developers the right tool for each job.

Understanding this system is not just an academic exercise. Every time you write kernel code that allocates memory — and almost all non-trivial kernel code does — you must choose the right allocator, the right flags, and respect the right context. A wrong choice causes kernel panics, memory corruption, or silent performance disasters. A right choice is elegant, safe, and blazingly fast.

The beauty here is architectural: each layer solves exactly the problems the layer below it cannot, without duplicating what the layer below already does well. This is systems programming philosophy made concrete in ~50,000 lines of C.

why this matters for the kernel

Nearly every kernel subsystem you will ever touch — drivers, filesystems, networking, scheduling — allocates kernel memory. When you write your first non-trivial kernel module, you will call kmalloc. When you look at a kernel crash dump, you will see slab cache names. When you read a kernel patch, you will see GFP flags. Without understanding what these mean and why they exist, you are copy-pasting incantations you do not understand — the most dangerous position in systems programming.

More concretely, memory management bugs are among the most catastrophic in the kernel: use-after-free vulnerabilities have been the basis of dozens of real-world privilege escalation exploits (CVE-2016-0728, Dirty COW variants, and many more). Understanding the allocator internals teaches you not just how to write correct code, but why certain patterns are dangerous and how security mitigations like KASAN, SLUB's freelist randomization, and KASLR interact with the allocator. This knowledge directly transfers to reading and writing security-sensitive kernel patches — a highly valued skill in the kernel community.

Linux Kernel Memory Allocation Layers

rendering diagram…

The layered memory allocation hierarchy: high-level APIs (kmalloc, vmalloc, kmem_cache_alloc) are built on SLUB, which is built on the buddy allocator, which manages physical RAM organized into zones. Memory pressure triggers reclaim which feeds pages back into the buddy system.

The concepts

01Physical Memory Layout: Zones and the page struct

Before the kernel can allocate anything, it needs a data structure to represent every physical page of RAM. That structure is struct page, defined in include/linux/mm_types.h. There is exactly one struct page for every physical page frame on the system. On a machine with 8 GB of RAM and 4 KB pages, there are 2 million struct page objects, all stored in a flat array called mem_map.

But not all physical memory is equal. The kernel divides RAM into zones, each describing a region with different capabilities:

  • ZONE_DMA (0–16 MB on x86): old ISA devices can only DMA into this range.
  • ZONE_DMA32 (0–4 GB on x86-64): 32-bit PCI devices that can't address above 4 GB.
  • ZONE_NORMAL: the workhorse — directly mapped into the kernel's virtual address space. On 64-bit systems this is essentially all RAM.
  • ZONE_HIGHMEM (x86-32 only): physical RAM above ~896 MB that cannot be permanently mapped into the 32-bit kernel virtual address space. This zone is largely irrelevant on 64-bit systems.
  • ZONE_MOVABLE: pages that can be migrated (used by memory hotplug and compaction).

Each zone has a watermark system: WMARK_MIN, WMARK_LOW, and WMARK_HIGH. The number of free pages in a zone is constantly monitored. When free pages fall below WMARK_LOW, kswapd (the kernel swap daemon) wakes up and starts reclaiming pages. Below WMARK_MIN, direct reclaim begins — the allocating process itself does the reclaim work before its allocation can succeed.

The struct page is an extraordinarily complex union of fields, because each page can be in one of many states: a file-backed page in the page cache, an anonymous page backing a process's heap, a page table page, a slab page, a compound page (part of a huge page), etc. The kernel reuses struct page fields for different meanings depending on context, making it one of the most "creative" data structures in the codebase.

/* Simplified struct page — the real one has ~60 fields in unions */
struct page {
    unsigned long flags;      /* PG_locked, PG_dirty, PG_uptodate, ... */
    union {
        struct address_space *mapping;  /* file-backed: points to inode */
        void *s_mem;                    /* slab: first object in slab */
    };
    union {
        pgoff_t index;          /* file offset within mapping */
        unsigned long private; /* used by slab/compound pages */
    };
    atomic_t _refcount;         /* page reference count */
    /* ... many more fields in unions ... */
};

/* Every page has an index in mem_map[]. To get the struct page
 * for a physical address: */
struct page *page = pfn_to_page(phys_addr >> PAGE_SHIFT);
/* And back: */
unsigned long pfn = page_to_pfn(page);
phys_addr_t pa  = (phys_addr_t)pfn << PAGE_SHIFT;

02The Buddy Allocator: Splitting and Coalescing

The buddy allocator is the kernel's primary physical page allocator. It manages free pages in orders: order-0 is a single page (4 KB), order-1 is two contiguous pages (8 KB), order-2 is four contiguous pages (16 KB), ..., order-10 is 1024 contiguous pages (4 MB). Each order has a free list.

The core insight is the buddy relationship: when you split an order-N block into two order-(N-1) blocks, the two halves are "buddies". Their physical addresses differ only in bit (N-1). This means given a block's address and order, you can compute its buddy's address with a single XOR.

Allocation: To allocate an order-2 block (4 pages), the kernel looks at the order-2 free list. If it's empty, it goes to order-3, splits that block into two order-2 buddies, puts one on the free list, and returns the other. If order-3 is also empty, it goes to order-4, splits it into two order-3 blocks, puts one on the order-3 free list, then splits the other into two order-2 blocks, etc. This cascades up until a free block is found or allocation fails.

Freeing: When you free an order-2 block, the kernel checks if its buddy is also free. If so, they are merged (coalesced) into an order-3 block. Then it checks if THAT block's buddy is free, merging again if so. This cascades up, potentially all the way to the maximum order.

This algorithm is extremely fast (O(log N) worst case) and naturally combats external fragmentation — the buddy coalescing process continuously reassembles large contiguous blocks. The /proc/buddyinfo file shows the current state of the buddy system in real time.

The buddy allocator is exposed through alloc_pages(gfp_mask, order) and __get_free_pages(gfp_mask, order) / free_pages(addr, order). But for most kernel code, you don't call it directly — you go through slab or kmalloc.

/* /proc/buddyinfo example output:
   Node 0, zone   Normal  4096  2048  1024  512  256  128  64  32  16   8   4
                  ^order0  ^o1   ^o2  ... each number = free blocks of that order */

/* Computing the buddy address (kernel internal macro): */
#define __find_buddy_pfn(page_pfn, order) \
    ((page_pfn) ^ (1 << (order)))

/* To allocate 8 contiguous pages (order 3) from ZONE_NORMAL: */
#include <linux/gfp.h>

struct page *pages = alloc_pages(GFP_KERNEL, 3); /* order=3 → 2^3=8 pages */
if (!pages) {
    /* allocation failed — must handle this */
    return -ENOMEM;
}
unsigned long addr = page_address(pages); /* kernel virtual address */

/* ... use the memory ... */

__free_pages(pages, 3); /* MUST pass the same order! */

03The Slab Allocator: Object Caches

The buddy allocator allocates in units of pages (multiples of 4 KB). But most kernel allocations are tiny: a struct task_struct is ~9 KB, a struct inode is ~600 bytes, a struct sk_buff is ~240 bytes. If you allocated a full page every time you needed a struct inode, you'd waste ~3500 bytes per inode — catastrophic on a filesystem with millions of files.

The slab allocator, introduced by Jeff Bonwick at Sun in 1994 and adopted by Linux in 1996, solves this with object caches. The idea:

  1. For each frequently allocated type (say struct inode), create a dedicated cache (a struct kmem_cache).
  2. The cache holds one or more slabs — each slab is one or more contiguous pages from the buddy allocator, pre-divided into fixed-size slots for objects of that type.
  3. Allocating an object means finding a free slot in a slab — often just a pointer pop from a free list, an O(1) operation.
  4. Freeing means returning the slot to the slab's free list.
  5. When a slab is fully free (all objects returned), the pages can be given back to the buddy allocator.

Three benefits beyond size efficiency:

  • Constructor/destructor support: an object's invariant fields (like mutex initialization) can be set up once when the slab is created and maintained across free/alloc cycles, since the object isn't zeroed between uses (unless you ask for it).
  • CPU caches: freed objects stay warm in CPU cache. The next allocation gets a cache-hot object.
  • Per-CPU magazines: modern slab implementations (SLUB especially) use per-CPU free lists so allocation/free on the same CPU requires no locking at all in the common case.

Linux has had three slab implementations: the original SLAB (now removed in 6.8+), SLUB (default since ~2.6.23, the current standard), and SLOB (a minimal allocator for tiny embedded systems, also removed in 6.4). SLUB simplified the design by eliminating many of SLAB's per-slab metadata structures and embedding metadata directly in the objects themselves when they are free.

You can inspect slab caches with /proc/slabinfo or the slabtop command.

/* Creating and using a custom slab cache in a kernel module */
#include <linux/slab.h>

struct my_object {
    int id;
    char name[32];
    spinlock_t lock;
};

static struct kmem_cache *my_cache;

int __init my_module_init(void)
{
    /* Create a cache for my_object.
     * SLAB_HWCACHE_ALIGN: align to hardware cache line.
     * NULL: no constructor (use if you'd set up locks once here). */
    my_cache = kmem_cache_create("my_object_cache",
                                  sizeof(struct my_object),
                                  0,                /* align: use natural */
                                  SLAB_HWCACHE_ALIGN,
                                  NULL);            /* ctor */
    if (!my_cache)
        return -ENOMEM;
    return 0;
}

void my_function(void)
{
    struct my_object *obj = kmem_cache_alloc(my_cache, GFP_KERNEL);
    if (!obj)
        return;

    obj->id = 42;
    spin_lock_init(&obj->lock);
    /* ... use obj ... */

    kmem_cache_free(my_cache, obj);
}

void __exit my_module_exit(void)
{
    kmem_cache_destroy(my_cache);
}

04kmalloc: The General-Purpose Kernel Allocator

kmalloc(size, gfp_flags) is the kernel's equivalent of userspace malloc. It allocates a contiguous block of physically contiguous kernel virtual memory of at least size bytes, returning a pointer directly usable by kernel code.

Internally, kmalloc is backed by a set of pre-sized slab caches: kmalloc-8, kmalloc-16, kmalloc-32, ..., kmalloc-1024, kmalloc-2048, kmalloc-4096, kmalloc-8192. A call to kmalloc(100, GFP_KERNEL) goes to the kmalloc-128 cache (the smallest cache large enough). For sizes above KMALLOC_MAX_CACHE_SIZE (usually 8 KB or 16 KB), kmalloc falls back to alloc_pages directly.

Physical contiguity: memory returned by kmalloc is both physically and virtually contiguous. This is critical for DMA operations — a hardware device performing DMA needs physically contiguous memory because it sees the raw physical address bus, not the kernel's virtual address space.

Size limitation: because it must be physically contiguous, kmalloc has a practical size limit (usually 4–8 MB, depending on the buddy system's maximum order and fragmentation). For large allocations, use vmalloc instead.

kzalloc: a convenience wrapper that calls kmalloc then memsets to zero. Prefer this for security-sensitive allocations — uninitialized kernel memory leaking to userspace is a classic info-leak vulnerability.

krealloc: analogous to realloc, but be careful — the returned pointer may differ from the input, and the old memory is freed.

Freeing: always use kfree(ptr) where ptr is exactly what kmalloc returned. Freeing with an offset pointer corrupts the slab. kfree(NULL) is always safe.

#include <linux/slab.h>
#include <linux/string.h>

int example_kmalloc(void)
{
    /* Basic allocation — may sleep (GFP_KERNEL) */
    char *buf = kmalloc(256, GFP_KERNEL);
    if (!buf)
        return -ENOMEM;

    memset(buf, 0, 256);   /* or: use kzalloc(256, GFP_KERNEL) */
    strcpy(buf, "hello kernel");

    /* ... */

    kfree(buf);   /* buf pointer must be exactly what kmalloc returned */
    buf = NULL;   /* good practice: avoid use-after-free bugs */
    return 0;
}

/* In interrupt context: MUST use GFP_ATOMIC (no sleeping allowed) */
irqreturn_t my_irq_handler(int irq, void *dev_id)
{
    /* GFP_ATOMIC: will NOT sleep; may fail under memory pressure */
    void *tmp = kmalloc(64, GFP_ATOMIC);
    if (!tmp)
        return IRQ_HANDLED;  /* handle gracefully */
    /* ... */
    kfree(tmp);
    return IRQ_HANDLED;
}

/* Typed allocation helper (preferred modern style): */
struct my_device *dev = kzalloc(sizeof(*dev), GFP_KERNEL);
/* sizeof(*dev) is idiomatic — correct even if the type changes */

05vmalloc: Virtually Contiguous, Physically Scattered

vmalloc(size) allocates memory that is virtually contiguous in the kernel's vmalloc address space but whose pages may be physically scattered throughout RAM. Each page is independently allocated from the buddy system and then mapped into a contiguous range of kernel virtual addresses using page tables.

The vmalloc address space is a special region of the kernel's virtual address space (VMALLOC_START to VMALLOC_END). On x86-64, this is a multi-terabyte region between the directly mapped physical memory area and the kernel module text area.

When to use vmalloc:

  • Large allocations (megabytes to gigabytes) where physical contiguity is not needed.
  • Loading kernel modules (module text and data live in vmalloc space).
  • Allocating large buffers where kmalloc would fail due to fragmentation.

When NOT to use vmalloc:

  • For DMA: hardware can't DMA to virtually contiguous but physically scattered memory (you'd need to use an IOMMU or DMA scatter-gather lists).
  • For performance-critical hot paths: vmalloc memory incurs TLB misses because each page has a separate page table entry. With kmalloc's physically contiguous memory, the kernel's direct mapping uses huge pages (2 MB or 1 GB pages on x86-64), making TLB coverage much more efficient.
  • For small allocations: the overhead of setting up page table mappings makes vmalloc inefficient for small sizes.

vfree(ptr) must be used to free vmalloc memory — never kfree.

vzalloc(size): zero-initializes, like kzalloc for vmalloc.

A useful mental model: kmalloc is like reserving seats in one continuous block in a theater (all together), while vmalloc is like reserving scattered individual seats and creating a virtual "row" by numbering them.

#include <linux/vmalloc.h>

int load_big_firmware(size_t fw_size)
{
    /* fw_size might be 10 MB — kmalloc would likely fail */
    void *fw_buf = vmalloc(fw_size);
    if (!fw_buf)
        return -ENOMEM;

    /* memory is zeroed? No — use vzalloc if you need that */
    memset(fw_buf, 0, fw_size);

    /* Use it — BUT: can't pass this to dma_map_single()!
     * The pages are physically scattered. */
    copy_from_user(fw_buf, user_ptr, fw_size);

    /* ... process firmware ... */

    vfree(fw_buf);   /* NOT kfree! */
    return 0;
}

/* You can check if an address is in vmalloc space: */
if (is_vmalloc_addr(ptr)) {
    vfree(ptr);
} else {
    kfree(ptr);
}

/* vmalloc_to_page(): get struct page for a vmalloc address
 * (useful for building scatter-gather lists manually) */
struct page *p = vmalloc_to_page(fw_buf + offset);

06GFP Flags: Telling the Allocator Your Context

GFP stands for Get Free Pages — the flags passed to all kernel memory allocation functions. They tell the allocator what it is allowed to do to satisfy the allocation. Getting GFP flags wrong is one of the most common sources of kernel bugs.

The flags fall into several categories:

Context flags (the most important):

  • GFP_KERNEL: the standard flag for normal kernel code running in process context. The allocator may sleep — it can block waiting for memory to be reclaimed, pages to be written back, or swap space to be used. Most allocations use this.
  • GFP_ATOMIC: for interrupt handlers, spinlock-held code, or any context where sleeping is forbidden. The allocator will never sleep. It uses a small emergency reserve pool. Will fail (return NULL) under memory pressure rather than waiting. Has a higher failure rate than GFP_KERNEL.
  • GFP_NOWAIT: like GFP_ATOMIC but does not use the emergency reserves. Even more likely to fail.
  • GFP_NOIO: may not initiate I/O. Used by block layer and storage drivers to prevent recursive I/O deadlocks.
  • GFP_NOFS: may not call into the filesystem. Used by filesystems themselves to prevent deadlocks when allocating while holding filesystem locks.

Zone modifiers:

  • __GFP_DMA: allocate from ZONE_DMA (0–16 MB).
  • __GFP_DMA32: allocate from ZONE_DMA32 (0–4 GB).
  • __GFP_HIGHMEM: allow allocation from ZONE_HIGHMEM (32-bit only).

Behavior flags:

  • __GFP_ZERO: zero the memory before returning (like kzalloc).
  • __GFP_NOWARN: don't print a warning if allocation fails (useful when failure is expected and handled).
  • __GFP_RETRY_MAYFAIL: try harder but eventually give up (for large allocations).
  • __GFP_NOFAIL: allocation MUST succeed, keep trying forever. Use with extreme care — can deadlock the system.
  • __GFP_COMP: create a compound page (used for huge pages).

GFP_KERNEL is actually a combination: __GFP_RECLAIM | __GFP_IO | __GFP_FS, meaning it can trigger all forms of memory reclaim.

The key rule: identify your execution context first, then choose your GFP flags. If you are in interrupt context (in_interrupt() returns true), you MUST use GFP_ATOMIC. If you hold a spinlock, you MUST use GFP_ATOMIC. Otherwise, GFP_KERNEL is almost always correct.

/* GFP_KERNEL: normal process context, can sleep */
struct my_buf *b = kmalloc(sizeof(*b), GFP_KERNEL);

/* GFP_ATOMIC: interrupt/spinlock context */
spin_lock_irqsave(&dev->lock, flags);
struct event *ev = kmalloc(sizeof(*ev), GFP_ATOMIC);
spin_unlock_irqrestore(&dev->lock, flags);

/* DMA-capable buffer: must be physically contiguous AND
 * addressable by the device */
void *dma_buf = kmalloc(4096, GFP_KERNEL | GFP_DMA32);

/* Zero-initialized, process context: */
struct config *cfg = kzalloc(sizeof(*cfg), GFP_KERNEL);
/* kzalloc(sz, flags) == kmalloc(sz, flags | __GFP_ZERO) */

/* Inside a filesystem write path — no FS recursion allowed: */
void *tmp = kmalloc(512, GFP_NOFS);

/* Check if sleeping is allowed (defensive programming): */
if (in_interrupt() || in_atomic())
    ptr = kmalloc(size, GFP_ATOMIC);
else
    ptr = kmalloc(size, GFP_KERNEL);

/* might_sleep() — debug assertion that sleeping is safe here.
 * Triggers a warning if called from atomic context.          */
void my_init_path(void) {
    might_sleep();
    ptr = kmalloc(big_size, GFP_KERNEL);
}

07alloc_pages and the Low-Level Page Interface

alloc_pages(gfp_mask, order) is the raw interface to the buddy allocator — it returns a struct page * pointing to the first page of a contiguous block of 2^order pages. This is the foundation everything else is built on.

Typically you wouldn't call alloc_pages directly unless you need to work with struct page pointers directly — for example, when building page tables, managing memory mappings, or implementing a custom allocator. Most code uses kmalloc or slab interfaces instead.

Key functions:

  • alloc_pages(gfp_mask, order) — allocate 2^order contiguous pages, return struct page *.
  • alloc_page(gfp_mask) — convenience for alloc_pages(gfp_mask, 0), single page.
  • __get_free_pages(gfp_mask, order) — like alloc_pages but returns the kernel virtual address directly (using the direct mapping). Cannot be used for HIGHMEM pages.
  • get_zeroed_page(gfp_mask) — allocate one page, zeroed.
  • __free_pages(page, order) — free pages, takes struct page * and order.
  • free_pages(addr, order) — free pages by virtual address.

Page address translation: on 64-bit Linux, all physical RAM is directly mapped into the kernel virtual address space at a fixed offset (PAGE_OFFSET, typically 0xffff888000000000 on x86-64). So the virtual address of a page is simply page_to_virt(page) or equivalently __va(page_to_phys(page)). The kmap() / kunmap() functions exist for HIGHMEM compatibility on 32-bit systems but are rarely needed on 64-bit.

Compound pages: when order > 0, the first page is the head page and the rest are tail pages. They form a compound page. The tail pages have a special PG_tail flag and a pointer back to the head. Compound pages are the building block of hugepages (2 MB on x86-64 uses order-9 compound pages).

#include <linux/gfp.h>
#include <linux/mm.h>

/* Allocate a single page, get its virtual address */
void example_page_alloc(void)
{
    struct page *page = alloc_page(GFP_KERNEL);
    if (!page)
        return;

    /* Convert to kernel virtual address */
    void *vaddr = page_address(page);
    memset(vaddr, 0, PAGE_SIZE);  /* PAGE_SIZE = 4096 on most archs */

    /* ... use the page ... */

    __free_page(page);  /* same as __free_pages(page, 0) */
}

/* Allocate 16 contiguous pages (order 4, 64 KB) */
void example_order4(void)
{
    struct page *pages = alloc_pages(GFP_KERNEL, 4);
    if (!pages)
        return;

    /* page_address() works because it's in ZONE_NORMAL (direct map) */
    unsigned long addr = (unsigned long)page_address(pages);

    /* These are a compound page — head + 15 tails */
    BUG_ON(!PageCompound(pages));   /* will be compound for order>0 */

    __free_pages(pages, 4);  /* MUST pass the same order */
}

/* Check a page's flags */
void inspect_page(struct page *p)
{
    printk("dirty=%d locked=%d referenced=%d uptodate=%d\n",
           PageDirty(p), PageLocked(p),
           PageReferenced(p), PageUptodate(p));
}

08Memory Reclaim: Keeping the System Alive Under Pressure

Allocation is only half the story. When free memory runs low, the kernel must reclaim pages — find pages that are in use but can be freed or written to swap. This is memory reclaim, and it is one of the most complex subsystems in the kernel.

The Page Cache: the kernel aggressively caches file data in RAM. When you read a file, its pages stay in the page cache so the next read hits RAM instead of disk. Under memory pressure, these clean (unmodified) pages can be freed instantly — the data is already on disk. Dirty (modified) pages must be written back first.

Anonymous pages: heap and stack memory not backed by a file. To reclaim these, the kernel must write them to swap space (a disk partition or file) and unmap them from the process's page tables.

The LRU lists: the kernel maintains Least Recently Used lists for pages: active_list (recently accessed) and inactive_list (not recently accessed). Pages migrate between these lists based on access patterns (tracked via the PG_referenced and PG_active flags, and hardware accessed bits in PTEs). Reclaim preferentially targets the inactive list.

kswapd: a per-NUMA-node kernel thread that wakes up when free memory falls below WMARK_LOW. It scans LRU lists and reclaims pages asynchronously in the background. This is the "gentle" reclaim path.

Direct reclaim: when an allocation cannot be satisfied even after kswapd has been running, the allocating process itself enters direct reclaim — it synchronously does reclaim work before its alloc_pages call can return. This makes the allocation slow but keeps the system alive.

OOM killer: if all reclaim efforts fail and memory is truly exhausted, the Out-Of-Memory killer selects a process to kill based on a scoring algorithm (favoring large, low-priority, non-root processes). You can inspect the OOM score at /proc/<pid>/oom_score. This is the kernel's last resort.

The interplay between allocator pressure, watermarks, kswapd, direct reclaim, and the OOM killer is what keeps a Linux system responsive even under extreme memory pressure — a remarkable engineering achievement.

/* Check current memory pressure from user space */
/* $ cat /proc/meminfo | grep -E 'MemFree|Active|Inactive|SwapFree'
   MemFree:        1024000 kB
   Active:         3072000 kB
   Inactive:       2048000 kB
   SwapFree:        512000 kB  */

/* Kernel: trigger memory compaction (requires root)
   $ echo 1 > /proc/sys/vm/compact_memory

   Force a drop of page cache (for testing, not production!):
   $ sync && echo 3 > /proc/sys/vm/drop_caches
   (1=pagecache, 2=dentries+inodes, 3=all)

   Watch kswapd activity:
   $ vmstat 1                     # si/so columns = swap in/out pages/sec
   $ cat /proc/vmstat | grep pgscank   # pages scanned by kswapd
*/

/* From kernel code: check if direct reclaim is happening */
#include <linux/sched.h>

void check_reclaim_context(void)
{
    if (current->flags & PF_MEMALLOC) {
        /* We are in a reclaim context — must use GFP_NOFS or
         * GFP_NOIO to avoid recursive reclaim deadlocks */
        printk("in reclaim context!\n");
    }
}

09Memory Debugging: KASAN, SLUB Debug, and /proc interfaces

Memory bugs — use-after-free, out-of-bounds access, double-free — are among the most dangerous and hardest to debug in kernel code. The kernel provides powerful built-in tools.

KASAN (Kernel Address SANitizer): a compile-time and runtime tool that detects out-of-bounds and use-after-free bugs. It works by adding shadow memory — one byte of shadow for every eight bytes of real memory, encoding whether those bytes are accessible. Every load/store is instrumented to check the shadow. When a violation occurs, KASAN prints a detailed report with stack traces for both the current access and the earlier free/alloc. Enable with CONFIG_KASAN=y.

SLUB debug (CONFIG_SLUB_DEBUG=y): SLUB can store red zones (canary values) around objects and free-object poison (0x6b fill on free, 0x5a fill on alloc). If an object is written after freeing, or overflows its boundary, the next alloc/free of that object detects the corruption. Enable per-cache with the slub_debug= kernel boot parameter.

/proc/slabinfo: shows all slab caches — name, number of active objects, total objects, object size, slabs used. The slabtop command provides a live top-like view sorted by memory usage.

/proc/buddyinfo: shows free blocks in the buddy system per zone per order. If high orders are all zero, your system suffers from memory fragmentation.

/proc/meminfo: the canonical memory statistics view — total, free, available, cached, buffers, swap, hugepages, etc.

/proc/vmstat: low-level VM statistics — page fault counts, kswapd scan/reclaim counts, THP promotion/fallback counts, compaction statistics.

KMEMLEAK: tracks all allocations and their reference graph; reports allocations with no live references after GC cycles — kernel memory leaks. Enabled with CONFIG_DEBUG_KMEMLEAK=y.

# Build a debug kernel with KASAN:
# In .config:
# CONFIG_KASAN=y
# CONFIG_KASAN_INLINE=y  (faster than CONFIG_KASAN_OUTLINE)
# CONFIG_KASAN_GENERIC=y

# KASAN report example (truncated):
# BUG: KASAN: slab-out-of-bounds in my_function+0x4c/0x80
# Write of size 4 at addr ffff888102345678 by task kworker/0:1/42
# Allocated by task 42:
#   kmalloc+0x52/0x80
#   my_function+0x28/0x80
# The buggy address belongs to the object at ffff888102345670
#  which belongs to the cache kmalloc-64 of size 64

# Enable SLUB red zones and poisoning at boot:
# slub_debug=PZF   (P=poison, Z=red zone, F=track free)
# or per-cache: slub_debug=PZF,kmalloc-64

# Check slab cache memory usage:
$ sudo slabtop -o  # one-shot output, sorted by size

# Find the biggest slab consumers:
$ sudo cat /proc/slabinfo | sort -k3 -n -r | head -20

# Watch memory stats live:
$ watch -n1 'cat /proc/meminfo | head -30'

# kmemleak scan (requires CONFIG_DEBUG_KMEMLEAK=y):
$ echo scan > /sys/kernel/debug/kmemleak
$ cat /sys/kernel/debug/kmemleak

Key insights

  • ✦The buddy allocator solves external fragmentation through coalescing — every free operation is an opportunity to rebuild larger blocks, and the XOR trick makes buddy identification O(1).
  • ✦Slab allocators exist because the buddy system is too coarse-grained: they trade a small amount of internal fragmentation for dramatically better CPU cache utilization and near-zero allocation cost in the common case.
  • ✦GFP flags encode your execution context, not just a hint — passing GFP_KERNEL from interrupt context is a correctness bug that will eventually deadlock or corrupt memory, not just a performance issue.
  • ✦Physical contiguity (kmalloc) and virtual contiguity (vmalloc) are different things with different costs: kmalloc is fast but limited in size; vmalloc is unlimited but has per-page TLB overhead and cannot be used for DMA.
  • ✦The watermark system (WMARK_MIN/LOW/HIGH) is what makes Linux feel responsive even under extreme memory pressure — reclaim is proactive, not reactive, so allocations rarely stall.
  • ✦Every struct page is ~64 bytes of metadata for a 4096-byte page — that's a 1.5% overhead just for bookkeeping, which on a 256 GB server is 4 GB of RAM just for page descriptors.
  • ✦Memory reclaim is fundamentally about policy — the LRU heuristic is an approximation of 'which pages are least valuable', and the OOM killer's scoring is deliberately not fully deterministic to prevent gaming.
  • ✦KASAN's shadow memory model (1 byte per 8 bytes) imposes ~12.5% memory overhead and ~2x slowdown — entirely acceptable for development but disabled in production kernels.

Common pitfalls

  • ⚠Using GFP_KERNEL while holding a spinlock — this can sleep, which deadlocks because the scheduler cannot context-switch while a spinlock is held. Always use GFP_ATOMIC in atomic context.
  • ⚠Freeing with the wrong order in alloc_pages/__free_pages — if you allocated order-3 but free with order-2, you corrupt the buddy allocator's free lists silently. There is no runtime check.
  • ⚠Mixing kfree and vfree — passing a vmalloc pointer to kfree (or vice versa) causes immediate kernel panic or silent heap corruption. Use is_vmalloc_addr() if the origin is uncertain.
  • ⚠Forgetting to check the return value of kmalloc/vmalloc — both can return NULL under memory pressure. Dereferencing NULL in kernel context causes an immediate kernel oops with no recovery.
  • ⚠Using kmalloc for large allocations (> ~128 KB) — these often fail due to fragmentation even when plenty of total RAM is free. Use vmalloc for large non-DMA buffers.
  • ⚠Holding a reference to a slab object after calling kmem_cache_free — SLUB will immediately reuse the slot, causing use-after-free with no immediate crash but data corruption later.
  • ⚠Using __GFP_NOFAIL carelessly — this flag tells the kernel to retry forever, which can deadlock the machine if the allocation truly cannot succeed (e.g., trying to allocate DMA32 memory when ZONE_DMA32 is exhausted).
  • ⚠Not accounting for NUMA in performance-critical code — alloc_pages may satisfy a request from a remote NUMA node, causing all memory accesses to cross an interconnect. Use GFP_THISNODE or alloc_pages_node() in latency-sensitive paths.

Practice

  1. 01
    medium

    Write a kernel module that creates a custom slab cache for a struct containing a 64-byte payload and a spinlock. Allocate 1000 objects, fill them with incrementing values, verify the values, then free them all. Use KASAN in your test kernel to verify no memory errors occur. Time the allocation loop with ktime_get() and print the result.

  2. 02
    medium

    Write a kernel module that demonstrates the buddy allocator directly: allocate pages at orders 0 through 10 (if available), print the physical addresses and verify that the buddy address relationship holds (pfn XOR (1<<order) gives the buddy pfn). Free them all correctly.

  3. 03
    hard

    Boot a Linux kernel with CONFIG_KASAN=y in QEMU. Write a kernel module that intentionally performs an out-of-bounds write (write one byte past the end of a kmalloc-32 allocation). Capture the KASAN report, identify the 'allocated by' and 'accessed by' stack traces, and explain every line of the report in a comment in your module source.

  4. 04
    easy

    From user space, write a C program that allocates memory in a loop until the system is under pressure, then examine /proc/buddyinfo, /proc/meminfo, and /proc/vmstat every second to observe kswapd waking up, free pages dropping through watermarks, and swap usage increasing. Plot the data with gnuplot or Python.

  5. 05
    hard

    Implement a minimal slab-like allocator entirely in user space (no kernel code) using mmap for page acquisition and a free list per size class (8, 16, 32, 64, 128 bytes). Benchmark it against glibc malloc for 10 million small allocations of random size. Measure throughput and peak memory usage.

Resources

📕Linux Kernel Development (3rd ed)
bookpaidRobert Love, Addison-Wesley. ISBN 978-0672329463

Robert Love's chapter on memory management is the clearest high-level treatment of buddy + slab for newcomers.

📕Understanding the Linux Virtual Memory Manager
bookfree

Mel Gorman's exhaustive online book covers every corner of Linux VM — the definitive deep reference, free online.

✎SLUB: The unqueued slab allocator (LWN)
articlefree

Jonathan Corbet's LWN article explaining why SLUB replaced SLAB and its core design insights.

📄Memory Management documentation (kernel.org)
docsfree

The official in-tree documentation for the MM subsystem — covers GFP flags, page allocator, and vmalloc in detail.

✎Linux kernel memory management — Bootlin slides
articlefree

Free Electrons / Bootlin training materials with clear diagrams of buddy, slab, and virtual memory layout.

✎What every programmer should know about memory
articlefree

Ulrich Drepper's landmark paper on memory hierarchy — essential background for understanding why slab cache locality matters.

📄KASAN documentation
docsfree

Official kernel docs for Kernel Address SANitizer — how to enable it, interpret reports, and tune it.

📜The Slab Allocator: An Object-Caching Kernel Memory Allocator (original paper)
paperfree

Jeff Bonwick's original 1994 USENIX paper introducing the slab allocator concept — reading the original is deeply illuminating.

📕Linux Device Drivers (3rd ed) — Chapter 8: Allocating Memory
bookfree

Corbet, Rubini, Kroah-Hartman's classic — chapter 8 covers kmalloc/vmalloc/alloc_pages from a driver author's perspective. Free online.

📄kernelnewbies.org — Linux MM
docsfree

Community-maintained wiki with friendly explanations of memory management concepts and links to patches.

Checkpoints — you can now…

  • ✓You can explain the buddy allocator split/coalesce algorithm from memory and compute the buddy address of any PFN at any order using XOR.
  • ✓You can write a kernel module that creates a custom slab cache, allocates objects from it, and frees them correctly — without triggering KASAN or SLUB warnings.
  • ✓You can choose the correct GFP flags for any execution context (process context, softirq, hardirq, spinlock held, filesystem path, block layer) and explain why each choice is correct.
  • ✓You can explain the difference between kmalloc and vmalloc in terms of physical contiguity, DMA suitability, size limits, and TLB impact, and choose the right one for a given use case.
  • ✓You can read /proc/buddyinfo and /proc/slabinfo and explain what each column means and what a pathological value looks like.
  • ✓You can trace the allocation path from a kmalloc call all the way down to the buddy allocator, naming each layer and data structure involved.
  • ✓You can interpret a KASAN out-of-bounds or use-after-free report and identify the offending code location and the allocation site.
  • ✓You can describe the three-watermark (MIN/LOW/HIGH) system and explain what kswapd, direct reclaim, and the OOM killer each do and when they trigger.