Concurrency & Locking in Drivers

A race condition occurs when the correctness of a computation depends on the relative timing of two or more threads of execution, and at least one of those threads writes to shared state.

To understand why this happens at the hardware level, consider a simple counter increment: counter++. This looks like one operation in C, but the compiler and CPU decompose it into three micro-operations:

1. LOAD: Read counter from memory into a register.
2. ADD: Increment the register value.
3. STORE: Write the register back to memory.

If two CPU cores execute this simultaneously on the same counter variable, the following interleaving is possible:

Core 0: LOAD  counter → reg0  (reg0 = 5)
Core 1: LOAD  counter → reg1  (reg1 = 5)
Core 0: ADD   reg0 + 1 → 6
Core 1: ADD   reg1 + 1 → 6
Core 0: STORE 6 → counter
Core 1: STORE 6 → counter   ← LOST UPDATE!

Both cores read 5, both increment to 6, and both write 6. One increment was lost. We expected 7.

This is called a lost update and it is the canonical race condition. The window of vulnerability — the sequence of instructions between the load and the store — is called the critical section.

The situation is made worse by three hardware realities:

• CPU caches: Modern CPUs have L1/L2/L3 caches. Writes to a cache line are not immediately visible to other CPUs. The MESI coherence protocol eventually propagates them, but eventually can be too late without explicit synchronization.
• Out-of-order execution: CPUs reorder loads and stores for performance. A write you wrote first in C might be observed by another CPU after a later write.
• Compiler reordering: Even before the CPU sees your code, the compiler can reorder, hoist, or eliminate memory accesses for optimization, breaking assumptions about ordering.

This is why synchronization primitives include memory barriers (also called fences) — instructions that tell both the compiler and CPU: do not move memory operations across this point. In the Linux kernel, smp_mb(), smp_rmb(), and smp_wmb() are the portable memory barrier APIs.

/* Broken: two threads running this concurrently lose updates */
static int counter = 0;
void broken_increment(void) {
    counter++;  /* LOAD, ADD, STORE — NOT atomic */
}

/* Fixed with atomic_t */
#include <linux/atomic.h>
static atomic_t counter = ATOMIC_INIT(0);
void safe_increment(void) {
    atomic_inc(&counter);  /* single atomic RMW instruction */
}

A spinlock is the simplest mutual-exclusion primitive: a thread trying to acquire a held spinlock simply loops (spins) in a tight busy-wait loop until the lock is released. There is no sleeping, no context switch, no scheduler involvement.

When to use a spinlock:

• The critical section is very short (tens to hundreds of nanoseconds — a few register operations, a list splice, a flag toggle).
• You are in interrupt context (softirq, hardirq, tasklet) — you cannot sleep here, so spinlocks are the only option.
• You are holding a lock that must be acquired from interrupt context — the lock must be a spinlock everywhere it is used.

Why spinlocks are dangerous on uniprocessor systems (UP): On a single CPU, spinning forever waits for something that cannot happen (the other thread runs only if you yield). The kernel solves this: on UP builds with CONFIG_SMP=n, spinlocks compile away to nothing but memory barriers, because the only real concurrency source is interrupt handlers. For interrupt-safe use, you must use spin_lock_irqsave() which also disables local interrupts.

The rule: Never hold a spinlock while doing anything that might sleep. If you call kmalloc(GFP_KERNEL), copy from user, wait on a semaphore, or do any I/O inside a spinlock, you will deadlock or corrupt the kernel.

Spinlock flavors in the kernel:

• spin_lock(lock) / spin_unlock(lock) — basic, no interrupt disabling. Use only if the lock is never acquired from interrupt context.
• spin_lock_irqsave(lock, flags) / spin_unlock_irqrestore(lock, flags) — disables local IRQs and saves their state. Use when the lock is taken from interrupt handlers.
• spin_lock_bh(lock) / spin_unlock_bh(lock) — disables softirqs (bottom halves). Use when the lock is taken from softirqs/tasklets but not from hardirqs.

The flags in irqsave is crucial: it records whether interrupts were enabled before you locked, so irqrestore restores the original state rather than blindly re-enabling them (which would be wrong if your caller had IRQs disabled for its own reasons).

#include <linux/spinlock.h>

struct my_device {
    spinlock_t lock;
    u32        reg_shadow;   /* cached copy of hardware register */
    struct list_head pending;
};

void my_device_init(struct my_device *dev) {
    spin_lock_init(&dev->lock);
    dev->reg_shadow = 0;
    INIT_LIST_HEAD(&dev->pending);
}

/* Called from process context AND interrupt handler */
void my_device_update_reg(struct my_device *dev, u32 val) {
    unsigned long flags;

    spin_lock_irqsave(&dev->lock, flags);
    dev->reg_shadow = val;
    writel(val, dev->mmio + REG_OFFSET);  /* or whatever hw write */
    spin_unlock_irqrestore(&dev->lock, flags);
}

A mutex (mutual exclusion semaphore) is a sleeping lock: a thread that cannot acquire a mutex is put to sleep by the scheduler, freeing the CPU for other work. It wakes up when the lock is released.

When to use a mutex:

• The critical section may be long (disk I/O, network operations, firmware download, user-space copies).
• You are in process context — a system call, a work queue work item, or a kernel thread.
• You can sleep — nothing in your call chain disables preemption or holds a spinlock.

The three cardinal rules of mutexes in the kernel:

1. Only the thread that locked the mutex may unlock it (unlike semaphores).
2. You cannot lock a mutex from interrupt context (the mutex implementation calls schedule() internally).
3. A mutex must not be locked recursively by the same thread (the standard kernel mutex is not re-entrant; use mutex_trylock() or restructure).

mutex vs. semaphore: The kernel has struct semaphore as well. A semaphore has a count and allows up to N holders simultaneously. A mutex is a binary semaphore (count=1) with ownership semantics. Prefer struct mutex for mutual exclusion — it is faster (uses optimistic spinning: tries to spin briefly before sleeping, similar to a hybrid lock) and has better debug support with CONFIG_DEBUG_MUTEXES.

Priority inversion and the PI mutex: Classic mutexes suffer from priority inversion: a high-priority thread waits for a mutex held by a low-priority thread, but a medium-priority thread preempts the low-priority thread, indirectly blocking the high-priority one indefinitely. The kernel offers struct rt_mutex (priority-inheritance mutex) for real-time drivers that need to avoid this.

Practical advice for drivers: Use a per-device mutex for operations that touch device state and may sleep (like open, read, write, ioctl). Use a spinlock (with irqsave) only for the small critical sections shared with your interrupt handler.

#include <linux/mutex.h>

struct my_device {
    struct mutex  io_lock;     /* protects slow I/O operations */
    spinlock_t    irq_lock;    /* protects data shared with IRQ handler */
    u8           *dma_buf;
    size_t        buf_len;
};

void my_device_init(struct my_device *dev) {
    mutex_init(&dev->io_lock);
    spin_lock_init(&dev->irq_lock);
}

/* read() file operation — process context, may sleep */
ssize_t my_read(struct file *filp, char __user *ubuf,
                size_t count, loff_t *pos)
{
    struct my_device *dev = filp->private_data;
    ssize_t ret;

    if (mutex_lock_interruptible(&dev->io_lock))
        return -ERESTARTSYS;  /* signal interrupted us */

    /* Long operation: DMA, firmware, copy_to_user — all fine here */
    ret = copy_to_user(ubuf, dev->dma_buf, min(count, dev->buf_len));

    mutex_unlock(&dev->io_lock);
    return ret ? -EFAULT : min(count, dev->buf_len);
}

The Linux kernel distinguishes sharply between two execution contexts:

Process context: Code running on behalf of a user process — system calls, kernel threads, work queue handlers. The current pointer is valid and identifies the process. You can sleep, block on I/O, acquire mutexes, call kmalloc(GFP_KERNEL).

Atomic context: Code that must not sleep. This includes:

• Hard interrupt handlers (ISRs) — called with the CPU's interrupt flag cleared (on x86) or with that IRQ line masked.
• Softirqs — deferred interrupt work, runs with softirqs disabled on the current CPU.
• Tasklets — built on softirqs, run in softirq context.
• Any code holding a spinlock — because another CPU spinning on that lock would starve.
• Any code with preemption disabled (preempt_disable()).

What you CANNOT do in atomic context:

• schedule() or any function that calls it (this includes msleep(), wait_event(), mutex_lock(), down()).
• kmalloc(GFP_KERNEL) — use kmalloc(GFP_ATOMIC) instead (allocates from a smaller emergency pool, may fail).
• copy_from_user() / copy_to_user() — these can page-fault, which sleeps.
• Any blocking I/O.

How to check at runtime: The kernel provides in_interrupt(), in_atomic(), and might_sleep(). If you annotate a function with might_sleep(), the kernel (in debug builds) will warn if that function is called from atomic context. This is how many bugs are caught during development.

The BUG_ON / WARN_ON pattern: Senior kernel developers liberally sprinkle WARN_ON(in_interrupt()) at the top of functions that must run in process context. This is defensive programming: it catches callers that violate the contract, rather than silently deadlocking.

Splitting work between ISR and process context: The classic design pattern for interrupt-driven drivers is the top half / bottom half split:

• Top half (ISR): Do the minimum — acknowledge the interrupt, read status registers, wake up a waiting thread or queue a work item. Release the CPU as fast as possible.
• Bottom half (work queue / tasklet): Do the heavy lifting — DMA buffer processing, protocol parsing, upcalls into higher layers — in a context where sleeping is allowed.

#include <linux/interrupt.h>
#include <linux/workqueue.h>
#include <linux/slab.h>

struct my_device {
    spinlock_t       lock;
    struct work_struct work;   /* bottom half */
    u32              status;   /* written by ISR, read by worker */
};

/* Top half: ISR — ATOMIC CONTEXT */
static irqreturn_t my_isr(int irq, void *data)
{
    struct my_device *dev = data;
    unsigned long flags;

    spin_lock_irqsave(&dev->lock, flags);
    dev->status = readl(dev->mmio + STATUS_REG);  /* fast HW read */
    spin_unlock_irqrestore(&dev->lock, flags);

    /* Schedule bottom half — safe from ISR */
    schedule_work(&dev->work);
    return IRQ_HANDLED;
}

/* Bottom half: work queue — PROCESS CONTEXT, can sleep */
static void my_worker(struct work_struct *w)
{
    struct my_device *dev = container_of(w, struct my_device, work);
    unsigned long flags;
    u32 status;

    spin_lock_irqsave(&dev->lock, flags);
    status = dev->status;
    spin_unlock_irqrestore(&dev->lock, flags);

    /* Now we can do slow work: allocate, copy, signal userspace */
    if (status & ERROR_BIT)
        dev_err(dev->dev, "hardware error: 0x%08x\n", status);
}

The most common confusion for new driver writers is choosing between a spinlock and a mutex. Here is a systematic framework:

Step 1: What context will this lock be acquired from?

• Only from process context (syscalls, kernel threads, work queues)? → mutex is preferred.
• From interrupt context (ISR, softirq, tasklet) as well? → Must use spinlock everywhere this lock is used (you cannot mix mutex and spinlock protecting the same data).
• From a bottom half (tasklet/softirq) but NOT from a hard IRQ? → spinlock with _bh variants.

Step 2: How long will the critical section be?

• Microseconds (register twiddling, list pointer swap)? → spinlock is efficient.
• Milliseconds or longer (DMA wait, firmware load, copy_to_user)? → mutex to avoid wasting CPU cycles spinning.

Step 3: Is ownership semantics needed?

• Mutexes have ownership: only the locker can unlock. This enables priority inheritance.
• Spinlocks have no ownership: any CPU can unlock any spinlock (though doing so intentionally is usually wrong).

Step 4: Are multiple simultaneous readers acceptable?

• If reads vastly outnumber writes, consider rwlock_t (spinlock variant) or struct rw_semaphore.
• Or consider RCU (Read-Copy-Update) for the ultimate read-side performance.

Common mistake: lock granularity. Beginners often use one big lock for an entire device (dev->big_lock). This is correct but slow. Advanced drivers use fine-grained locks per sub-component (RX ring lock, TX ring lock, config lock). Start coarse-grained and correct; optimize only when profiling shows contention.

The lock/unlock symmetry rule: Every lock acquisition must have exactly one corresponding unlock on every possible code path — including error paths. The pattern goto out_unlock or RAII-style cleanup labels are idiomatic kernel C for this.

/*
 * Decision summary as code comments — a template for a typical
 * character device that has both an ISR and user-facing syscalls.
 */
struct my_dev {
    /*
     * Protects: hw_state, irq_count, pending_list
     * Taken from: ISR (hardirq) + process context
     * Verdict: spinlock + irqsave
     */
    spinlock_t  hw_lock;
    u32         hw_state;
    int         irq_count;
    struct list_head pending_list;

    /*
     * Protects: user_buf, open_count, device configuration
     * Taken from: process context only (open/read/write/ioctl)
     * May sleep: yes (copy_to/from_user inside)
     * Verdict: mutex
     */
    struct mutex user_lock;
    char        *user_buf;
    int          open_count;
};

For single integer counters and flags, a full spinlock is overkill. The kernel provides atomic types backed by CPU read-modify-write (RMW) instructions that are inherently indivisible.

atomic_t and atomic64_t: These wrap an int / s64 with operations guaranteed to be atomic with respect to other CPUs. The implementation on x86 uses the LOCK prefix on instructions like LOCK XADD, LOCK CMPXCHG. On ARM it uses load-linked/store-conditional (LDREX/STREX) or LSE atomics.

Key operations:

• atomic_set(&a, v) — initialize or forcibly set (not RMW, so only safe before sharing).
• atomic_read(&a) — reads (with compiler barrier).
• atomic_inc(&a), atomic_dec(&a) — increment/decrement.
• atomic_add_return(&a, v) — add and return the new value (useful for counting open file descriptors).
• atomic_cmpxchg(&a, old, new) — compare-and-swap, returns old value. The foundation of lock-free algorithms.
• atomic_dec_and_test(&a) — decrement and return true if result is zero. Classic reference-counting pattern.

atomic_t is NOT a substitute for all locking. It only protects single variables. If you need to update two fields atomically together (e.g., head and tail of a queue), you need a real lock.

Bitwise atomics for flags: set_bit(), clear_bit(), test_and_set_bit(), test_and_clear_bit() operate atomically on individual bits within a long. These are perfect for a flags field that multiple parts of your driver read and write (e.g., DEVICE_OPEN, DEVICE_BUSY, DEVICE_ERROR bits).

Memory ordering with atomics: Atomic operations on their own do NOT provide ordering guarantees beyond the single operation. If you need to ensure that a memory write is visible before an atomic flag is set, use smp_store_release() / smp_load_acquire() or explicit barriers. This is the subtle heart of lock-free programming.

#include <linux/atomic.h>
#include <linux/bitops.h>

struct my_dev {
    atomic_t    open_count;   /* number of current openers */
    atomic_t    error_count;  /* total hardware errors */
    unsigned long flags;      /* bit flags: OPEN, BUSY, ERROR */
};

#define FLAG_OPEN  0
#define FLAG_BUSY  1
#define FLAG_ERROR 2

static int my_open(struct inode *inode, struct file *filp) {
    struct my_dev *dev = container_of(inode->i_cdev,
                                      struct my_dev, cdev);
    /* Allow only one opener at a time using atomic test-and-set */
    if (test_and_set_bit(FLAG_OPEN, &dev->flags))
        return -EBUSY;  /* already open */

    atomic_inc(&dev->open_count);
    filp->private_data = dev;
    return 0;
}

static int my_release(struct inode *inode, struct file *filp) {
    struct my_dev *dev = filp->private_data;
    clear_bit(FLAG_OPEN, &dev->flags);
    /* dec_and_test: true when last reference dropped */
    if (atomic_dec_and_test(&dev->open_count))
        dev_dbg(dev->dev, "all openers gone\n");
    return 0;
}

The ioctl system call is the classic catch-all for device-specific operations that don't fit read/write. Understanding its locking history is a masterclass in kernel evolution.

The old ioctl (pre-2.6.36): Earlier kernels had an ioctl file operation that was protected by the Big Kernel Lock (BKL) — a single global recursive spinlock covering the entire kernel. This made old drivers trivially safe but was a severe scalability bottleneck on multi-core machines.

unlocked_ioctl (current): The unlocked_ioctl file operation (introduced to replace the BKL-protected ioctl) has no locking provided by the kernel. The driver author is entirely responsible for serialization. The name is a reminder: you are unlocked, proceed carefully.

What reentrancy means: A function is reentrant if it can safely be called again before a previous call has returned — either from a different CPU or (on a preemptible kernel) from a preempting thread. Your unlocked_ioctl handler WILL be called by multiple threads simultaneously on an SMP machine. If it touches any shared device state, you must protect it.

Typical unlocked_ioctl pattern:

1. Validate the cmd and arg from userspace (never trust them).
2. copy_from_user() any input data into a kernel-side struct.
3. Acquire the device mutex.
4. Perform the operation.
5. Release the mutex.
6. copy_to_user() any output (can be done outside the lock if output buf is local).

The compat_ioctl: On a 64-bit kernel running 32-bit userspace, pointer sizes differ. If your ioctl passes pointers or has differently-sized structs, you must implement compat_ioctl to translate the 32-bit structs to 64-bit. Forgetting this is a common driver bug and security vulnerability (CVE-class issues arise from compat paths reading the wrong memory).

Avoiding the lock for read-only ioctls: If an ioctl is truly read-only with respect to device state (like querying a version number from a static structure), you can skip the lock. But be very careful: even reads that appear innocent can race with a concurrent write that is tearing a multi-word struct. When in doubt, lock.

#include <linux/uaccess.h>
#include <linux/mutex.h>

#define MY_IOC_MAGIC   'M'
#define MY_IOC_GET_STATUS  _IOR(MY_IOC_MAGIC, 0, struct my_status)
#define MY_IOC_SET_CONFIG  _IOW(MY_IOC_MAGIC, 1, struct my_config)

static long my_unlocked_ioctl(struct file *filp,
                               unsigned int cmd,
                               unsigned long arg)
{
    struct my_dev *dev = filp->private_data;
    int ret = 0;

    /* Validate cmd belongs to our driver */
    if (_IOC_TYPE(cmd) != MY_IOC_MAGIC) return -ENOTTY;
    if (_IOC_NR(cmd) > MY_IOC_MAXNR)   return -ENOTTY;

    switch (cmd) {
    case MY_IOC_GET_STATUS: {
        struct my_status st;

        if (mutex_lock_interruptible(&dev->user_lock))
            return -ERESTARTSYS;
        st.errors = dev->error_count;
        st.state  = dev->hw_state;
        mutex_unlock(&dev->user_lock);

        /* copy_to_user outside lock: st is a stack copy */
        if (copy_to_user((void __user *)arg, &st, sizeof(st)))
            ret = -EFAULT;
        break;
    }
    case MY_IOC_SET_CONFIG: {
        struct my_config cfg;
        if (copy_from_user(&cfg, (void __user *)arg, sizeof(cfg)))
            return -EFAULT;  /* before lock: no state to undo */

        if (mutex_lock_interruptible(&dev->user_lock))
            return -ERESTARTSYS;
        ret = apply_config(dev, &cfg);  /* may sleep: firmware DL */
        mutex_unlock(&dev->user_lock);
        break;
    }
    default:
        ret = -ENOTTY;
    }
    return ret;
}

RCU (Read-Copy-Update) is the most sophisticated synchronization mechanism in the Linux kernel. It allows reads to proceed with zero overhead — no atomic operations, no memory barriers, no locks — while writes are more expensive but still non-blocking for readers.

The core insight: Instead of locking shared data to update it, RCU writers create a new copy of the data, update the copy, then atomically swing a pointer to point to the new version. Old readers finish with the old version; new readers see the new version. The old version is freed only after all pre-existing readers have finished a grace period — a time interval during which every CPU has gone through at least one context switch (proving no CPU is still in an old read-side critical section).

Read-side critical section: Enter with rcu_read_lock() (disables preemption, zero cost), exit with rcu_read_unlock(). Inside, dereference RCU-protected pointers with rcu_dereference() (adds a READ_ONCE-like barrier).

Write side: Protect with a separate lock (spinlock or mutex) against concurrent writers. After updating the pointer with rcu_assign_pointer(), call synchronize_rcu() (blocks until grace period ends) or call_rcu() (registers a callback, non-blocking) before freeing the old object.

Where RCU shines in drivers:

• Per-CPU driver state that is almost always read.
• Lists of registered callbacks or handlers (e.g., a list of event subscribers).
• Network packet filtering (the kernel's Netfilter uses RCU extensively).
• Driver configuration that is read on every packet/request but rarely reconfigured.

Limitations: RCU is complex. The memory management contract is strict (you must not free objects until the grace period ends). Readers cannot sleep inside rcu_read_lock(). And it only helps when reads dominate writes; if writes are frequent, the write-side synchronization (synchronize_rcu()) becomes the bottleneck.

#include <linux/rcupdate.h>
#include <linux/slab.h>

struct my_config {
    u32  speed;
    u32  mode;
    struct rcu_head rcu;  /* for call_rcu callback */
};

/* Global RCU-protected pointer to current config */
static struct my_config __rcu *current_cfg;
static DEFINE_SPINLOCK(cfg_write_lock); /* serializes writers only */

/* Read side: zero cost, can happen millions of times/sec */
static u32 get_current_speed(void) {
    struct my_config *cfg;
    u32 speed;

    rcu_read_lock();
    cfg = rcu_dereference(current_cfg);
    speed = cfg ? cfg->speed : 0;
    rcu_read_unlock();  /* cfg pointer no longer safe after this */
    return speed;
}

/* Write side: replaces config atomically */
static int update_config(u32 new_speed, u32 new_mode) {
    struct my_config *new_cfg, *old_cfg;

    new_cfg = kmalloc(sizeof(*new_cfg), GFP_KERNEL);
    if (!new_cfg) return -ENOMEM;
    new_cfg->speed = new_speed;
    new_cfg->mode  = new_mode;

    spin_lock(&cfg_write_lock);
    old_cfg = rcu_dereference_protected(current_cfg,
                    lockdep_is_held(&cfg_write_lock));
    rcu_assign_pointer(current_cfg, new_cfg);
    spin_unlock(&cfg_write_lock);

    /* Wait for all readers seeing old_cfg to finish */
    if (old_cfg) {
        synchronize_rcu();   /* or: call_rcu(&old_cfg->rcu, free_cb) */
        kfree(old_cfg);
    }
    return 0;
}

A deadlock is a state where two or more threads each hold a lock the other thread needs, so neither can proceed. The system hangs forever (or until a watchdog fires).

Classic ABBA deadlock:

Thread 1: lock(A) → lock(B)
Thread 2: lock(B) → lock(A)

If Thread 1 holds A and Thread 2 holds B simultaneously, both spin forever.

How to prevent deadlocks:

1. Consistent lock ordering: Always acquire locks in the same globally defined order. If you need locks A and B, every piece of code in the kernel must lock A before B, never B before A. Document your lock ordering. The kernel's lock validator (lockdep) enforces this automatically.
2. Try-lock with backoff: Use spin_trylock() or mutex_trylock() and back off if the lock is not available (release held locks and retry). Risky to implement correctly.
3. Lock hierarchies: Assign a numeric level to each lock type. A thread holding level-3 lock may only acquire level-4+ locks, never a lower level.
4. Minimize lock nesting: The safest code grabs at most one lock at a time. If you need two, question whether the design is right.

Self-deadlock: A non-recursive mutex will deadlock if the same thread tries to lock it twice. This happens when a function you call tries to acquire a lock you already hold. Audit your call chains.

Lockdep: the kernel's deadlock detector: Enable CONFIG_PROVE_LOCKING in your kernel config. lockdep tracks every lock acquisition and release at runtime, building a graph of lock ordering. When it detects a potential cycle (even before the actual deadlock occurs), it prints a warning with the full lock chain. This is one of the most valuable debugging tools in driver development. Always develop and test with lockdep enabled.

/*
 * WRONG: ABBA deadlock pattern
 * Thread 1 calls wrong_fn_a, Thread 2 calls wrong_fn_b simultaneously
 */
static DEFINE_SPINLOCK(lock_a);
static DEFINE_SPINLOCK(lock_b);

void wrong_fn_a(void) {
    spin_lock(&lock_a);   /* acquire A */
    spin_lock(&lock_b);   /* acquire B */
    /* ... */
    spin_unlock(&lock_b);
    spin_unlock(&lock_a);
}

void wrong_fn_b(void) {
    spin_lock(&lock_b);   /* acquire B */
    spin_lock(&lock_a);   /* acquire A  <-- DEADLOCK if wrong_fn_a running */
    /* ... */
    spin_unlock(&lock_a);
    spin_unlock(&lock_b);
}

/*
 * CORRECT: enforce A-before-B ordering everywhere
 */
void correct_fn_b(void) {
    spin_lock(&lock_a);   /* always A first */
    spin_lock(&lock_b);   /* then B */
    /* ... */
    spin_unlock(&lock_b);
    spin_unlock(&lock_a);
}

The fastest lock is no lock at all. If each CPU has its own private copy of a variable, there is no sharing and therefore no need for synchronization. This is the idea behind per-CPU variables.

How they work: The kernel allocates one instance of the variable for each CPU in the system. Each CPU accesses only its own instance. The memory is laid out so that CPU 0's copy is in one cache line, CPU 1's in another, eliminating false sharing.

Usage: Declare with DEFINE_PER_CPU(type, name). Access with get_cpu_var(name) (which also disables preemption, because you must not be migrated to another CPU mid-access) and put_cpu_var(name). For read-only access from IRQ context use per_cpu(name, cpu) to access any CPU's copy by number.

Where per-CPU variables shine:

• Statistics counters (packet counts, error counts) — each CPU increments its own counter; a reader sums across all CPUs.
• Per-CPU work queues and pending lists.
• NAPI (the Linux network receive path) uses per-CPU poll lists.
• kmalloc slabs use per-CPU caches to avoid spinlock contention on every allocation.

Limitations: Per-CPU variables are not useful when multiple CPUs genuinely need to share and coordinate on a single piece of state (e.g., a hardware command ring that all CPUs push to). They are for stats, caches, and per-CPU queues.

Relationship to preemption: You must disable preemption while accessing a per-CPU variable to ensure you are not migrated to a different CPU between reading the address and using the value. get_cpu_var() does this automatically with preempt_disable() / preempt_enable().

#include <linux/percpu.h>

/* Per-CPU packet counter — no locking needed! */
static DEFINE_PER_CPU(unsigned long, pkt_count);

/* Called from interrupt handler (per-CPU ISR, no migration) */
void count_packet(void) {
    /* this_cpu_inc is safe from IRQ context, no preempt_disable needed
     * because IRQs are already disabled in hardirq context */
    this_cpu_inc(pkt_count);
}

/* Sum across all CPUs to get total */
unsigned long get_total_packets(void) {
    unsigned long total = 0;
    int cpu;
    for_each_possible_cpu(cpu)
        total += per_cpu(pkt_count, cpu);
    return total;
}

/* Accessing from process context: disable preemption manually */
void process_context_access(void) {
    unsigned long *cnt;
    cnt = &get_cpu_var(pkt_count);  /* disables preemption */
    (*cnt)++;                        /* safe: pinned to this CPU */
    put_cpu_var(pkt_count);         /* re-enables preemption */
}

When a driver needs to block a user process until an event occurs (data available, hardware ready, DMA complete), the right tool is a wait queue, not a spinlock in a busy loop.

The problem with polling: Code like while (!data_ready) cpu_relax(); wastes a full CPU core and — if not in atomic context — is usually wrong. cpu_relax() is a hint to the CPU (inserts a PAUSE instruction on x86 to reduce power and improve hyperthreading), but it is not a scheduler yield.

Wait queues: A wait queue is a list of sleeping processes waiting for a condition. The driver puts the process to sleep with wait_event_interruptible(), and wakes it with wake_up_interruptible() from wherever the event is detected (usually an interrupt handler).

The wake/check race: There is a subtle race: what if the condition becomes true after you check it but before you actually sleep? You would sleep forever. The wait_event_* macros handle this correctly by using a TASK_INTERRUPTIBLE state and re-checking the condition after being added to the wait queue, before actually calling schedule().

Variants:

• wait_event(wq, condition) — sleeps until condition is true. Not interruptible (cannot be killed by signal).
• wait_event_interruptible(wq, condition) — returns -ERESTARTSYS if a signal arrives. Prefer this for user-visible blocking.
• wait_event_timeout(wq, condition, timeout) — adds a deadline.
• wait_event_interruptible_timeout(wq, condition, timeout) — interruptible + timeout.

Waking up: Call wake_up(&wq) or wake_up_interruptible(&wq) from your ISR, work queue, or timer. wake_up_interruptible only wakes processes sleeping with TASK_INTERRUPTIBLE; wake_up wakes all sleeping states.

#include <linux/wait.h>

struct my_dev {
    wait_queue_head_t read_wq;  /* readers wait here */
    spinlock_t        lock;
    bool              data_ready;
    char              data[256];
    size_t            data_len;
};

void my_dev_init(struct my_dev *dev) {
    init_waitqueue_head(&dev->read_wq);
    spin_lock_init(&dev->lock);
    dev->data_ready = false;
}

/* ISR: hardware says data is ready */
irqreturn_t my_isr(int irq, void *data) {
    struct my_dev *dev = data;
    unsigned long flags;

    spin_lock_irqsave(&dev->lock, flags);
    /* Fill dev->data from hardware FIFO */
    dev->data_len = read_hw_fifo(dev->data, sizeof(dev->data));
    dev->data_ready = true;
    spin_unlock_irqrestore(&dev->lock, flags);

    wake_up_interruptible(&dev->read_wq);  /* wake blocked readers */
    return IRQ_HANDLED;
}

/* read() — blocks until data arrives */
ssize_t my_read(struct file *filp, char __user *buf,
                size_t count, loff_t *pos) {
    struct my_dev *dev = filp->private_data;
    ssize_t ret;

    /* Sleep until data_ready; return -ERESTARTSYS on signal */
    if (wait_event_interruptible(dev->read_wq, dev->data_ready))
        return -ERESTARTSYS;

    /* Now we have data — grab lock to read consistently */
    spin_lock_irq(&dev->lock);
    ret = min(count, dev->data_len);
    dev->data_ready = false;
    spin_unlock_irq(&dev->lock);

    if (copy_to_user(buf, dev->data, ret))
        return -EFAULT;
    return ret;
}

The Linux kernel ships with lockdep, one of the most sophisticated runtime lock validators in any software system. It proves at runtime that your locking is correct — specifically, it proves the lock-ordering graph has no cycles (which would cause deadlocks) and flags single-lock rule violations (locking a mutex from interrupt context, etc.).

How lockdep works: Every lock operation is instrumented. Lockdep maintains a lock class graph — a directed graph where an edge from class A to class B means 'A was held when B was acquired'. If it finds a cycle in this graph, a deadlock is possible. It reports the exact call chain that created the cycle, even before any actual deadlock occurs. It also tracks per-lock rules (e.g., 'this lock is ever taken in hardirq context') and validates every new acquisition against those rules.

Key kernel configs to enable for driver development:

CONFIG_PROVE_LOCKING=y         # enables lockdep core
CONFIG_DEBUG_LOCK_ALLOC=y      # detects lock-use-after-free
CONFIG_DEBUG_MUTEXES=y         # mutex-specific checks
CONFIG_DEBUG_SPINLOCK=y        # spinlock checks
CONFIG_DEBUG_ATOMIC_SLEEP=y    # warns if sleep attempted in atomic ctx
CONFIG_LOCKDEP_SUPPORT=y       # architecture support (usually auto)

The lockdep_assert_held() annotation: You can explicitly assert inside a function that a particular lock must be held by the caller. This documents the contract and lockdep will verify it:

void my_private_fn(struct my_dev *dev)
    __must_hold(&dev->lock)  /* sparse annotation */
{
    lockdep_assert_held(&dev->lock);
    /* ... */
}

Sparse, the static checker: Beyond runtime, the kernel uses the sparse tool for static analysis. The __must_hold, __acquires, and __releases annotations document locking contracts. Running make C=1 invokes sparse on changed files. Many locking bugs are caught before the code even runs.

The KCSAN data race detector: CONFIG_KCSAN=y enables the Kernel Concurrency Sanitizer — a dynamic detector that instruments memory accesses and reports pairs of accesses to the same location without synchronization. It catches races that lockdep cannot (e.g., lock-free code that is subtly wrong).

/* Example: documenting and asserting locking contracts */
#include <linux/lockdep.h>

struct my_dev {
    spinlock_t lock;
    u32        state;
};

/*
 * __must_hold tells sparse (static checker) that the caller
 * must hold dev->lock when calling this function.
 * lockdep_assert_held catches violations at runtime.
 */
static void __update_state(struct my_dev *dev, u32 new_state)
    __must_hold(&dev->lock)
{
    lockdep_assert_held(&dev->lock);
    dev->state = new_state;
}

void public_update(struct my_dev *dev, u32 new_state) {
    unsigned long flags;
    spin_lock_irqsave(&dev->lock, flags);
    __update_state(dev, new_state);  /* lock is held: OK */
    spin_unlock_irqrestore(&dev->lock, flags);
}

/*
 * To test with lockdep: build a QEMU VM kernel with CONFIG_PROVE_LOCKING=y
 * Load your driver module and exercise the ioctl/read/write paths.
 * Lockdep will print a WARNING to dmesg if any ordering violation is found.
 * Zero lockdep warnings = strong confidence in correctness.
 */