Time, Timers & the Kernel
The kernel's relationship with time — from coarse jiffy ticks to nanosecond-precision hrtimers — is what makes scheduling, networking, audio, and real-time systems possible.
Overview
Every running program has an implicit assumption baked deep into it: that time passes, that delays are honored, that "sleep for 100ms" actually means something. But who enforces that contract? Who tells the CPU that a scheduler quantum has expired, that a network retransmit timeout has fired, that a high-resolution audio buffer needs to be filled right now? The answer is the Linux kernel's timekeeping subsystem — a layered, beautifully engineered system that spans from the raw hardware oscillator to the nanosleep() system call you use every day.
This module peels back every layer. We start at the bottom: the hardware clock source, the oscillator ticking away at millions or billions of cycles per second, and the clockevent device that can interrupt the CPU at a programmed future moment. On top of those primitives the kernel builds two separate timer abstractions: the classic "jiffy" system (coarse, tick-based, simple, fast) and hrtimers (high-resolution, nanosecond-accurate, event-driven). Both coexist and serve different purposes — you'll understand why by the end.
We then look at the modern tickless kernel (NO_HZ), which is one of the most important power and real-time improvements in recent kernel history. When nothing is scheduled, why keep interrupting the CPU 250 or 1000 times per second? NO_HZ lets the CPU idle undisturbed, and we'll see exactly how that works without breaking every timer in the system. Finally we look at the user-facing APIs — ktime_t, delayed_work, timer_list — that kernel developers use daily.
By the end of this module you will be able to read kernel timer code fluently, write your own hrtimer and delayed-work callbacks, explain the HZ vs. tick vs. jiffies relationship precisely, and understand why a real-time musician cares deeply about NO_HZ_FULL.
Almost every kernel subsystem touches time. The scheduler uses ticks and hrtimers to enforce time slices and implement sched_yield, nanosleep, and SCHED_DEADLINE. The networking stack fires retransmit timers and keepalive probes. Block devices use delayed work to coalesce I/O. Sound cards depend on sub-millisecond precision for buffer management. If you want to contribute to any of these areas — or to the core timekeeping code itself — you need a solid mental model of how the kernel tracks, represents, and acts on time.
Beyond raw contribution value, the timekeeping subsystem is a masterclass in layered abstraction and hardware portability. It shows how a single conceptual interface (clocksource) can wrap a TSC on x86, a generic timer on ARM, a HPET, or a Xen paravirt clock. It shows how the kernel handles the messy reality that hardware clocks drift, wrap around, and are measured in wildly different units. Studying this code teaches you the engineering discipline of clean abstraction over messy hardware — a skill that transfers to every other kernel subsystem you'll ever touch.
Linux Timer Subsystem: From Hardware to Callback
Hardware oscillators feed both the clocksource (for reading time) and the clock_event_device (for scheduling future interrupts). A single timer interrupt triggers hrtimer expiry (hardirq), jiffies update, and eventually softirq delivery of timer_list callbacks, which in turn can queue delayed_work for execution in process context.
The concepts
01Hardware Time Sources: Oscillators, Counters, and the Clocksource Abstraction
At the very bottom of Linux timekeeping is a hardware counter — a register in some piece of silicon that increments at a fixed frequency driven by an oscillator. Examples:
- TSC (Time Stamp Counter) on x86: a 64-bit counter incremented every CPU cycle. Read with
RDTSC. On modern CPUs it is invariant — it ticks at a constant rate regardless of CPU frequency scaling (constant_tscin/proc/cpuinfo). - HPET (High Precision Event Timer): a motherboard-level counter, at least 64 bits wide, running at ≥10 MHz. Accessible via MMIO. Slower to read than TSC but shared across CPUs.
- ARM Generic Timer (arch timer): a per-CPU counter exposed via
CNTVCT_EL0in AArch64, running at a fixed frequency (often 24 MHz) stored inCNTFRQ_EL0. This is what most ARM Linux systems use. jiffiesitself is NOT a hardware counter — it is a software counter maintained by the kernel. More on that shortly.
The kernel wraps all of these behind a single abstraction: struct clocksource. Each clocksource has:
- A
read()function that returns a rawu64counter value. - A
multandshiftused to convert counter cycles to nanoseconds:ns = (cycles * mult) >> shift. - A
rating(1–499) expressing quality; higher is better. TSC with verified constant rate gets 300, HPET gets 250,jiffiesas a clocksource gets 1.
The kernel picks the highest-rated clocksource. You can inspect the current choice:
cat /sys/devices/system/clocksource/clocksource0/current_clocksource
# tsc
cat /sys/devices/system/clocksource/clocksource0/available_clocksource
# tsc hpet acpi_pm
The mult/shift trick is elegant: because floating-point is forbidden in the kernel, all time math must be integer arithmetic. If your counter runs at 3.2 GHz, you can't divide by 3,200,000,000 cheaply. Instead, the kernel pre-computes mult = (1e9 << shift) / freq_hz at boot. Multiplying cycles by mult and right-shifting by shift gives nanoseconds, all in integer math with controllable precision.
Analogy: think of the clocksource as an odometer. It measures elapsed distance (cycles). It never resets. Higher-rated sources are more accurate odometers.
/* Reading the current time via clocksource internals (simplified) */
struct clocksource *cs = curr_clocksource; /* kernel global */
u64 cycles = cs->read(cs); /* e.g. rdtsc */
u64 ns = clocksource_cyc2ns(cycles, cs->mult, cs->shift);
/* clocksource_cyc2ns: return (cycles * mult) >> shift */
02Clock Events: The Interrupt Side of Time
A clocksource tells you what time it is now. But to make something happen in the future, you need a different piece of hardware: a clock event device (struct clock_event_device). This device can be programmed to fire an interrupt at a specific future counter value (oneshot mode) or at regular intervals (periodic mode).
Examples:
- LAPIC timer (x86): per-CPU, accessible very cheaply, supports both oneshot and periodic. This is the primary clock event device on x86 SMP systems.
- ARM Generic Timer (
arch_timer): the same hardware that providesCNTVCT_EL0for reading also has a comparator registerCNTP_CVAL_EL0— write a future value there and the CPU fires an interrupt when the counter reaches it. So on ARM, the same hardware serves as both clocksource and clock event. - HPET comparators: the HPET has multiple comparator channels, each of which can fire an interrupt.
Key clock_event_device fields:
set_next_event(delta, dev): program the next interruptdeltacycles from now.set_state_periodic(dev): switch to periodic mode.set_state_oneshot(dev): switch to oneshot mode.features: flags likeCLOCK_EVT_FEAT_ONESHOT,CLOCK_EVT_FEAT_PERIODIC.event_handler: the function called when the interrupt fires. This is the hook that connects hardware interrupts to the rest of the kernel's timer machinery.
The separation of clocksource (reading) and clock_event_device (programming future interrupts) is deliberate and beautiful: on some architectures the same hardware does both (ARM generic timer), but on others they are separate chips (TSC for reading, LAPIC for events). The abstraction hides this completely.
When CONFIG_HZ=1000 (periodic mode), the clock event device fires every 1ms. In tickless mode it fires only when the next timer actually expires — which might be 50ms from now, or even longer.
/* Simplified: programming a clock event (from kernel/time/clockevents.c) */
static int lapic_next_event(unsigned long delta,
struct clock_event_device *evt)
{
apic_write(APIC_TMICT, delta); /* write initial count */
return 0;
}
struct clock_event_device lapic_clockevent = {
.name = "lapic",
.features = CLOCK_EVT_FEAT_PERIODIC | CLOCK_EVT_FEAT_ONESHOT,
.set_next_event = lapic_next_event,
.rating = 100,
};
03jiffies, HZ, and the Tick
The jiffy is the kernel's fundamental unit of coarse time. It is simply an integer counter, incremented once per tick. A tick is one clock interrupt. The kernel configuration constant HZ defines how many ticks (and thus jiffy increments) happen per second:
HZ=100→ tick every 10ms (low latency profiles, embedded, some older configs)HZ=250→ tick every 4ms (Debian/Ubuntu default)HZ=1000→ tick every 1ms (real-time distributions, audio workstations)
So: 1 jiffy = 1/HZ seconds.
The variable jiffies (on 64-bit kernels actually jiffies_64, with jiffies as the lower 32 bits) lives in <linux/jiffies.h> and is readable from any kernel code. It starts at some value at boot and counts up forever — or rather, wraps around after about 49 days on a 32-bit counter at HZ=1000.
Kernel code uses jiffies for coarse timers:
unsigned long timeout = jiffies + msecs_to_jiffies(200); /* 200ms from now */
/* ... later ... */
if (time_after(jiffies, timeout)) { /* handle timeout */ }
The helper time_after(a, b) handles wrap-around correctly using modular arithmetic — this is critical and easy to get wrong without it.
Jiffies are also used for scheduling: the scheduler tick fires every jiffy, checks if the current task's time slice has expired, and triggers a reschedule if needed. This is the clock tick driving the entire CFS (Completely Fair Scheduler).
Why does HZ matter to you as a user? If HZ=100, the smallest sleep you can reliably get is 10ms, even if you ask for 1ms. The kernel rounds up to the next tick boundary. Higher HZ gives finer granularity but increases overhead: at HZ=1000, the timer interrupt fires 1000 times per second per CPU — 1000 context switches to/from interrupt handler. This was a real source of power consumption before NO_HZ.
Analogy: jiffies is like the second hand of a clock that only moves in discrete steps. If HZ=100 it's moving in 10ms steps. You can ask "what time is it to the nearest step" but no finer — for finer precision you need the hrtimer system.
#include <linux/jiffies.h>
#include <linux/delay.h>
/* Convert and compare */
unsigned long future = jiffies + msecs_to_jiffies(500); /* 500 ms */
if (time_before(jiffies, future))
pr_info("Still waiting...\n");
/* jiffies_to_msecs, usecs_to_jiffies, etc. all live in <linux/jiffies.h> */
/* On 64-bit: use get_jiffies_64() to avoid races with 32-bit truncation */
u64 j64 = get_jiffies_64();
04ktime_t: Representing Nanosecond Time
For high-resolution work, jiffies are too coarse. The kernel needs to represent and compare timestamps at nanosecond precision. Enter ktime_t.
ktime_t is defined in <linux/ktime.h>. On 64-bit kernels it is simply a typedef s64 ktime_t — a signed 64-bit integer storing nanoseconds since some epoch. On older 32-bit kernels it was a union with separate seconds and nanoseconds fields, but that complexity is gone from modern kernels.
Key operations:
ktime_t now = ktime_get(); /* monotonic clock, nanoseconds */
ktime_t wall = ktime_get_real(); /* wall-clock (CLOCK_REALTIME) */
ktime_t boot = ktime_get_boottime(); /* includes suspend time */
s64 delta_ns = ktime_to_ns(ktime_sub(b, a)); /* b - a in ns */
ktime_t future = ktime_add_ms(now, 100); /* 100 ms from now */
ktime_get() is the workhorse: it reads the current clocksource, applies the mult/shift conversion, and adds it to a base offset maintained by the timekeeping subsystem. It returns a monotonically increasing value — it never goes backward, even across NTP adjustments (which only affect CLOCK_REALTIME/ktime_get_real()).
The distinction between clock types matters:
ktime_get()→CLOCK_MONOTONIC: never jumps, never goes back. Use for measuring intervals.ktime_get_real()→CLOCK_REALTIME: wall clock, can jump (NTP,settimeofday). Use for timestamps you'll compare with external systems.ktime_get_boottime()→CLOCK_BOOTTIME: like monotonic but includes time spent suspended. Important for mobile devices that sleep.ktime_get_clocktai()→CLOCK_TAI: International Atomic Time, no leap seconds. Used in networking (PTP/IEEE 1588).
The internal conversion from raw cycles to ktime_t nanoseconds happens inside update_wall_time() (called on each tick or hrtimer interrupt), which reads the clocksource counter, converts cycles to nanoseconds using the mult/shift, and updates the tk_core struct (the timekeeping core).
#include <linux/ktime.h>
#include <linux/timekeeping.h>
void measure_something(void)
{
ktime_t start, end;
s64 elapsed_ns;
start = ktime_get();
do_work();
end = ktime_get();
elapsed_ns = ktime_to_ns(ktime_sub(end, start));
pr_info("Work took %lld ns\n", elapsed_ns);
}
05Low-Resolution Timers: struct timer_list
The classic kernel timer, struct timer_list, is a jiffy-granularity, software timer. It fires a callback function after a specified number of jiffies. It is the right tool when you need something like "retry this operation in 200ms" and sub-millisecond precision is irrelevant.
Internally, the kernel maintains a timer wheel — a hash-table-like structure where each slot corresponds to a jiffy value. When a timer is armed with mod_timer(timer, jiffies + N), it is placed into the wheel slot for jiffy jiffies + N. On each tick, the wheel advances, and all timers whose slot is now current are fired.
The timer wheel in modern kernels (since 4.8) has five levels of cascading buckets, like a multi-level odometer, giving O(1) insertion and O(1) expiry for the common case:
- Level 0: 64 buckets of 1 jiffy each (nearest 64 jiffies)
- Level 1: 64 buckets of 8 jiffies each
- Level 2: 64 buckets of 64 jiffies each
- etc.
Usage:
#include <linux/timer.h>
struct timer_list my_timer;
void my_callback(struct timer_list *t)
{
pr_info("Timer fired!\n");
/* Re-arm if needed: */
mod_timer(t, jiffies + msecs_to_jiffies(1000));
}
void setup(void)
{
timer_setup(&my_timer, my_callback, 0);
mod_timer(&my_timer, jiffies + msecs_to_jiffies(1000));
}
void teardown(void)
{
del_timer_sync(&my_timer); /* wait for callback to finish */
}
Important: timer_list callbacks run in softirq context (specifically TIMER_SOFTIRQ). This means:
- No sleeping allowed.
- No
mutex_lock(usespin_lockonly). - Preemption is disabled.
Also note: del_timer_sync() vs del_timer(). The _sync variant waits until any currently-executing callback finishes on another CPU before returning. Always use _sync in module cleanup code to avoid use-after-free bugs.
For sub-jiffy precision or tight latency requirements, use hrtimer instead.
#include <linux/timer.h>
#include <linux/jiffies.h>
static struct timer_list poll_timer;
static void poll_callback(struct timer_list *t)
{
/* softirq context: no sleep, no mutex */
check_hardware_status();
mod_timer(t, jiffies + HZ); /* re-arm: 1 second */
}
static int __init mydrv_init(void)
{
timer_setup(&poll_timer, poll_callback, 0);
mod_timer(&poll_timer, jiffies + HZ);
return 0;
}
static void __exit mydrv_exit(void)
{
del_timer_sync(&poll_timer);
}
06High-Resolution Timers: hrtimer
The hrtimer subsystem provides nanosecond-precision timers, independent of the jiffy tick. Rather than the timer wheel, hrtimers are stored in a per-CPU red-black tree sorted by expiry time. The soonest-expiring hrtimer is always at the leftmost node — O(log n) insertion, O(1) lookup of next expiry.
When an hrtimer is armed, the clock event device is reprogrammed (in oneshot mode) to fire at exactly the hrtimer's expiry time. When the interrupt fires, the hrtimer callback runs, and the clockevent is reprogrammed for the next hrtimer in the tree.
This is fundamentally different from the jiffy timer wheel: hrtimers are event-driven, not tick-driven. The interrupt fires exactly when needed, not on every jiffy.
hrtimer API:
#include <linux/hrtimer.h>
struct hrtimer my_hrtimer;
enum hrtimer_restart my_hr_callback(struct hrtimer *timer)
{
do_precise_work();
/* To re-arm: */
hrtimer_forward_now(timer, ms_to_ktime(10)); /* 10 ms */
return HRTIMER_RESTART;
/* Or: return HRTIMER_NORESTART; to stop */
}
void setup(void)
{
hrtimer_init(&my_hrtimer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
my_hrtimer.function = my_hr_callback;
hrtimer_start(&my_hrtimer, ms_to_ktime(10), HRTIMER_MODE_REL);
}
void teardown(void)
{
hrtimer_cancel(&my_hrtimer);
}
Clock IDs for hrtimers:
CLOCK_MONOTONIC: most common, never jumps.CLOCK_REALTIME: wall clock, can jump.CLOCK_BOOTTIME: includes suspend.CLOCK_TAI: TAI time.
Mode flags:
HRTIMER_MODE_REL: expiry is relative to now.HRTIMER_MODE_ABS: expiry is an absolute ktime_t value.HRTIMER_MODE_PINNED: pin to current CPU (avoid migration).
hrtimer callbacks run in hardirq context (from the clockevent interrupt handler) unless HRTIMER_MODE_SOFT is used (which defers to a softirq). This means:
- Absolutely no sleeping.
- No blocking operations.
- Very short execution time expected.
The precision you actually get depends on the hardware. On a typical x86 desktop with TSC + LAPIC, you can reliably get ~1–10 µs jitter. On a PREEMPT_RT kernel with careful CPU isolation (see isolcpus, nohz_full), sub-microsecond jitter is achievable.
hrtimers are also the foundation of nanosleep(2), clock_nanosleep(2), POSIX interval timers, and the scheduler's SCHED_DEADLINE policy.
#include <linux/hrtimer.h>
#include <linux/ktime.h>
static struct hrtimer precise_timer;
static ktime_t interval;
static enum hrtimer_restart precise_callback(struct hrtimer *timer)
{
/* hardirq context by default */
update_dac_buffer(); /* must be very fast */
/* Advance by exactly one interval (avoids drift accumulation) */
hrtimer_forward_now(timer, interval);
return HRTIMER_RESTART;
}
static int __init audio_init(void)
{
interval = ktime_set(0, 125000); /* 125 µs = 8 kHz */
hrtimer_init(&precise_timer, CLOCK_MONOTONIC, HRTIMER_MODE_REL);
precise_timer.function = precise_callback;
hrtimer_start(&precise_timer, interval, HRTIMER_MODE_REL);
return 0;
}
07Tickless Operation: NO_HZ, Dynticks, and Dynamic Tick
The traditional kernel with CONFIG_HZ=1000 fires a timer interrupt 1000 times per second per CPU, even when the CPU is completely idle. This is wasteful:
- Power: waking the CPU 1000x/sec prevents deep sleep states (C-states), burning battery.
- Virtualization: generates 1000 unnecessary VM exits per CPU per second, slowing the hypervisor.
- Real-time: the tick itself is a source of jitter for latency-sensitive applications.
The solution is CONFIG_NO_HZ — the "tickless" or "dynticks" kernel. There are three variants:
-
CONFIG_HZ_PERIODIC: Old behavior. Tick fires every 1/HZ seconds unconditionally. Don't use this on modern systems. -
CONFIG_NO_HZ_IDLE(default for most distros): The tick is suppressed when the CPU is idle. When the last runnable task goes to sleep, the kernel looks at the nearest pending timer (from both the timer wheel and the hrtimer tree), programs the clockevent to fire at exactly that time, and lets the CPU sleep until then. When the CPU wakes up (either from a real interrupt or from the timer), the tick resumes. This saves significant power on idle CPUs. -
CONFIG_NO_HZ_FULL: The tick is suppressed even when the CPU has exactly one runnable task (plus kernel threads). This is for dedicated real-time or HPC CPUs. Used withnohz_full=3,4,5in the kernel command line to designate specific CPUs as "full tickless". These CPUs avoid the 1000x/sec interference entirely. The tradeoff: certain kernel work (likedo_timer()andcalc_global_load()) must be handled by a designated "housekeeping" CPU (typically CPU 0) which always ticks.
How does NO_HZ_IDLE work mechanically?
When cpu_idle_loop() is about to call arch_cpu_idle() (the halt instruction), it first calls tick_nohz_idle_enter(), which:
- Finds the next timer expiry across all timer wheels and hrtimer trees.
- Calls
clockevents_program_event()to program the clockevent for that future time. - Updates
jiffiesto account for the time that will pass while idle.
When the CPU wakes:
tick_nohz_idle_exit()updatesjiffiesby the actual elapsed time.- Any timers that expired during the sleep are fired immediately.
- The tick resumes normally.
You can observe tick behavior:
# Watch timer interrupt counts per CPU
watch -n1 'grep "^LOC\|^TIM" /proc/interrupts'
# On an idle CPU with NO_HZ_IDLE, counts should increment very slowly
The jiffies update during tickless periods is handled by tick_do_update_jiffies64(), which reads the clocksource and computes how many jiffies should have elapsed — this is the bridge between the hardware cycle counter and the software jiffy counter.
# Check NO_HZ configuration on a running kernel
grep -E 'NO_HZ|HZ=' /boot/config-$(uname -r)
# CONFIG_NO_HZ_IDLE=y
# CONFIG_HZ=250
# See nohz_full CPUs (if configured)
cat /sys/devices/system/cpu/nohz_full 2>/dev/null || echo 'not configured'
# Count timer interrupts per second on each CPU (LOC = local APIC timer)
grep 'LOC:' /proc/interrupts
08The Timekeeping Core: xtime, wall_to_monotonic, and update_wall_time()
How does the kernel maintain an accurate wall-clock time across ticks, NTP adjustments, leap seconds, and suspend/resume cycles? The answer is the timekeeping core, centered on struct timekeeper (in kernel/time/timekeeping.c).
Key fields of struct timekeeper:
tkr_mono: astruct tk_read_basecontaining the clocksource pointer, mult, shift, and abaseoffset in nanoseconds. Readingktime_get()reads the clocksource cycles, converts to ns, and addsbase.xtime_sec: the UNIX timestamp in whole seconds (forCLOCK_REALTIME).wall_to_monotonic: offset to convert wall clock to monotonic. Whensettimeofdayjumps the wall clock, this offset is adjusted soCLOCK_MONOTONICremains continuous.offs_real: the offset between monotonic and real-time clocks.offs_boot: the offset accounting for suspend time.
update_wall_time() is called on every tick (or when the system wakes from tickless idle). It:
- Reads the clocksource counter.
- Computes elapsed cycles since last update.
- Converts to nanoseconds (mult/shift).
- Adds to the monotonic base.
- Applies any NTP frequency/phase corrections (from
ntp_tick_length()). - Updates
xtime_secwhen a full second has elapsed. - Handles leap second insertion/deletion.
NTP corrections are applied as a frequency adjustment: the mult value is slightly tweaked up or down (by parts per million) to make the TSC-based time track the NTP reference. This is the software equivalent of trimming a crystal oscillator.
For reading time from userspace with minimal overhead, modern x86 and ARM kernels use the vDSO (virtual Dynamic Shared Object): the kernel maps a read-only page into every process's address space containing a copy of the timekeeping data and the clock_gettime() implementation. Userspace reads time without a syscall — it reads the vDSO data directly, computing the result the same way the kernel would. This is why clock_gettime(CLOCK_MONOTONIC) is so fast (tens of nanoseconds) on Linux.
/* Simplified view of ktime_get() — the actual hot path */
static __always_inline u64 __ktime_get_ns(void)
{
struct tk_read_base *tkr = &tk_core.timekeeper.tkr_mono;
u64 cycles = tkr->clock->read(tkr->clock);
/* delta = cycles since last update */
u64 delta = clocksource_delta(cycles, tkr->cycle_last, tkr->mask);
/* convert cycles → ns, add base */
return tkr->base + clocksource_cyc2ns(delta, tkr->mult, tkr->shift);
}
09Delayed Work: workqueues and schedule_delayed_work()
Both timer_list and hrtimer callbacks run in interrupt context (hardirq or softirq). This means they cannot sleep, allocate memory with GFP_KERNEL, or do any blocking I/O. For deferred work that does need a process context — like touching disk, sleeping, or calling code that might block — the kernel provides workqueues.
A workqueue is a set of kernel threads (workers) that process work items. You submit a work item and a kernel worker thread picks it up and runs it in process context.
struct delayed_work combines an hrtimer with a work item: after a specified delay, the hrtimer fires (in interrupt context), and then queues the actual work item to a workqueue. The workqueue thread runs the callback in process context.
#include <linux/workqueue.h>
struct delayed_work my_dwork;
void my_work_func(struct work_struct *work)
{
/* process context: can sleep, use GFP_KERNEL, etc. */
struct my_data *data = container_of(
container_of(work, struct delayed_work, work),
struct my_data, dwork);
if (need_retry(data))
schedule_delayed_work(&data->dwork, msecs_to_jiffies(100));
}
void setup(struct my_data *data)
{
INIT_DELAYED_WORK(&data->dwork, my_work_func);
schedule_delayed_work(&data->dwork, msecs_to_jiffies(500));
}
void teardown(struct my_data *data)
{
cancel_delayed_work_sync(&data->dwork);
}
The delay argument to schedule_delayed_work() is in jiffies — so granularity is 1/HZ. For sub-jiffy precision in process context, you'd need a different pattern (hrtimer that queues a work item, or schedule_hrtimeout()).
The kernel's system_wq workqueue is used by schedule_work() / schedule_delayed_work(). Other workqueues exist for specific purposes:
system_highpri_wq: for high-priority worksystem_unbound_wq: work that is not bound to a specific CPUsystem_freezable_wq: frozen during suspend
You can create your own workqueue with alloc_workqueue() if you need isolation or specific concurrency guarantees.
cancel_delayed_work_sync() is the equivalent of del_timer_sync() — it cancels the pending work AND waits for any currently-running callback to finish.
#include <linux/workqueue.h>
#include <linux/jiffies.h>
/* Pattern: exponential backoff retry using delayed_work */
struct retry_state {
struct delayed_work dwork;
unsigned int attempts;
};
static void retry_fn(struct work_struct *work)
{
struct delayed_work *dw = to_delayed_work(work);
struct retry_state *rs = container_of(dw, struct retry_state, dwork);
if (try_operation() == 0) {
pr_info("Success after %u attempts\n", rs->attempts);
return;
}
rs->attempts++;
/* exponential backoff, cap at 32 seconds */
unsigned long delay = min(HZ << rs->attempts, 32 * HZ);
schedule_delayed_work(&rs->dwork, delay);
}
10nanosleep(), schedule_timeout(), and the User-Kernel Bridge
How do user-space sleep functions like sleep(3), usleep(3), and nanosleep(2) actually work?
nanosleep(2) → hrtimer_nanosleep() in the kernel:
- Allocates an hrtimer on the current task's stack.
- Arms it for the requested duration (
CLOCK_MONOTONICorCLOCK_REALTIME). - Calls
schedule()— the task goes to sleep (stateTASK_INTERRUPTIBLE). - When the hrtimer fires, it calls
wake_up_process()on the sleeping task. - The task becomes runnable, the scheduler picks it up, and
nanosleepreturns.
If a signal arrives before the timer fires, the task is woken early and nanosleep returns -EINTR with the remaining time in the rem parameter — this is why robust code loops on nanosleep.
Inside the kernel, schedule_timeout(timeout_jiffies) does the same thing but with a timer_list (jiffy granularity) instead of an hrtimer:
/* Inside a kernel thread or syscall context: */
set_current_state(TASK_INTERRUPTIBLE);
schedule_timeout(msecs_to_jiffies(100)); /* sleep ~100ms */
/* or the more convenient: */
msleep(100); /* uninterruptible (TASK_UNINTERRUPTIBLE) */
msleep_interruptible(100); /* interruptible */
For sub-millisecond busy waits, the kernel provides:
udelay(n): busy-loop fornmicroseconds. Uses a calibrated loop count (loops_per_jiffy).ndelay(n): busy-loop fornnanoseconds.mdelay(n): busy-loop fornmilliseconds (=udelay(n*1000), not recommended for long durations).
Busy waits burn CPU and hold any spinlocks, so use them only for very short durations (< ~10µs). For longer delays, always use a sleeping variant.
The usleep_range(min, max) function is preferred over udelay for delays in the 10µs–20ms range: it uses an hrtimer with a slack (the range), allowing the timer to coalesce with nearby timers and saving power.
/* Kernel driver: three ways to wait, each appropriate in different contexts */
/* 1. Atomic context (interrupt handler, spinlock held): MUST busy-wait */
udelay(5); /* 5 µs busy-wait — OK for very short delays */
/* 2. Process context, interruptible: */
if (msleep_interruptible(50)) {
/* Woken by signal — handle it */
return -ERESTARTSYS;
}
/* 3. Process context, power-friendly, non-latency-critical: */
usleep_range(10000, 20000); /* sleep 10-20 ms, allows coalescing */
11Putting It All Together: The Timer Interrupt Path
Let's trace exactly what happens on a modern x86 system with NO_HZ_IDLE when a timer interrupt fires, to see how all the pieces connect:
-
Hardware: The LAPIC timer (a
clock_event_device) was programmed to fire at cycle T. At cycle T, it raises IRQ 0 (the local timer interrupt). -
CPU: Saves registers, enters interrupt context, calls the IRQ handler:
local_apic_timer_interrupt(). -
tick_handle_oneshot_broadcast()orhrtimer_interrupt(): The clock event'sevent_handleris called. For hrtimers, this ishrtimer_interrupt(). -
hrtimer_interrupt(): a. Reads the current time viaktime_get(). b. Walks the per-CPU hrtimer red-black tree, firing all timers whose expiry ≤ now. c. For each expired hrtimer, calls its.functioncallback. d. Finds the next pending hrtimer expiry. e. Reprograms the clockevent for that future time. -
update_process_times()(if this is also a tick): Updatesjiffies, calls the scheduler tick (scheduler_tick()), updates CPU usage statistics, and kicks off theTIMER_SOFTIRQto run expiredtimer_listtimers. -
TIMER_SOFTIRQ(runs after IRQ returns): Callsrun_timer_softirq(), which walks the timer wheel for any expiredtimer_listtimers and calls their callbacks. -
Back to userspace (or the idle loop): Returns via
iret/eret.
This means in a single timer interrupt, you can have:
- Multiple hrtimer callbacks run (in hardirq context)
- The scheduler tick executed
- Multiple timer_list callbacks run (in softirq context)
All of this typically takes < 10µs on modern hardware for a handful of timers. The kernel is carefully designed so the interrupt path is fast — expensive work is always deferred to softirq or workqueues.
/* Trace timer interrupts in real time using perf or ftrace */
# Method 1: ftrace hrtimer events
cd /sys/kernel/debug/tracing
echo 'hrtimer_expire_entry hrtimer_expire_exit' > set_event
echo 1 > tracing_on
sleep 0.1
echo 0 > tracing_on
cat trace | head -40
# Method 2: see per-CPU hrtimer stats
cat /proc/timer_stats # may require CONFIG_TIMER_STATS (older kernels)
# Method 3: count local timer interrupts per CPU
grep '^LOC:' /proc/interrupts
Key insights
- ✦jiffies and hrtimers are two completely separate timer systems that coexist: jiffies for coarse, low-overhead work; hrtimers for precision. Most real kernel code uses both.
- ✦HZ is a compile-time constant, but with NO_HZ_IDLE the tick does NOT fire HZ times per second on idle CPUs — the tick fires only when needed. jiffies is updated in software to account for the skipped ticks.
- ✦ktime_t is just a nanosecond count in a 64-bit integer. All the complexity lives in the conversion from hardware cycles to nanoseconds (mult/shift), which must use integer arithmetic with no floating point.
- ✦The separation of clocksource (reading time) and clock_event_device (triggering future interrupts) is a deliberate abstraction that allows the kernel to mix-and-match hardware: read from TSC, interrupt from LAPIC.
- ✦timer_list callbacks run in softirq context; hrtimer callbacks run in hardirq context (by default). Neither can sleep. Use delayed_work when your callback needs process context.
- ✦CLOCK_MONOTONIC never jumps backward; CLOCK_REALTIME can. For measuring intervals, always use CLOCK_MONOTONIC. This distinction has caused real production bugs in systems that used wall-clock time for timeouts.
- ✦The vDSO makes clock_gettime() a userspace operation (no syscall). It works because the kernel maps its timekeeping data read-only into every process — one of the cleverest performance tricks in Linux.
- ✦hrtimer_forward_now() vs hrtimer_forward(): use _now() in the callback to avoid drift accumulation. If your callback runs 1µs late and you forward by a fixed interval, you accumulate error. _now() resets the base to the actual current time.
Common pitfalls
- ⚠Using `jiffies` comparison without time_before()/time_after() helpers: raw `jiffies > deadline` breaks on 32-bit wraparound (happens every ~49 days at HZ=1000). Always use the helper macros.
- ⚠Calling del_timer() instead of del_timer_sync() in module cleanup: if the callback is running on another CPU when you free your data structure, you get a use-after-free. Always use del_timer_sync() (and cancel_delayed_work_sync()).
- ⚠Sleeping inside a timer_list or hrtimer callback: both run in interrupt context where sleeping is forbidden. The kernel may warn or deadlock. Use delayed_work if you need process context.
- ⚠Using CLOCK_REALTIME for timeout/interval measurement: NTP or settimeofday can move the clock backward mid-operation, causing a timeout that should fire in 100ms to wait for minutes or fire immediately. Use CLOCK_MONOTONIC.
- ⚠Forgetting that `msecs_to_jiffies(1)` may return 0 at HZ=100 (since 1ms < 1 jiffy = 10ms). Always check the return value or use a larger minimum. `usecs_to_jiffies()` can also return 0 for small values.
- ⚠Accumulating drift in periodic hrtimers by using HRTIMER_MODE_REL and re-arming with hrtimer_start() each time: each callback incurs the callback execution time as drift. Use hrtimer_forward_now() which correctly advances the expiry.
- ⚠Forgetting that NO_HZ_FULL requires a dedicated housekeeping CPU (isolcpus + nohz_full kernel params). Without proper configuration, various kernel subsystems that expect regular ticks will malfunction silently.
Practice
- 01medium
Write a minimal loadable kernel module that uses an hrtimer to fire every 500 microseconds and logs (via pr_info) how many nanoseconds have actually elapsed since the last fire (measure with ktime_get()). Load it, collect 10 seconds of dmesg output, and compute the jitter: mean, max, and standard deviation of the inter-fire intervals.
- 02easy
Write a kernel module that implements a periodic task using delayed_work instead of a timer_list. The work function should: (1) allocate a small buffer with kmalloc(GFP_KERNEL), (2) memset it, (3) kfree it, (4) reschedule itself with a 100ms delay. This proves that delayed_work runs in process context (sleeping allocations are allowed). Verify it works and explain why this would crash if implemented with timer_list.
- 03easy
Read /sys/devices/system/clocksource/clocksource0/ on your machine. Switch the clocksource from 'tsc' to 'hpet' (if available) or 'acpi_pm' using the available_clocksource file. Write a small C program that calls clock_gettime(CLOCK_MONOTONIC) in a tight loop 10 million times and measures the total time. Compare the throughput (calls/second) between clocksources. Explain the difference in terms of the hardware read cost.
- 04medium
Write a userspace C program that demonstrates CLOCK_MONOTONIC vs CLOCK_REALTIME behavior under a simulated clock jump. Use two threads: thread A sleeps for 5 seconds using clock_nanosleep(CLOCK_REALTIME, ...). Thread B (after 1 second) runs `sudo date -s '1 hour ago'` via system(). Observe: does thread A wake up early, on time, or late? Repeat with CLOCK_MONOTONIC. Document the difference.
- 05hard
Enable full tickless mode on a spare CPU: add `isolcpus=1 nohz_full=1 rcu_nocbs=1` to GRUB_CMDLINE_LINUX in /etc/default/grub, update-grub, and reboot. Then: (a) run a CPU-bound process pinned to CPU 1 with taskset, (b) in parallel watch `grep 'LOC:' /proc/interrupts` updating every 100ms. Verify that CPU 1's local timer interrupt count barely moves while the process runs. Compare with CPU 0 (the housekeeping CPU).
Resources
The authoritative source: covers NO_HZ, hrtimers, timer_list, timekeeping, and more with kernel developer authorship.
Thomas Gleixner's original article introducing hrtimers — explains the motivation for separating clocksource from clock_event_device.
Covers the tickless idle patch and how NO_HZ_IDLE works, written when it was first merged.
Chapter 7 covers time, delays, and deferred work at a driver-developer level. Free PDF at the URL.
Bootlin's kernel training slides have an excellent timers chapter with diagrams of the hrtimer and timer wheel.
Detailed explanation of the 5-level timer wheel rewrite in kernel 4.8, with complexity analysis.
Chapter 6 covers kernel timing in exhaustive depth. The gold standard reference book, though somewhat dated (2.6 era).
Gentle introduction to timer_list and hrtimer with working example code — good first read before the kernel docs.
Compile `#include <x86intrin.h>\nuint64_t t(){return __rdtsc();}` to see exactly what the TSC read looks like at the assembly level.
Checkpoints — you can now…
- ✓You can explain the difference between a clocksource and a clock_event_device, and give a concrete example of each on x86 and ARM.
- ✓You can write a working kernel module using timer_list, correctly arm it, handle wrap-around using time_after(), and clean it up with del_timer_sync().
- ✓You can write a working kernel module using hrtimer with HRTIMER_MODE_REL, use hrtimer_forward_now() to avoid drift, and return HRTIMER_RESTART correctly.
- ✓You can explain why NO_HZ_IDLE saves power and what happens to jiffies when the CPU is idle for 500ms with no tick.
- ✓You can explain when to use timer_list vs hrtimer vs delayed_work, citing the context constraints (interrupt vs process) as the primary decision factor.
- ✓You can read /proc/interrupts and /sys/devices/system/clocksource/ and interpret what they tell you about your system's timer configuration.
- ✓You can explain the CLOCK_MONOTONIC vs CLOCK_REALTIME distinction and articulate a real scenario where using the wrong one causes a bug.