systemd & Init Systems

After the kernel finishes its own initialization — setting up memory management, the scheduler, interrupt handlers, and mounting the initial RAM filesystem — it needs to hand control to userspace. It does this by directly executing a single binary, which receives process ID 1.

PID 1 is special in ways that no other process is:

Orphan adoption. When any process dies, its children become orphans. The kernel re-parents orphaned processes to PID 1. This means PID 1 is responsible for calling wait() on all orphaned processes in the system, or those processes will become zombies (dead but occupying a slot in the process table) forever. A broken PID 1 that does not reap orphans will eventually exhaust the process table.

Signal immunity. The kernel never delivers SIGKILL or SIGTERM to PID 1 unless the process has explicitly installed a handler for them. This is a safety mechanism — if someone accidentally sends SIGKILL to PID 1, the machine should not instantly become unusable. Internally, the kernel checks is_global_init(p) before delivering the signal.

The subreaper mechanism. Since Linux 3.4, a process can call prctl(PR_SET_CHILD_SUBREAPER, 1) to make itself the reaper for its own subtree of descendants. This is how container runtimes like containerd avoid depending on the host's PID 1.

Mount namespace root. PID 1 starts with the root mount namespace. All other mount namespaces are derived from it (unless a new one is created with unshare).

The kernel locates the init binary by trying, in order: the init= kernel command-line argument, then /sbin/init, /etc/init, /bin/init, and finally /bin/sh. If none are found, the kernel panics with "No working init found." — one of the most instructive kernel panics to see, because it tells you exactly what the kernel was looking for.

// Simplified kernel path: how PID 1 is launched
// (from init/main.c in the Linux kernel source)

static int run_init_process(const char *init_filename)
{
    const char *const *p;
    argv_init[0] = init_filename;
    // kernel_execve replaces the current kernel thread
    // with the userspace binary
    return kernel_execve(init_filename,
                         (const char *const *)argv_init,
                         (const char *const *)envp_init);
}

void __init kernel_init_freeable(void)
{
    // ... hardware init, rootfs mount ...
    
    // Try user-specified init= first
    if (execute_command) {
        if (!run_init_process(execute_command))
            return;
        panic("Requested init %s failed.", execute_command);
    }
    // Fall through to well-known paths
    if (!run_init_process("/sbin/init") ||
        !run_init_process("/etc/init")  ||
        !run_init_process("/bin/init")  ||
        !run_init_process("/bin/sh"))
        return;
    panic("No working init found. "
          "Try passing init= option to kernel.");
}

SysV init (from Unix System V, 1983) dominated Linux for its first two decades. Its model is simple enough to fit in your head:

Runlevels. The system is always in one of 7 runlevels (0–6). Runlevel 0 = halt, 1 = single-user (recovery), 2–5 = various multi-user configurations (distro-specific), 6 = reboot. Changing runlevel means stopping some services and starting others.

/etc/inittab. The init daemon reads this file on startup and on SIGHUP. Each line describes a process to run, in which runlevels, and what to do if it exits. The special sysinit entry runs /etc/init.d/rcS first, which mounts filesystems, sets the hostname, configures networking at a basic level.

/etc/init.d/ scripts. Each service has a shell script here that accepts start, stop, restart, status as arguments. The scripts for a given runlevel are symlinked into /etc/rc<N>.d/ with names like S20ssh (start, priority 20) or K80apache2 (kill/stop, priority 80). Init runs the K scripts in numeric order, then the S scripts in numeric order — strictly sequentially.

The sequential problem. If S30networking takes 10 seconds to get a DHCP lease, everything after it waits. If S50sshd depends on networking, that is fine — it starts after. But if S60httpd and S70postgres are independent of each other, they still start one after the other. On modern hardware with many cores, this is pure waste.

Dependency modeling. SysV init has no formal dependency model. The ordering is entirely by priority number, maintained by hand. If a distro packager puts S20foo and S19bar but foo actually needs bar, you have a race condition in the start scripts. This happened constantly.

No process supervision. If sshd crashes at 3am, SysV init does not restart it. You needed external tools (daemontools, runit, monit) layered on top.

SysV's shell-script approach is auditable and transparent. You can read exactly what any service does. That transparency was genuinely valuable. The problems were structural: no parallelism, no dependency solver, no supervision.

#!/bin/sh
# Example SysV-style init script: /etc/init.d/myservice
# This is what you'd write for every service pre-systemd.

NAME=myservice
DAEMON=/usr/bin/myservice
PIDFILE=/var/run/$NAME.pid

case "$1" in
  start)
    echo "Starting $NAME"
    # Start daemon, write PID to file
    start-stop-daemon --start --quiet \
        --make-pidfile --pidfile $PIDFILE \
        --background --exec $DAEMON
    ;;
  stop)
    echo "Stopping $NAME"
    start-stop-daemon --stop --quiet \
        --pidfile $PIDFILE
    rm -f $PIDFILE
    ;;
  restart)
    $0 stop
    $0 start
    ;;
  status)
    if [ -f $PIDFILE ] && kill -0 $(cat $PIDFILE) 2>/dev/null; then
        echo "$NAME is running (PID $(cat $PIDFILE))"
    else
        echo "$NAME is not running"
    fi
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac
# No dependency declaration, no automatic restart,
# no structured logging. You're on your own.

systemd's core insight is to make dependencies explicit and machine-readable, then use them to compute a startup order and execute it with maximum parallelism.

Units. The fundamental abstraction is a unit — a declarative description of a system resource. There are many unit types: .service (a daemon), .socket (a network/IPC socket), .target (a synchronization point, like a runlevel), .mount (a filesystem mountpoint), .timer (a cron-like scheduled trigger), .path (watch a file/directory), .device (a kernel device), .slice (a cgroup subtree), .scope (an externally-managed set of processes).

Unit files live in (in priority order): /etc/systemd/system/ (admin-managed), /run/systemd/system/ (runtime-generated), /usr/lib/systemd/system/ (package-installed). A file in /etc overrides one with the same name in /usr/lib.

The dependency graph. Each unit can declare:

• Requires= — if this unit starts, the required unit must also start; if required unit fails, this unit fails too
• Wants= — softer: try to start the wanted unit, but don't fail if it can't
• Before= / After= — ordering constraints (independent of Requires/Wants)
• BindsTo= — like Requires, but also stops this unit if the bound unit stops
• Conflicts= — cannot be active at the same time

On startup, systemd reads all unit files, builds a directed acyclic graph (DAG) of these dependencies, does a topological sort to find the parallelism opportunities, and starts as many units as possible simultaneously — subject only to After= ordering.

Targets as milestones. There are no runlevels in systemd, but targets play the same conceptual role and more. multi-user.target is roughly runlevel 2–4. graphical.target adds a display manager. network-online.target is reached only after the network has an IP address. Services declare After=network-online.target if they need real connectivity.

systemctl is your interface.

systemctl status nginx.service    # show state, recent log lines
systemctl start nginx.service     # start now
systemctl enable nginx.service    # create symlink so it starts at boot
systemctl disable nginx.service   # remove symlink
systemctl daemon-reload           # re-read unit files after editing
systemctl list-units --type=service --state=failed
systemctl isolate rescue.target   # switch to single-user mode

Why parallel startup is safe. Socket activation (covered below) is key: a service that provides a socket can be "ready" (the socket exists and accepts connections) before its backing process has even started. Consumers that connect to the socket will be transparently buffered. This breaks the most common dependency cycle in startup: A needs B's socket, and B needs A's socket.

# /etc/systemd/system/myapp.service
# A real, minimal-but-complete service unit file

[Unit]
Description=My Application Server
Documentation=https://myapp.example.com/docs
# Don't start until network has an IP address
After=network-online.target
Wants=network-online.target

[Service]
# 'simple': systemd considers the service up when the
# main process starts (exec line). For daemons that
# notify readiness, use Type=notify instead.
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/server --config /etc/myapp/config.toml
# On failure: restart after 5 seconds, up to 5 times
Restart=on-failure
RestartSec=5s
StartLimitIntervalSec=60s
StartLimitBurst=5
# Security hardening (free, just add the lines)
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/myapp /var/log/myapp

[Install]
# This target is what 'systemctl enable' hooks us into
WantedBy=multi-user.target

One of the trickiest parts of service management is knowing when a service is ready. "Ready" does not mean "the process started" — it means "the service is accepting requests." A database server might take 30 seconds after exec to replay its write-ahead log before it can serve queries. If another service that depends on it starts too early, it will get connection refused errors.

SysV init had no answer to this except sleep loops in shell scripts (while ! nc -z localhost 5432; do sleep 1; done). systemd provides a formal readiness protocol.

Service types:

• Type=simple (default): the main process is started, and systemd immediately considers the service "active". No readiness signal. Use for services that are ready as soon as the binary starts (simple servers, utilities).

• Type=forking: the classic Unix double-fork daemon model. The ExecStart process forks, the parent exits, and the child becomes the daemon. systemd waits for the parent to exit and then uses a PIDFile to track the child. Fragile — PID files can be stale, and tracking the "real" main process is unreliable.

• Type=notify: the process sends a readiness notification to systemd via the sd_notify() API (a simple datagram on a Unix socket whose path is in $NOTIFY_SOCKET). systemd waits for READY=1 before considering the service active. This is the correct choice for any daemon that has an initialization phase.

• Type=notify-reload: like notify, but also uses sd_notify() for reload completion.

• Type=dbus: the service is considered ready when it acquires a specific D-Bus name.

• Type=oneshot: the service runs a command, exits, and systemd considers it "done". Used for setup tasks. Combine with RemainAfterExit=yes to keep the unit "active" after the command finishes (useful for iptables rules, modprobe, etc.).

• Type=idle: like simple, but the process is not started until all active jobs are dispatched — useful for services that should not compete with startup I/O.

The sd_notify wire protocol is intentionally simple: just a newline-separated list of assignments sent as a UDP datagram to the socket. READY=1 says ready. STATUS=Loading configuration... updates the status string shown in systemctl status. MAINPID=<pid> tells systemd the real PID after a fork. WATCHDOG=1 pets the watchdog. STOPPING=1 says graceful shutdown is in progress.

This protocol is so simple you can implement it in a shell script with a single echo to a Unix socket.

#include <systemd/sd-daemon.h>
#include <stdio.h>
#include <unistd.h>

// Compile: gcc -o myserver myserver.c $(pkg-config --cflags --libs libsystemd)
// Unit file must have Type=notify

int main(void) {
    // --- Expensive initialization ---
    printf("Loading database...\n");
    sleep(3);  // simulate slow init

    // Tell systemd we're ready to serve requests.
    // This unblocks any units with After=myapp.service.
    sd_notify(0, "READY=1\nSTATUS=Listening on port 8080");

    printf("Server ready.\n");

    // --- Main event loop ---
    while (1) {
        // Pet the watchdog if WatchdogSec= is set in the unit.
        // If we stop doing this, systemd will restart us.
        sd_notify(0, "WATCHDOG=1");
        sleep(10);
    }
    return 0;
}

Socket activation is one of systemd's most elegant features, and it solves two problems at once: parallel startup and on-demand activation.

The core idea. Instead of a service creating its own listening socket, systemd creates the socket first and passes the open file descriptor to the service when it starts. Because the socket exists independently of the service process, any other service that needs to connect to it can do so before the backing service has even started — the kernel will buffer the incoming connection in the socket's backlog. This breaks circular startup dependencies.

How it works mechanically. A .socket unit declares the socket (address, type, permissions). When the socket becomes "active", systemd creates the file descriptor and holds it. The paired .service unit is not started immediately — it is started on-demand when the first connection arrives (or at boot if WantedBy= is set in the socket unit).

When systemd eventually launches the service, it passes the pre-opened file descriptors via environment variables: LISTEN_FDS (number of passed FDs) and LISTEN_PID (the PID that should receive them). The FDs themselves start at number 3. The sd_listen_fds() function in libsystemd handles the unpacking.

Socket store. If the service is restarted (due to crash or upgrade), systemd can hold the socket open during the restart. Clients that connect during the restart gap will be buffered, not rejected. This enables zero-downtime restarts for many protocols.

inetd heritage. This is conceptually similar to the ancient inetd "internet superserver" — but with far better performance (no fork-per-connection) and integration with the rest of systemd's dependency/cgroup machinery.

D-Bus activation is a variant: a service is started when a D-Bus name is first requested. Systemd's D-Bus broker uses this to lazily start many system services.

Socket activation is also central to how containers work with systemd: systemd --user instances in containers can use socket activation to expose services without needing to bind privileged ports.

# /etc/systemd/system/echo.socket
[Unit]
Description=Echo Server Socket

[Socket]
# Create a TCP socket on port 7777
ListenStream=127.0.0.1:7777
# Also works for Unix sockets:
# ListenStream=/run/myapp.sock
Accept=no   # Pass socket to service, don't fork per-connection

[Install]
WantedBy=sockets.target

---

# /etc/systemd/system/echo.service
[Unit]
Description=Echo Server (socket-activated)
# Systemd auto-links these via name convention,
# but explicit is clearer:
Requires=echo.socket

[Service]
Type=simple
ExecStart=/usr/local/bin/echo-server
# The server reads from LISTEN_FDS, not from bind()

---

# The server-side code (echo-server.c) using sd_listen_fds:
#include <systemd/sd-daemon.h>
#include <unistd.h>

int main(void) {
    int n = sd_listen_fds(0);
    if (n < 1) { /* fallback: bind ourselves */ }
    // SD_LISTEN_FDS_START == 3
    int fd = SD_LISTEN_FDS_START;
    // fd is already bound+listening — just accept()
    // ...
    return 0;
}

In SysV init, a runlevel is a single integer that describes the global state of the system. Changing the runlevel is an all-or-nothing operation. systemd replaces this with targets — named synchronization milestones that can be composed and layered.

Key system targets:

Compatibility symlinks. For backwards compatibility, /sbin/telinit 3 works because systemd interprets it as systemctl isolate multi-user.target. The SysV runlevel numbers map to targets via symlinks in /usr/lib/systemd/system/runlevel<N>.target.

isolate vs start. systemctl start foo.target activates foo.target and everything it wants/requires, without stopping anything. systemctl isolate foo.target activates everything in the target and stops everything not in it. Isolation is the switch-runlevel operation.

Default target. systemctl get-default shows the target that boots by default (usually graphical.target). systemctl set-default multi-user.target changes it by creating a symlink at /etc/systemd/system/default.target.

Custom targets. You can create your own targets to group services. A .target unit file is trivially simple — it just has [Unit] with a description. Services declare WantedBy=mytarget.target in their [Install] section to attach themselves.

Targets are far more expressive than runlevels because multiple targets can be active simultaneously (e.g., network-online.target and multi-user.target are both active on a running server), and the relationships between them are explicit rather than implicit.

# Check current state
systemctl get-default
# graphical.target

# List all active targets
systemctl list-units --type=target

# Switch to multi-user (drop GUI) without rebooting
sudo systemctl isolate multi-user.target

# Make the server boot without GUI permanently
sudo systemctl set-default multi-user.target

# Create a custom target that groups your app services
cat /etc/systemd/system/myapp.target
# [Unit]
# Description=My Application Stack
# Requires=multi-user.target
# After=multi-user.target
# AllowIsolate=yes

# Services then have: WantedBy=myapp.target in [Install]
# Then:
sudo systemctl enable myapp.target
sudo systemctl start myapp.target

Traditional Unix logging is syslog: text lines written to /var/log/messages (or /var/log/syslog), with a timestamp and hostname prepended. Syslog is ubiquitous and simple. It is also fragile in ways that only become apparent at scale: log rotation can lose messages, multi-line log entries get split, structured data (a JSON exception body) gets stringified and loses its structure, and correlating log lines from different services for a single request requires heroic grep work.

systemd-journald is a structured logging daemon. It collects log entries from:

• Kernel messages (the kmsg ring buffer, the same source as dmesg)
• syslog (via a compatibility socket, so existing syslog-using programs work)
• systemd.journal.socket — programs using the native journal protocol
• stdout/stderr of every service managed by systemd (since systemd redirects the pipes and captures them)
• Audit subsystem events

The key difference: fields, not strings. Each journal entry is a set of key=value pairs. Mandatory fields include _PID, _UID, _GID, _COMM (process name), _EXE, _CMDLINE, SYSLOG_IDENTIFIER, _BOOT_ID, _MACHINE_ID, PRIORITY (syslog priority 0–7), and MESSAGE. Services can add arbitrary fields using sd_journal_send() or by writing specially-formatted lines to stderr.

Storage. Journal files are binary, stored in /var/log/journal/<machine-id>/ (persistent) or /run/log/journal/ (volatile, lost on reboot). They use a hash table + data objects format designed for O(1) seek to a specific boot, unit, or time range.

journalctl is your query interface:

journalctl -u nginx.service          # all logs for nginx
journalctl -u nginx.service -f       # follow (like tail -f)
journalctl -u nginx.service --since "1 hour ago"
journalctl -p err                    # only error priority and above
journalctl -b                        # current boot only
journalctl -b -1                     # previous boot (crashed systems!)
journalctl _PID=1234                 # from specific PID
journalctl -o json-pretty            # structured output
journalctl --disk-usage              # how much space logs use
journalctl --vacuum-size=500M        # trim to 500MB

The syslog forwarding controversy. journald can forward to a syslog socket for compatibility. But because it captures stdout from services, it introduced a subtle behavior change: if a service writes to stdout and journald is slow (e.g., under heavy I/O), the service can block. This caused production incidents before RateLimitIntervalSec and RateLimitBurst were tuned appropriately.

For kernel developers, the journal's _KERNEL_SUBSYSTEM and _KERNEL_DEVICE fields, combined with udev's emission of journal entries, make it possible to trace "which service failed because of which device" in a structured query rather than grep.

// Writing structured log entries from a C program
// (instead of printf to stderr, which just gives you a string)

#include <systemd/sd-journal.h>

void log_request(const char *method, const char *path,
                 int status_code, long latency_us)
{
    sd_journal_send(
        // MESSAGE is what humans see in journalctl
        "MESSAGE=%s %s -> %d (%ldus)",
        method, path, status_code, latency_us,
        // These fields are queryable: journalctl HTTP_STATUS=500
        "HTTP_METHOD=%s", method,
        "HTTP_PATH=%s",   path,
        "HTTP_STATUS=%d", status_code,
        "LATENCY_US=%ld", latency_us,
        // Priority: LOG_INFO, LOG_WARNING, LOG_ERR etc (from syslog.h)
        "PRIORITY=%i",    status_code >= 500 ? 3 : 6,
        NULL  // sentinel
    );
}

// Now in a shell:
// journalctl HTTP_STATUS=500 --since "10 minutes ago" -o json-pretty
// gives you all 500 errors as structured JSON, with all fields.

Control groups (cgroups) are a Linux kernel feature that organizes processes into a hierarchy and lets the kernel apply resource limits (CPU, memory, I/O, network) and accounting to each group. systemd uses cgroups as the foundation of its process tracking — and this integration is what makes systemd fundamentally different from SysV init.

The tracking problem. How does the init system know which processes belong to a service? SysV's answer was PID files: the daemon writes its PID to /var/run/sshd.pid, and init reads it back. This is inherently racy — PID files can be stale, and a service can fork child processes that outlive the main process. If you systemctl stop sshd, do you want to kill just the sshd binary, or every process it spawned (connection handlers, helper utilities)?

With cgroups, systemd creates a cgroup for each service (e.g., system.slice/sshd.service) before starting it. Every process spawned by that service inherits the cgroup. There is no escape — even if a child tries to double-fork to evade tracking, the cgroup membership is inherited across fork(). When you stop a service, systemd sends signals to all processes in the cgroup, not just the main PID.

The cgroup hierarchy. systemd owns the root cgroup and structures it as:

• system.slice — system services
• user.slice — user sessions (one sub-slice per UID)
• machine.slice — VMs and containers

Each service gets a leaf cgroup: system.slice/nginx.service. Each user session gets user.slice/user-1000.slice/session-5.scope.

Resource control in unit files. Because systemd controls the cgroup, you can set resource limits directly in unit files:

[Service]
CPUQuota=50%          # max 50% of one CPU
MemoryMax=512M        # OOM-kill if exceeds 512MB
IOWeight=100          # relative I/O weight (default 100)
TasksMax=64           # max number of processes/threads

These are translated into cgroup v2 settings (cpu.max, memory.max, etc.) by systemd.

systemd-cgls and systemd-cgtop give you a live view of the cgroup tree and per-service resource usage, analogous to ps but organized by service rather than by PID.

cgroups v1 vs v2. systemd supported both but has been pushing the ecosystem toward the unified cgroup v2 hierarchy since 2019. With cgroup v2, all controllers (cpu, memory, io, pid) share a single unified tree, eliminating a nasty class of bugs where a process could be in different positions in different controller hierarchies. Checking cat /sys/fs/cgroup/cgroup.controllers tells you which controllers your kernel supports in the unified hierarchy.

# See the cgroup tree organized by systemd
systemd-cgls
# Output:
# Control group /:
# -.slice
# ├─system.slice
# │ ├─sshd.service
# │ │ └─1234 /usr/sbin/sshd -D
# │ ├─nginx.service
# │ │ ├─5678 nginx: master process
# │ │ └─5679 nginx: worker process
# │ └─...
# └─user.slice
#   └─user-1000.slice
#     └─session-3.scope
#       ├─9876 /bin/bash
#       └─9877 systemd-cgls

# See resource usage per service (like top, but by service)
systemd-cgtop

# Check a service's actual cgroup path
systemctl show nginx.service -p ControlGroup
# ControlGroup=/system.slice/nginx.service

# Read cgroup v2 memory stats directly from kernel
cat /sys/fs/cgroup/system.slice/nginx.service/memory.current
# 45678592  (bytes)

# Set a memory limit on a running service without editing unit file
systemctl set-property nginx.service MemoryMax=256M

No discussion of systemd is complete without engaging honestly with the controversy surrounding it. The debates were not just about technical preferences — they touched on fundamental questions about software architecture, trust, and the Unix philosophy. As a future kernel contributor, you need to understand both sides.

The Unix philosophy objection. The classic formulation is: "Write programs that do one thing and do it well. Write programs that work together." Critics argued that systemd violates this by absorbing logging (journald), networking configuration (systemd-networkd), DNS resolution (systemd-resolved), boot loader management (systemd-boot), container runtime (systemd-nspawn), and more into a single project maintained by a single company (Red Hat/IBM). If systemd has a bug, it can affect all of these simultaneously.

The rebuttal. Poettering's counter-argument was that the Unix philosophy was developed for an era of pipes and text filters, not for the problem of managing a modern OS boot process. The tight coupling between a service manager, a logger, and a cgroup manager is not accidental complexity — it is necessary complexity, because these three things need to know about each other to work correctly. A logger that does not know which service a message came from cannot tag it with _SYSTEMD_UNIT. A cgroup manager that does not know the startup order cannot correctly account for transient services.

The binary log objection. journald's binary format was deeply controversial. If your system crashes and you need to read logs to debug it, you need journalctl to parse them. If journalctl is broken, you are stuck. With plain text syslog, any tool — cat, grep, awk — works. The response was that journald always forwards to syslog for compatibility, and the binary format enables the structured queries and O(1) seeks that make it useful at scale.

The PID 1 scope objection. Linus Torvalds expressed concern about putting too much logic in PID 1, because a crash in PID 1 is a system-wide crash. The response was that systemd is heavily tested, and the alternative (spreading the logic across dozens of shell scripts) has its own failure modes that are harder to reason about.

The dependency objection. Because systemd uses D-Bus, cgroups, Linux-specific kernel interfaces, and optionally many other components, it is difficult to port to non-Linux systems (BSDs, older kernels). OpenBSD and FreeBSD do not use systemd. This is a real limitation for anyone who values portability.

What the debates actually produced. The friction forced systemd to improve: better documentation, more unit file options, better compatibility layers, the systemd-sysv-generator to auto-convert legacy init scripts, and clearer separation between the core (PID 1) and the optional components. The debate also produced serious alternative init systems — OpenRC (used by Alpine Linux and Gentoo), runit (Void Linux), s6 (a signal-handling-focused init used in container base images) — that have genuine technical merit and are worth understanding.

The practical reality. As of 2024, systemd is the init system on Debian, Ubuntu, Fedora, RHEL, openSUSE, Arch, and most embedded Linux distributions. Understanding it is not optional for kernel development on these platforms.

systemd provides several unit types that act as triggers for service units, replacing many traditional tools.

.timer units — replacing cron. A timer unit activates a paired service unit on a schedule. The advantage over cron is systemd integration: the service runs in a proper cgroup with resource limits, its output goes to the journal, and systemctl list-timers gives you a clear view of when everything last ran and when it will run next.

Timers have two scheduling modes:

• Monotonic timers: relative to events — OnBootSec=5min (5 minutes after boot), OnUnitActiveSec=1h (1 hour after last activation). These use the monotonic clock and are not affected by system time changes.
• Realtime timers: like cron — OnCalendar=*-*-* 02:00:00 (every day at 2am), OnCalendar=Mon,Wed,Fri 18:00. These use CLOCK_REALTIME.

The Persistent=yes option in a timer unit means: if the system was off during the scheduled time, run the job as soon as the system comes back up. This is the behavior you want for daily backup jobs.

.path units — replacing inotifywait loops. A path unit watches a file or directory and activates a paired service when it changes. PathExists= triggers when a file appears. PathChanged= triggers on modification. DirectoryNotEmpty= triggers when a directory becomes non-empty. This is perfect for "process files as they arrive in a drop directory" workflows.

.device units — hardware-triggered activation. These are automatically created by systemd/udev for devices that appear in the kernel device tree. You can use them in After= and Requires= to start a service only when specific hardware is present — e.g., only start the USB backup daemon when a particular USB drive is connected.

systemd-run — transient units. You can create a unit on the fly without a unit file:

systemd-run --unit=mytask --on-calendar="*:0/15" myscript.sh

This is extremely useful for testing timer schedules and for one-off jobs that need resource limits applied.

# /etc/systemd/system/backup.timer
[Unit]
Description=Daily backup timer

[Timer]
# Run at 2:30am every day
OnCalendar=*-*-* 02:30:00
# If the system was off at 2:30, run when it comes back up
Persistent=yes
# Add a random delay up to 5 minutes to avoid thundering herd
# on systems that all boot at the same time
RandomizedDelaySec=5min

[Install]
WantedBy=timers.target

---

# /etc/systemd/system/backup.service
[Unit]
Description=Daily backup job

[Service]
Type=oneshot
User=backup
ExecStart=/usr/local/bin/run-backup.sh
# CPU and I/O are nice'd so backup doesn't interfere
Nice=10
IOSchedulingClass=best-effort
IOSchedulingPriority=7

---

# Check timer status
systemctl list-timers backup.timer
# NEXT                         LEFT  LAST                         PASSED  UNIT
# Mon 2024-01-15 02:30:00 UTC  8h    Sun 2024-01-14 02:31:23 UTC  15h     backup.timer

Everything described so far applies to the system instance of systemd, running as PID 1. But systemd also runs a per-user instance (systemd --user) for each logged-in user (or for linger-enabled users even when not logged in). This is a complete parallel service management stack, but running unprivileged as the user.

Why it matters. Before systemd --user, there was no standardized way to manage per-user background services. You might put things in .bashrc (only runs in interactive shells), .xinitrc (only runs in X sessions), or use a tmux session that you pray survives. systemd --user gives you proper service management — restart on failure, journal logging, socket activation — without root.

Unit file locations. User units live in ~/.config/systemd/user/ (user-managed) or /usr/lib/systemd/user/ (package-installed). Commands use the --user flag:

systemctl --user status syncthing
systemctl --user enable --now syncthing.service
journalctl --user -u syncthing.service -f

XDG_RUNTIME_DIR. The user systemd instance uses $XDG_RUNTIME_DIR/systemd/ (typically /run/user/1000/systemd/) for its runtime socket and state. This directory is created by pam_systemd on login and destroyed on last logout.

loginctl enable-linger <user>. By default, the user systemd instance starts on first login and stops on last logout. enable-linger keeps it running even with no active sessions — essential for background sync daemons, bots, and personal servers that should run continuously.

Environment variables. One subtlety: the user systemd instance does not inherit your login shell's PATH or other environment variables. You need to set them in ~/.config/environment.d/ (files ending in .conf with KEY=value lines) or use systemctl --user import-environment.

D-Bus and desktop integration. On a desktop system, the user's D-Bus session bus is managed by dbus-broker or dbus-daemon, started as part of the user systemd session. Applications that need D-Bus (most GUI apps) depend on this.

# Create a user service for a personal sync daemon
# No root required!

mkdir -p ~/.config/systemd/user

cat > ~/.config/systemd/user/mybot.service << 'EOF'
[Unit]
Description=My personal IRC bot
After=network-online.target

[Service]
Type=simple
ExecStart=%h/bots/ircbot.py --config %h/.config/ircbot.toml
# %h expands to $HOME in user units
Restart=on-failure
RestartSec=10s
# Logs go to journalctl --user -u mybot

[Install]
WantedBy=default.target
EOF

# Enable and start
systemctl --user daemon-reload
systemctl --user enable --now mybot.service

# Keep it running even after logout
loginctl enable-linger $USER

# Check status
systemctl --user status mybot.service
journalctl --user -u mybot.service --since today