Containers from the Kernel's View: Namespaces & cgroups

A container is not a kernel object. There is no struct container anywhere in the Linux source tree. Instead, a container is a process (or process tree) that has been placed into a set of namespaces to restrict its view of the system, and into a set of cgroup hierarchies to restrict its resource consumption, with its root filesystem swapped out via chroot or pivot_root.

Contrast this with a virtual machine: a VM emulates entire hardware — a CPU, memory bus, NIC, disk controller — and runs a complete, separate OS kernel on top of that emulated hardware. The VM's kernel has no knowledge of the host kernel; they share nothing except physical hardware (through a hypervisor). This is strong isolation but has real cost: booting a VM takes seconds (kernel init, device probe, userspace init), and every VM carries its own full kernel in memory.

A container shares the host kernel. There is exactly one kernel running. The container's processes are real Linux processes visible to the host. The isolation comes from the kernel's own access-control machinery: namespaces make the process think it's alone, cgroups make sure it can't starve others. This is why containers start in milliseconds (no kernel boot — just clone() + execve()) and why a kernel exploit can potentially escape a container (there's only one kernel to exploit).

The key insight: containers are a UX convention built on top of kernel features. The container runtime (Docker, containerd, runc) is essentially a sophisticated wrapper around clone(2), unshare(2), pivot_root(2), and the cgroup filesystem API.

A namespace wraps a global system resource and presents a process with an isolated instance of that resource. Changes within a namespace are visible only to processes in that namespace. As of Linux 5.6+, there are eight namespace types:

Every process has exactly one of each namespace type. When Linux boots, init (PID 1) is placed into the initial namespace for every type — the 'real' global view. When you create a new namespace, the child process gets a fresh, private instance.

Namespaces are reference-counted kernel objects. They persist as long as at least one process is a member, or as long as a bind mount holds a reference to /proc/<pid>/ns/<type>. This is how ip netns exec works: it opens the namespace file, keeps a reference, and later joins the namespace via setns(2) — even after all the original processes in that namespace have exited.

The three syscalls that manipulate namespaces are:

• clone(2): creates a new child process in new namespace(s) — pass CLONE_NEW* flags
• unshare(2): moves the calling process into new namespace(s) it creates
• setns(2): moves the calling process into an existing namespace (specified by file descriptor to /proc/pid/ns/X)

You can inspect any process's namespaces: ls -la /proc/$$/ns/ — each symlink shows the namespace type and its inode number. Two processes with the same inode number for a namespace type are in the same namespace.

# Inspect current process namespaces
ls -la /proc/$$/ns/
# lrwxrwxrwx ... pid -> pid:[4026531836]
# lrwxrwxrwx ... net -> net:[4026531992]
# ...

# Check if two processes share a namespace: compare inode numbers
stat -L /proc/1/ns/pid /proc/$$/ns/pid
# If st_ino matches, they're in the same PID namespace

The PID namespace is perhaps the most mind-bending. When you create a new PID namespace, the first process in it is assigned PID 1 — just like init on a real system. But from the host's perspective, that process has a completely different PID (say, 3847).

This dual identity is real and important: a process has a different PID in each namespace it's visible from. A process in a nested PID namespace is visible (with its host PID) in all ancestor namespaces, but the outer namespaces can't be seen from inside. Hierarchy is strictly one-directional.

The init role matters: if PID 1 in a PID namespace exits, the kernel sends SIGKILL to all other processes in that namespace. This is why container runtimes (like tini or Docker's built-in init) run as PID 1 — they must handle signals properly and reap zombie children, or the whole container misbehaves.

Orphaned children (whose parent exits) are normally reparented to PID 1 of the current PID namespace, not the global init. This means zombie reaping works correctly inside containers.

Note a subtlety: unshare --fork --pid --mount-proc works, but simply calling unshare(CLONE_NEWPID) from a shell doesn't move the current process into the new namespace — it moves the next child it spawns. PID namespaces take effect at clone()/fork() time, not unshare() time, for the calling process. The --fork flag handles this by making unshare(1) fork a child that becomes PID 1 in the new namespace.

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

// Stack for the child process
static char child_stack[1024*1024];

static int child_fn(void *arg) {
    printf("Inside new PID namespace\n");
    printf("My PID (inside ns): %d\n", getpid());  // will print 1
    return 0;
}

int main() {
    printf("Host PID: %d\n", getpid());
    pid_t child = clone(child_fn,
                        child_stack + sizeof(child_stack),
                        CLONE_NEWPID | SIGCHLD,
                        NULL);
    if (child == -1) { perror("clone"); exit(1); }
    printf("Child's host PID: %d\n", child);
    waitpid(child, NULL, 0);
    return 0;
}
// Compile: gcc -o pidns pidns.c
// Run as root (or with CAP_SYS_ADMIN): ./pidns

A network namespace gives a process its own completely independent network stack: its own set of network interfaces, routing tables, firewall rules (iptables/nftables), and port number space. Two processes in different network namespaces can both listen on port 80 without conflict — they each have their own port space.

When you create a new network namespace, it starts with only one interface: lo (loopback), which is down. This is why fresh Docker containers have their own eth0 but the host doesn't see it — it's a virtual interface that lives only in that network namespace.

How do containers communicate with the outside world? Through veth pairs — virtual Ethernet cable pairs. A veth pair is two virtual network interfaces connected back-to-back: anything written to one end immediately appears on the other. The container runtime places one end of the veth pair inside the container's network namespace (renamed eth0) and the other end in the host network namespace (e.g., veth3a7b). The host end is then connected to a bridge (like docker0), and NAT rules in iptables masquerade the container's traffic.

This architecture means network namespaces are fundamental to Kubernetes pod networking too: every pod gets a network namespace, and the CNI plugin (Flannel, Calico, Cilium) is responsible for connecting it to the cluster network.

The ip netns command from iproute2 is a thin wrapper around these kernel features. Internally it creates a bind mount of /proc/<pid>/ns/net under /run/netns/<name> to keep the namespace alive without requiring a process to stay in it.

# Create a named network namespace
sudo ip netns add mycontainer

# List it
ip netns list

# Run a command inside it (only loopback exists)
sudo ip netns exec mycontainer ip link

# Create a veth pair
sudo ip link add veth-host type veth peer name veth-cont

# Move one end into the namespace
sudo ip link set veth-cont netns mycontainer

# Bring up and assign IPs
sudo ip addr add 10.200.0.1/24 dev veth-host
sudo ip link set veth-host up
sudo ip netns exec mycontainer ip addr add 10.200.0.2/24 dev veth-cont
sudo ip netns exec mycontainer ip link set veth-cont up
sudo ip netns exec mycontainer ip link set lo up

# Test: ping from host into namespace
ping -c1 10.200.0.2

# Cleanup
sudo ip netns del mycontainer

The mount namespace isolates the filesystem mount tree. Every process sees a tree of mounts rooted at /. When you create a new mount namespace, the child starts with a copy of the parent's mount tree (actually a tagged shared/slave/private mount propagation — more on this shortly), and any new mounts or unmounts in the child are invisible to the parent.

Mount propagation is a nuance that trips up even experienced engineers: mounts can be shared (changes propagate both ways), slave (changes propagate from master to slave, not back), private (no propagation), or unbindable. By default, systemd marks most mounts as shared. A new container namespace inherits this and could accidentally see the host's mounts propagating in. Container runtimes immediately call mount(MS_REC | MS_PRIVATE) to make the new mount namespace private.

The other crucial piece is pivot_root. Simply doing chroot to a new filesystem root is inadequate for isolation — a process with CAP_SYS_CHROOT can escape a chroot jail. pivot_root(new_root, put_old) is different: it actually replaces the root mount of the current mount namespace, atomically moving the old root to put_old. After pivot_root, you unmount the old root entirely. Now there's literally no path back to the host filesystem — the mount namespace no longer contains it.

The sequence a real container runtime follows:

1. clone(CLONE_NEWNS | ...) — new mount namespace
2. mount(MS_REC | MS_PRIVATE) — stop propagation
3. Bind-mount the container's rootfs to a new location
4. pivot_root(new_root, old_root) — swap the root
5. umount(old_root, MNT_DETACH) — remove the host filesystem from view
6. Mount /proc, /sys, /dev as fresh instances

This is exactly what runc (the OCI runtime used by Docker) does, and you can see it in the runc source code.

# Demonstrate mount namespace isolation
# (Run as root)

# Create a new mount namespace
unshare --mount bash

# Inside new namespace: mount tmpfs somewhere
mount -t tmpfs tmpfs /mnt
echo "hello from container" > /mnt/greeting

# Open another terminal — on the host:
ls /mnt   # empty! the tmpfs is invisible outside

# Back inside: the mount is still there
ls /mnt   # greeting
exit

UTS namespace (CLONE_NEWUTS): UTS stands for Unix Time-sharing System — a legacy name from BSD. It isolates two system identifiers: hostname and NIS domain name. When a container runs hostname myapp-container, it changes only its UTS namespace's hostname, not the host's. Simple but important for software that uses hostname for service discovery.

IPC namespace (CLONE_NEWIPC): Isolates System V IPC objects (semaphores, message queues, shared memory segments) and POSIX message queues. Without this, processes in one container could use ipcs to see and interfere with shared memory segments of another container on the same host. This is particularly important for databases (PostgreSQL uses SysV shared memory) — you don't want containers accidentally sharing memory.

User namespace (CLONE_NEWUSER): This is the most powerful and most nuanced namespace. It maps user IDs and group IDs between the namespace and the host. A process can be UID 0 (root) inside its user namespace but UID 65534 (nobody) outside. This is the basis for rootless containers: a normal unprivileged user can create user namespaces and, within them, hold full root capabilities — but those capabilities only apply inside the namespace and on resources owned by that user namespace.

User namespaces have important security properties:

• They're the only namespace type that can be created without root privileges (since Linux 3.8, though some distros disable this via kernel.unprivileged_userns_clone sysctl).
• Creating other namespace types (net, mount, etc.) from within a user namespace requires no extra privileges — this is how rootless containers work entirely.
• The UID mapping is written to /proc/<pid>/uid_map after clone(), before the child drops privileges.

The UID mapping file format is: <inside-uid-start> <outside-uid-start> <count>. For example, 0 1000 1 maps UID 0 inside to UID 1000 outside — any file the container creates as root is owned by UID 1000 on the host.

#define _GNU_SOURCE
#include <sched.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <fcntl.h>
#include <string.h>
#include <sys/wait.h>

static char stack[1024*1024];

static int child(void *arg) {
    printf("Inside user ns: uid=%d gid=%d\n", getuid(), getgid());
    // We appear as UID 0 inside the user namespace
    return 0;
}

int main() {
    printf("Outside: uid=%d\n", getuid());
    pid_t pid = clone(child, stack + sizeof(stack),
                      CLONE_NEWUSER | SIGCHLD, NULL);
    if (pid < 0) { perror("clone"); exit(1); }

    // Write UID map: inside UID 0 -> outside UID (our real UID)
    char path[64], map[64];
    snprintf(path, sizeof(path), "/proc/%d/uid_map", pid);
    snprintf(map, sizeof(map), "0 %d 1", getuid());
    int fd = open(path, O_WRONLY);
    write(fd, map, strlen(map)); close(fd);

    // Must write "deny" to setgroups before gid_map
    snprintf(path, sizeof(path), "/proc/%d/setgroups", pid);
    fd = open(path, O_WRONLY);
    write(fd, "deny", 4); close(fd);

    snprintf(path, sizeof(path), "/proc/%d/gid_map", pid);
    snprintf(map, sizeof(map), "0 %d 1", getgid());
    fd = open(path, O_WRONLY);
    write(fd, map, strlen(map)); close(fd);

    waitpid(pid, NULL, 0);
    return 0;
}

While namespaces control what a process sees, control groups (cgroups) control what a process can use. cgroups organize processes into a hierarchy and apply resource limits, accounting, and priority to each group.

Linux has had two cgroup implementations: cgroups v1 (introduced in 2.6.24) and cgroups v2 (mainlined in 4.5, now the standard). The key architectural difference: v1 allowed each resource controller (cpu, memory, blkio, etc.) to have its own separate hierarchy tree, which led to inconsistencies and complexity. v2 has a unified hierarchy: all controllers share a single tree, and a process can belong to exactly one cgroup at a level.

The cgroups v2 interface is a filesystem mounted at /sys/fs/cgroup (on modern systemd systems). You interact with it by reading and writing files:

• cgroup.procs: list of PIDs in this cgroup (write a PID to move it)
• cgroup.controllers: available controllers
• cgroup.subtree_control: which controllers are enabled for children
• memory.max: hard memory limit (bytes, or max for unlimited)
• memory.high: soft memory limit — processes are throttled, not killed
• cpu.max: two numbers: quota period in microseconds — e.g., 100000 1000000 means 100ms of CPU per 1000ms = 10% CPU
• io.max: per-device I/O rate limits
• pids.max: maximum number of PIDs in the group

The memory controller in cgroupv2 is particularly sophisticated. It tracks not just RSS (resident set size) but also page cache, kernel memory, and swap. When a cgroup hits memory.max, the OOM killer targets processes within that cgroup, not randomly across the system. This is fundamental to container isolation: a runaway process in one container can't cause OOM kills in another.

The cpu controller uses the Completely Fair Scheduler (CFS) bandwidth control mechanism. cpu.max 100000 1000000 creates a CFS bandwidth period of 1 second, with 100ms quota — the kernel enforces that all processes in the cgroup consume at most 100ms of CPU per second. This is hard throttling; for soft priority weighting, use cpu.weight (1–10000, default 100).

cgroups are also the mechanism behind systemd slices — when you run a systemd service, it lives in a cgroup. systemctl status shows you the cgroup tree.

# Modern systems: cgroupv2 at /sys/fs/cgroup
# Verify:
stat -f /sys/fs/cgroup | grep Type  # tmpfs = v1, cgroup2 = v2
grep cgroup /proc/mounts

# Create a new cgroup
sudo mkdir /sys/fs/cgroup/mycontainer

# Enable memory and CPU controllers for this cgroup
# (must be enabled in parent first)
echo "+memory +cpu +pids" | sudo tee /sys/fs/cgroup/cgroup.subtree_control

# Set a 100MB memory limit
echo $((100*1024*1024)) | sudo tee /sys/fs/cgroup/mycontainer/memory.max

# Set a CPU limit: 50% of one core (50ms per 100ms period)
echo "50000 100000" | sudo tee /sys/fs/cgroup/mycontainer/cpu.max

# Move current shell into the cgroup
echo $$ | sudo tee /sys/fs/cgroup/mycontainer/cgroup.procs

# Verify
cat /proc/$$/cgroup   # should show mycontainer

# Memory usage stats
cat /sys/fs/cgroup/mycontainer/memory.current

# Cleanup (must be empty first)
# Move shell back: echo $$ > /sys/fs/cgroup/cgroup.procs
# sudo rmdir /sys/fs/cgroup/mycontainer

The best way to understand containers is to build one without using Docker. We'll use only shell tools and Linux syscalls to create full process isolation. This is essentially what runc does (minus the OCI spec compliance and production hardening).

The ingredients we need:

1. A root filesystem (we'll extract a Docker image or use debootstrap)
2. A new set of namespaces (pid, net, uts, ipc, mount, user)
3. A pivot_root to swap the filesystem
4. A cgroup for resource limits
5. Mount /proc, /dev, /sys inside the container

Using unshare(1) — which calls unshare(2) under the hood — we can do most of this from the shell. For a rootless container (no sudo needed), user namespaces are the key.

The unshare --user --map-root-user --pid --fork --mount --uts --ipc --net invocation creates all namespaces in one command. The --map-root-user flag automatically writes the UID map making us root inside the namespace.

After entering the namespaces, we use chroot (or pivot_root for stronger isolation) to switch to our container rootfs, remount /proc so that process listings are accurate (without this, you'd see the host's processes), and then exec our init.

This 'container' lacks:

• seccomp filters (syscall allowlisting)
• Linux Security Modules (AppArmor/SELinux policies)
• Capability bounding sets
• Read-only root filesystem ...all of which real container runtimes add. But it demonstrates that the core mechanism is entirely these kernel features.

#!/bin/bash
# Build a minimal container by hand (rootless, no Docker)
# Requirements: util-linux >= 2.28, a rootfs directory
#
# 1. Get a rootfs (e.g. Alpine Linux)
#    mkdir rootfs && cd rootfs
#    curl -o alpine.tar.gz https://dl-cdn.alpinelinux.org/alpine/v3.19/releases/x86_64/alpine-minirootfs-3.19.0-x86_64.tar.gz
#    tar xf alpine.tar.gz && cd ..
#
# 2. Run this script

ROOTFS="$(pwd)/rootfs"
CGROUP="/sys/fs/cgroup/handbuilt"

# Create cgroup for resource limits
if [ -w /sys/fs/cgroup ]; then
    mkdir -p "$CGROUP"
    echo "50000 100000" > "$CGROUP/cpu.max"     # 50% CPU
    echo $((64*1024*1024)) > "$CGROUP/memory.max" # 64MB RAM
fi

exec unshare \
    --user \
    --map-root-user \
    --pid \
    --fork \
    --mount \
    --uts \
    --ipc \
    --net \
    /bin/bash -c "
        # We are now root in our own namespaces
        # Set a hostname
        hostname handbuilt-container

        # Bind-mount rootfs to itself (needed for pivot_root)
        mount --bind $ROOTFS $ROOTFS
        cd $ROOTFS

        # Create a place for the old root
        mkdir -p old_root

        # Swap the root filesystem
        pivot_root . old_root

        # Mount essential pseudo-filesystems
        mount -t proc proc /proc
        mount -t sysfs sys /sys
        mount -t tmpfs tmpfs /tmp
        mount -t devtmpfs dev /dev 2>/dev/null || \
            mount -t tmpfs devtmpfs /dev

        # Unmount the old host filesystem
        umount -l /old_root
        rmdir /old_root

        # Bring up loopback
        ip link set lo up 2>/dev/null || true

        echo '=== Inside container ==='
        echo \"Hostname: \$(hostname)\"
        echo \"PID: \$(\$\$)\"
        echo \"UID: \$(id)\"
        ps aux

        # Drop into a shell
        exec /bin/sh
    "

One of the most important design decisions in cgroups v2 is the no internal processes rule: a cgroup that has child cgroups cannot directly contain processes. Processes must live at the leaves of the hierarchy. This was introduced to eliminate races and inconsistencies that plagued v1, where a process could be in an intermediate node and have contradictory controller settings from multiple paths.

Delegation is the mechanism that makes cgroups safe for non-root users to use. A cgroup subtree can be delegated to a non-root user by chowning the cgroup directory and certain controller files to that user. Once delegated, the user can create sub-cgroups under their delegated root and move processes they own into them — all without root. This is the foundation of systemd user sessions and rootless container engines like Podman.

The freezer (cgroupv2 cgroup.freeze) is a simple but powerful feature: write 1 to freeze all processes in a cgroup, 0 to thaw them. Docker uses this for docker pause/docker unpause. The kernel's OOM killer respects cgroup boundaries: memory.oom.group = 1 makes the OOM killer treat the entire cgroup as a unit, killing all its processes when any one hits the memory limit — important for multi-process applications.

cgroup namespaces (CLONE_NEWCGROUP) are the final piece: they virtualize the cgroup filesystem view. Inside a container, the container's root cgroup appears to be the root of the cgroup hierarchy — processes can't walk up to see the host's cgroup tree or discover sibling containers' cgroups. Without this, a process inside a container could read /proc/self/cgroup and discover the full cgroup path on the host, leaking information about the host's organization.

Putting it all together: a process in a container lives in a cgroup namespace, sees its container's cgroup as /, is in a user namespace (limiting its privileges), and is in a mount namespace where /sys/fs/cgroup is a view of only its delegated subtree.

# Inspect cgroup hierarchy (cgroupv2)
tree /sys/fs/cgroup -L 2 2>/dev/null || ls -la /sys/fs/cgroup

# See which cgroup the current process is in
cat /proc/$$/cgroup
# Output: 0::/user.slice/user-1000.slice/session-1.scope

# Systemd's cgroup tree (very instructive)
systemd-cgls

# Or with journalctl to see resource usage
systemd-cgtop

# Check cgroupv2 controllers available
cat /sys/fs/cgroup/cgroup.controllers
# cpu io memory pids

# Freeze a cgroup (as root)
mkdir /sys/fs/cgroup/test
echo $$ > /sys/fs/cgroup/test/cgroup.procs
echo 1 > /sys/fs/cgroup/test/cgroup.freeze  # this shell freezes!
echo 0 > /sys/fs/cgroup/test/cgroup.freeze  # thaw

Understanding the security boundaries of containers is as important as understanding the isolation mechanisms. There are several categories of things that namespaces and cgroups do not isolate:

The kernel itself: All containers share the host kernel. A kernel vulnerability (use-after-free, privilege escalation) can compromise all containers simultaneously. This is the fundamental difference from VMs. High-security environments (multi-tenant cloud) add an additional VM layer (e.g., AWS Firecracker, Google gVisor) on top of container isolation.

Kernel keyring: The KEYRING namespace doesn't exist yet (as of Linux 6.x). Session keyrings can be accessed across containers in the same user session.

/proc and /sys without careful mounting: A container with access to the host's /proc can read process information, hardware state, and kernel parameters. Container runtimes carefully remount these with MS_RDONLY or bind-mount minimal views.

Unrestricted syscalls: A process in a container can call any syscall the kernel supports. clone(CLONE_NEWUSER) inside a container can create further nested namespaces. Some syscalls like perf_event_open, ptrace, keyctl can be used for attacks. This is why container runtimes apply seccomp-bpf filters — a BPF program that runs on every syscall and returns allow/deny. Docker's default seccomp profile blocks ~44 dangerous syscalls.

Linux Security Modules: Namespaces interact with AppArmor and SELinux, but don't replace them. A container with a permissive or absent LSM policy can do far more than one with a tight AppArmor profile.

Time namespace caveat: Even with time namespaces, the system clock (wall clock via clock_gettime(CLOCK_REALTIME)) is shared — only boot-time and monotonic clocks are virtualized. Container processes can read the real wall time.

The take-away: container security is defense in depth — namespaces + cgroups + seccomp + LSM + drop capabilities + read-only root + rootless. No single mechanism is sufficient.

# Check which syscalls Docker blocks by default
# (requires docker and jq)
docker run --rm alpine cat /proc/1/status | grep Seccomp
# Seccomp: 2  (2 = filter mode, 0 = disabled, 1 = strict)

# Show capabilities of a container process
docker run --rm alpine cat /proc/1/status | grep Cap
# CapPrm, CapEff, CapBnd, CapAmb lines

# Decode capability bitmask
capsh --decode=00000000a80425fb
# shows human-readable list of allowed capabilities

# See what a container's cgroup looks like from inside vs outside
docker run --rm alpine cat /proc/1/cgroup
# Inside: 0::/   (appears to be root)
# Outside: 0::/system.slice/docker-<id>.scope