Inter-Process Communication (IPC)

The kernel's most fundamental promise is isolation: each process gets its own virtual address space. When process A writes to address 0x400000, that write is invisible to process B, even if B also has a mapping at 0x400000. The MMU enforces this via page tables — A and B's page tables point to different physical frames.

This isolation is wonderful for safety and security. A buggy process cannot corrupt another's state. But it creates a problem: how do two programs that need to cooperate actually exchange data or coordinate actions?

The answer is IPC: the kernel provides controlled tunnels through the isolation barrier. These tunnels vary in:

• Bandwidth: how much data per second can flow
• Latency: how long does a message take to arrive
• Synchronization: does the sender block until the receiver is ready?
• Addressability: can you target a specific process, or is the channel anonymous?
• Persistence: does the channel outlive the processes using it?

Think of isolation like separate apartments in a building. IPC is the building's infrastructure: you can pass notes under the door (signals), use pneumatic tubes (pipes), leave parcels in a shared mailroom (message queues), or knock down the wall and share a room (shared memory). Each approach has different cost and capability.

A pipe is the oldest and simplest IPC mechanism in Unix. Conceptually, it is a kernel-managed FIFO byte stream: one end is write-only, the other is read-only. Data written to the write end is buffered in the kernel until it is read from the read end.

pipe(2) returns two file descriptors: fd[0] (read end) and fd[1] (write end). Inside the kernel, a pipe is backed by a circular buffer (historically 4KB, now 64KB on modern Linux — 16 pages). When you write to the write end, the kernel copies your data into this buffer. When you read from the read end, the kernel copies data out.

The key insight about blocking: if you try to read from an empty pipe, your process blocks — the kernel puts you to sleep on a wait queue. When a writer adds data, the kernel wakes you up. Similarly, if the buffer is full, the writer blocks. This automatic flow control is elegant: you never poll, never spin-wait.

Named pipes (FIFOs): pipe(2) creates an anonymous pipe visible only to processes that share the file descriptors (typically parent/child after fork). mkfifo(3) creates a named pipe — a filesystem entry that any process can open by path. Named pipes enable communication between unrelated processes.

The EOF signal: when all write ends of a pipe are closed and no data remains, reads return 0. This is how your shell pipeline terminates: ls | grep foo — when ls exits, its write end closes, grep reads 0 bytes and exits too.

Copying happens: every byte that travels through a pipe is copied twice — once from userspace into the kernel buffer, once from the kernel buffer into the reader's userspace. For high-bandwidth scenarios, this double-copy is expensive. Linux 2.6.17 added splice(2) and tee(2) to avoid some copies by moving pages between kernel buffers directly, and Linux 5.7 added pipe2 with O_DIRECT flag for packet-mode pipes.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>

int main(void) {
    int fd[2];
    if (pipe(fd) == -1) { perror("pipe"); return 1; }

    pid_t pid = fork();
    if (pid == 0) {
        /* Child: reader */
        close(fd[1]);          /* close unused write end */
        char buf[64];
        ssize_t n = read(fd[0], buf, sizeof(buf) - 1);
        buf[n] = '\0';
        printf("child got: %s\n", buf);
        close(fd[0]);
        return 0;
    }
    /* Parent: writer */
    close(fd[0]);              /* close unused read end */
    const char *msg = "hello from parent";
    write(fd[1], msg, strlen(msg));
    close(fd[1]);              /* closing signals EOF to child */
    wait(NULL);
    return 0;
}

A signal is the lightest-weight IPC primitive: it delivers a single integer (the signal number) to a process or thread. There is no data payload beyond that number. Signals are asynchronous — they can arrive at any moment, interrupting whatever the target process is doing.

The delivery model: when the kernel delivers a signal to a process, it sets a bit in that process's task_struct (the kernel's per-process data structure). The next time the process returns from a syscall or is about to be re-scheduled, the kernel checks these pending bits and dispatches the signal handler before returning to userspace.

This asynchronous delivery is powerful but also the source of endless bugs. A signal can arrive between any two machine instructions in your program. If your signal handler accesses a data structure that was partially modified when the signal hit, you have a race condition. This is why signal handlers must only call async-signal-safe functions — a small, carefully chosen set that can execute correctly even when called from an arbitrary context.

Sending signals: kill(2) sends a signal to a process by PID (despite the name, most signals are not fatal). raise(3) sends to yourself. sigqueue(3) extends this with a small integer payload (POSIX real-time signals). The shell uses signals constantly: Ctrl+C sends SIGINT, Ctrl+Z sends SIGTSTP, kill -9 <pid> sends SIGKILL.

SIGKILL and SIGSTOP cannot be caught: these two are handled entirely by the kernel — no userspace handler is consulted. This is the kernel's escape hatch to always be able to terminate or stop a runaway process.

signalfd(2): because asynchronous signal delivery is hard to program correctly, Linux 2.6.22 added signalfd — you can block signals from async delivery and instead read them like file descriptor events, which is far easier to reason about and works beautifully with epoll.

Real-world use: signals are used for process lifecycle (SIGCHLD tells a parent its child changed state), terminal interaction (SIGWINCH when terminal is resized), graceful shutdown (SIGTERM to daemons), and profiling (SIGALRM/SIGPROF from setitimer).

#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <stdatomic.h>

static volatile sig_atomic_t g_got_usr1 = 0;

static void handle_usr1(int sig) {
    /* Only async-signal-safe operations here! */
    g_got_usr1 = 1;
}

int main(void) {
    struct sigaction sa = {
        .sa_handler = handle_usr1,
        .sa_flags   = SA_RESTART,
    };
    sigemptyset(&sa.sa_mask);
    sigaction(SIGUSR1, &sa, NULL);

    printf("my pid is %d — send me SIGUSR1\n", getpid());
    while (!g_got_usr1)
        pause();   /* sleep until any signal */

    puts("received SIGUSR1!");
    return 0;
}

Where a pipe is a raw byte stream, a message queue preserves message boundaries and adds priority. You send a message as an atomic unit; the receiver gets exactly that unit back — no need to frame/deframe your protocol.

Linux implements both System V message queues (msgget/msgsnd/msgrcv) and POSIX message queues (mq_open/mq_send/mq_receive). POSIX mqueues are the modern API — they appear as files under /dev/mqueue, integrate with select/poll/epoll, and support async notification via mq_notify.

Priority: each message has an integer priority. mq_receive always returns the highest-priority pending message first, then FIFO within a priority level. This is useful for systems where some messages (e.g., shutdown commands) must be processed before others (e.g., telemetry).

Blocking semantics: like pipes, sends block if the queue is full; receives block if the queue is empty. O_NONBLOCK makes operations fail immediately instead with EAGAIN.

The copy story: like pipes, message queues involve a kernel copy. The sender copies into kernel memory; the receiver copies out. There is no zero-copy path.

Limits: /proc/sys/fs/mqueue/ exposes tunable limits on queue depth and message size. On a 64-bit system defaults allow 10 messages of up to 8192 bytes each per queue.

When to use: message queues shine when you need discrete messages with priorities and want the persistence that comes with a named queue (the queue exists until explicitly unlinked, even if no process has it open — processes can post to it, terminate, and the messages wait for later consumers).

#include <stdio.h>
#include <mqueue.h>
#include <string.h>

#define QNAME "/demo_queue"

int main(void) {
    struct mq_attr attr = {
        .mq_flags   = 0,
        .mq_maxmsg  = 8,
        .mq_msgsize = 64,
    };
    mqd_t mq = mq_open(QNAME, O_CREAT|O_RDWR, 0600, &attr);
    if (mq == (mqd_t)-1) { perror("mq_open"); return 1; }

    /* Send a message with priority 5 */
    const char *msg = "hello queue";
    mq_send(mq, msg, strlen(msg)+1, /*priority=*/5);

    /* Receive it back */
    char buf[64];
    unsigned int prio;
    mq_receive(mq, buf, sizeof(buf), &prio);
    printf("got (prio=%u): %s\n", prio, buf);

    mq_close(mq);
    mq_unlink(QNAME);   /* remove the queue */
    return 0;
}

Shared memory is the highest-bandwidth IPC mechanism because it involves no kernel copies after setup. Two (or more) processes map the same physical pages into their respective virtual address spaces. A write by process A is immediately visible to process B — no write() syscall, no kernel buffer, no copy.

How it works at the MMU level: the kernel sets up both processes' page tables to point to the same physical frames. From the CPU's perspective, each process is just reading/writing its own (virtual) memory. The MMU translates both virtual addresses to the same physical address.

POSIX shared memory: shm_open creates a named shared memory object (appears in /dev/shm), then ftruncate sets its size, then mmap maps it into the address space. Any process that knows the name and has permission can map it.

Anonymous shared memory: after fork(), parent and child can share memory by mapping an anonymous region with mmap(MAP_SHARED|MAP_ANONYMOUS) before forking. The child inherits the mapping.

The synchronization problem: shared memory gives you speed but takes away safety. If both A and B write to the same location simultaneously, you have a data race. Shared memory is almost always used with a synchronization primitive:

• pthread_mutex_t with PTHREAD_PROCESS_SHARED attribute
• sem_open POSIX semaphores
• Atomic operations via stdatomic.h

Without synchronization, shared memory programs produce subtle, nondeterministic bugs that are agonizing to debug. This is not a theoretical concern — it is the most common mistake beginners make.

memfd_create(2): since Linux 3.17, you can create anonymous memory files without touching the filesystem at all. These can be passed between processes via Unix domain socket SCM_RIGHTS (file descriptor passing). This is how modern sandboxed systems (Wayland, DBus with fd passing) share large buffers without a filesystem namespace.

#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <sys/wait.h>
#include <unistd.h>
#include <semaphore.h>

typedef struct {
    sem_t sem;
    int   counter;
} shared_t;

int main(void) {
    /* Create anonymous shared mapping */
    shared_t *sh = mmap(NULL, sizeof(shared_t),
                        PROT_READ|PROT_WRITE,
                        MAP_SHARED|MAP_ANONYMOUS, -1, 0);
    sem_init(&sh->sem, /*pshared=*/1, /*value=*/1);
    sh->counter = 0;

    for (int i = 0; i < 4; i++) {
        if (fork() == 0) {
            for (int j = 0; j < 1000; j++) {
                sem_wait(&sh->sem);
                sh->counter++;      /* protected increment */
                sem_post(&sh->sem);
            }
            exit(0);
        }
    }
    for (int i = 0; i < 4; i++) wait(NULL);
    printf("counter = %d (expected 4000)\n", sh->counter);
    sem_destroy(&sh->sem);
    munmap(sh, sizeof(shared_t));
    return 0;
}

Sockets are the most general IPC mechanism. Originally designed for network communication, they also work between processes on the same machine via Unix domain sockets (address family AF_UNIX). A socket can be a stream (like a pipe, but bidirectional and connection-oriented) or a datagram (message-preserving, like UDP).

Unix domain sockets (AF_UNIX) are the backbone of modern Linux IPC:

• DBus uses them for system bus communication
• systemd's socket activation uses them
• The X Window System and Wayland use them
• Docker's daemon listens on /var/run/docker.sock
• PostgreSQL clients connect via /var/run/postgresql/.s.PGSQL.5432

Why use a socket instead of a pipe? Sockets are bidirectional, work between unrelated processes, support connection semantics (you can tell who connected), and support credential passing (SCM_CREDENTIALS) and file descriptor passing (SCM_RIGHTS) as ancillary data.

File descriptor passing is one of the most powerful features of Unix: process A can open a file, then send the open file descriptor to process B through a socket. Process B receives a new file descriptor pointing to the same underlying open file description in the kernel. This is how privilege separation works in modern security architectures: a privileged process opens a sensitive file and passes the fd to an unprivileged process, which never needs to know the path or have permission to open it directly.

Stream vs. datagram: SOCK_STREAM gives you a reliable, ordered byte stream (you still need to frame your messages). SOCK_SEQPACKET gives you a reliable, ordered, message-preserving channel — the best of both worlds for local IPC but only available with AF_UNIX.

Bandwidth: Unix domain sockets on modern Linux achieve 10-50 GB/s for local IPC, making them competitive with pipes. Combined with fd passing, they're often the right choice for high-performance local IPC.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <sys/un.h>

#define SOCK_PATH "/tmp/demo.sock"

int main(void) {
    /* Server side in parent, client in child */
    unlink(SOCK_PATH);
    int srv = socket(AF_UNIX, SOCK_STREAM, 0);
    struct sockaddr_un addr = { .sun_family = AF_UNIX };
    strncpy(addr.sun_path, SOCK_PATH, sizeof(addr.sun_path)-1);
    bind(srv, (struct sockaddr*)&addr, sizeof(addr));
    listen(srv, 1);

    if (fork() == 0) {
        /* Child: client */
        sleep(0);   /* ensure server is listening */
        int cli = socket(AF_UNIX, SOCK_STREAM, 0);
        connect(cli, (struct sockaddr*)&addr, sizeof(addr));
        write(cli, "hi server", 9);
        close(cli);
        exit(0);
    }
    int conn = accept(srv, NULL, NULL);
    char buf[32];
    ssize_t n = read(conn, buf, sizeof(buf)-1);
    buf[n] = '\0';
    printf("server got: %s\n", buf);
    close(conn); close(srv);
    unlink(SOCK_PATH);
    wait(NULL);
    return 0;
}

This is one of the most important design axes in IPC. Understanding it transforms IPC from a bag of mechanisms into a coherent design space.

Synchronous IPC means the caller blocks until the operation completes. read() on a pipe blocks if the pipe is empty. msgrcv() blocks if the queue is empty. Synchronous IPC is easy to reason about — you write sequential code, the operation completes, you continue. The cost is that your thread cannot do other work while waiting.

Asynchronous IPC means the caller initiates an operation and continues without waiting. The kernel notifies the caller when the operation completes (via signal, callback, event fd, or io_uring completion ring). Examples: mq_notify registers a callback for when a message arrives; signalfd lets you wait on signals with epoll; io_uring queues I/O operations and harvests completions in a ring buffer.

The reactor pattern: most high-performance servers use asynchronous IPC with an event loop. A single thread registers interest in many file descriptors (pipes, sockets, signal fds, timer fds) using epoll_wait. When any of them becomes ready, epoll_wait returns and the thread handles that event. This is how nginx, Redis, and many other servers handle thousands of concurrent connections with a single thread.

io_uring: introduced in Linux 5.1, io_uring is the most radical rethinking of async I/O in Linux. It uses two shared-memory ring buffers (a submission queue and a completion queue) between the kernel and userspace. You submit operations without a syscall (just write to the SQ ring), and the kernel writes completions to the CQ ring. For high-frequency IPC, this eliminates most syscall overhead entirely.

The fundamental tradeoff: synchronous IPC is simpler to program and reason about. Asynchronous IPC is more efficient when latency matters and you have multiple concurrent channels. Most real systems use a mix: async I/O for network communication, synchronous shared-memory access for hot data paths, and signals for lifecycle events.

/* epoll-based reactor: wait on a pipe AND a timer, non-blocking */
#include <stdio.h>
#include <sys/epoll.h>
#include <sys/timerfd.h>
#include <unistd.h>
#include <stdint.h>

int main(void) {
    int pfd[2]; pipe(pfd);  /* pfd[0]=read, pfd[1]=write */

    /* timerfd fires every 1 second */
    int tfd = timerfd_create(CLOCK_MONOTONIC, 0);
    struct itimerspec ts = {
        .it_interval = {1, 0},
        .it_value    = {1, 0},
    };
    timerfd_settime(tfd, 0, &ts, NULL);

    int ep = epoll_create1(0);
    struct epoll_event ev;
    ev.events = EPOLLIN; ev.data.fd = pfd[0];
    epoll_ctl(ep, EPOLL_CTL_ADD, pfd[0], &ev);
    ev.data.fd = tfd;
    epoll_ctl(ep, EPOLL_CTL_ADD, tfd, &ev);

    /* From another thread or process, write to pfd[1] */
    struct epoll_event evs[2];
    for (int i = 0; i < 3; i++) {
        int n = epoll_wait(ep, evs, 2, -1);
        for (int j = 0; j < n; j++) {
            if (evs[j].data.fd == tfd) {
                uint64_t exp; read(tfd, &exp, 8);
                printf("timer fired\n");
            } else {
                char buf[64]; ssize_t r = read(pfd[0], buf, 63);
                buf[r] = 0; printf("pipe: %s\n", buf);
            }
        }
    }
    return 0;
}

Every IPC mechanism fits into one of two categories at the data level: it either copies data from one address space to another, or it shares a region of physical memory between address spaces. Understanding this distinction is essential for performance analysis.

Copy-based IPC (pipes, message queues, sockets): the sender's data is copied into a kernel buffer, then optionally copied again into the receiver's buffer. This is safe — after the send call returns, the sender can modify or free the original buffer without affecting the receiver. But copying is expensive for large payloads: copying 1 GB through a pipe requires 2 GB of memory bandwidth.

Share-based IPC (shared memory, memfd): no copy after setup. Both processes access the same physical memory. This is blazing fast — writing a 1 GB frame to shared memory requires only 1 GB of memory bandwidth, and the receiver pays zero additional cost to access it. The tradeoff: explicit synchronization is now the programmer's responsibility.

splice(2) and zero-copy tricks: Linux allows some copy-based IPC to avoid the userspace-kernel-userspace double-copy by keeping data in kernel pages and just remapping them. splice moves data between a file descriptor and a pipe without ever copying to userspace. sendfile does the same for socket-to-file transfers. This is how high-performance file servers work.

vmsplice(2): a particularly powerful call that 'donates' userspace pages into a pipe buffer — the kernel remaps those pages rather than copying, achieving zero-copy for pipe writes when the sender no longer needs the data.

The practical heuristic:

• < 1KB messages, high frequency: use a socket or pipe (copies are fast for small data, and you get automatic synchronization)

• 64KB messages, low frequency: consider shared memory with a small control channel (socket or pipe) to signal readiness

• Video, audio, large frames: shared memory + semaphore is almost always the right answer
• Cross-machine: sockets (obviously; shared memory doesn't work across machines)

/* splice: move data between two fds without copying to userspace */
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    if (argc < 3) { fprintf(stderr, "usage: %s src dst\n", argv[0]); return 1; }
    int src = open(argv[1], O_RDONLY);
    int dst = open(argv[2], O_WRONLY|O_CREAT|O_TRUNC, 0644);
    int pipe_fds[2]; pipe(pipe_fds);

    /* Two splice calls: file->pipe, pipe->file, zero userspace copy */
    ssize_t n;
    while ((n = splice(src, NULL, pipe_fds[1], NULL,
                       65536, SPLICE_F_MOVE)) > 0) {
        splice(pipe_fds[0], NULL, dst, NULL, n, SPLICE_F_MOVE);
    }
    return 0;
}

With all mechanisms on the table, here is how to reason about the choice:

Pipes: anonymous, byte-stream, copy-based, directional, blocking by default. Best for: shell pipelines, parent-child data flow, simple producer-consumer.

Named pipes (FIFOs): same as pipes but with a filesystem name, allowing unrelated processes to connect. Best for: simple unrelated-process communication without a server model.

Signals: asynchronous, zero data payload (just a number), delivered to a process. Best for: lifecycle events, interrupts, simple notifications. Terrible for: data transfer, high-frequency communication.

System V / POSIX message queues: message-preserving, prioritized, persistent (outlive senders/receivers), copy-based. Best for: task queues, command buses, priority-ordered work. Awkward API; consider using a socket + framing protocol instead in new code.

Shared memory: zero-copy, maximum bandwidth, requires explicit synchronization, no inherent ordering. Best for: large data (video, audio, ML tensors), shared caches, ring buffers between cooperating processes.

Unix domain sockets: bidirectional, connection-oriented or datagram, support fd-passing and credential passing, copy-based but fast for small messages. Best for: general-purpose local IPC, privilege-separated architectures, any case where you need to multiplex on many channels with epoll.

A key design principle: decouple control from data. A common pattern is to use a lightweight channel (socket or pipe) to pass control messages ("the next video frame is ready at offset 0x200000 in the shared region") while the actual data lives in shared memory. This gives you the synchronization guarantees of message passing and the bandwidth of shared memory simultaneously. D-Bus, Wayland, and Android's Binder all do variations of this.

Let's peer inside the kernel to see what actually happens when a process reads from a pipe. This bridges the gap between "IPC mechanism" and "kernel implementation" — essential for anyone working toward kernel contribution.

task_struct: every process is represented by a struct task_struct in the kernel. This structure contains the process's signal pending bitmap, its file descriptor table, its memory descriptor (mm_struct), and the wait queue entry used for blocking I/O.

Wait queues: when a process calls read() on an empty pipe, the kernel calls pipe_read() in fs/pipe.c. Finding no data, it calls pipe_wait(), which adds the process's task to a wait queue associated with the pipe and calls schedule() — surrendering the CPU. When data arrives (another process calls write()), the writer calls wake_up_interruptible() on the same wait queue, which sets the waiting task's state back to TASK_RUNNING and adds it to the run queue. The scheduler will eventually run it again, and it will re-check the condition.

copy_to_user and copy_from_user: the kernel cannot simply dereference a userspace pointer — the MMU is in kernel mode, and the address might be invalid, paged out, or from a malicious process. copy_from_user(kernel_buf, user_ptr, len) safely copies len bytes from userspace, handling page faults. copy_to_user does the reverse. These are the copies you pay for in pipe and socket IPC.

The read() syscall path: read() → sys_read() → vfs_read() → file->f_op->read_iter() → (for a pipe) pipe_read() → loop: check buffer, copy data with copy_to_user, update pipe head, wake writers, return bytes copied.

Understanding this call chain means you can read kernel source code, add tracepoints, write BPF programs to observe the behavior, and eventually write patches — because you know exactly where the interesting actions happen.

/* Tracing pipe_read with bpftrace (run as root):
   bpftrace -e 'kprobe:pipe_read { printf("pid %d read pipe\n", pid); }'

   Or with strace to see the syscall overhead:
   strace -e trace=read,write,pipe ./your_pipe_program

   To see copy_to_user in action with perf:
   perf trace -e 'syscalls:sys_enter_read' ./your_program
*/