Package Management & Building from Source

A Linux package is not magic — it is an archive (usually compressed with zstd, xz, or gzip) containing:

1. Compiled binaries and libraries — the built artifacts
2. Metadata — name, version, architecture, description, and crucially: the list of other packages this one depends on
3. Install/remove scripts — shell scripts that run before and after the files are placed (e.g., to add a system user, regenerate caches)
4. File manifest — the exact list of files and their checksums

On Debian/Ubuntu, packages are .deb files. On Fedora/RHEL, they are .rpm files. On Arch, they are .pkg.tar.zst files. You can inspect them directly:

# Inspect a .deb without installing it
ar x /var/cache/apt/archives/curl_7.88.1-10_amd64.deb
tar -tvf data.tar.zst          # list every installed file
tar -xOf control.tar.zst ./control  # print metadata

# On RPM-based systems
rpm -qpl some-package.rpm      # list files
rpm -qpi some-package.rpm      # print metadata/description

# On Arch
tar -tvf some-package.pkg.tar.zst

The key insight: a package is just a structured archive plus metadata. The package manager is the software that reads that metadata, resolves the dependency graph, and unpacks files into the right places. The package itself is passive data.

# Peek inside a .deb package (Debian/Ubuntu)
cd /tmp
apt-get download curl          # download without installing
ar x curl_*.deb                # ar is the archive tool
tar -tvf data.tar.zst | head -20  # see every file it installs

The three dominant package managers serve the same purpose but have meaningfully different designs.

apt (Debian, Ubuntu, Mint) Uses .deb packages from repositories listed in /etc/apt/sources.list and /etc/apt/sources.list.d/. The workflow:

• apt update — fetches the package index (a compressed list of all available packages and their metadata) from each repository. Nothing is installed; you are updating the local database.
• apt install foo — resolves dependencies, shows what will change, and installs.
• apt remove foo — removes binaries but keeps config files.
• apt purge foo — removes binaries and config files.
• dpkg -l — list all installed packages (lower-level tool that apt wraps).
• dpkg -L curl — list every file the curl package installed.
• dpkg -S /usr/bin/curl — reverse lookup: which package owns this file?

dnf (Fedora, RHEL, CentOS Stream) dnf replaced yum and is built on libsolv, a very sophisticated SAT-solver based dependency resolver. Key commands mirror apt but syntax differs: dnf install, dnf remove, dnf update, rpm -ql curl, rpm -qf /usr/bin/curl.

pacman (Arch Linux) Arch's philosophy is "rolling release" — there are no version numbers for the distro, packages are always updated to latest upstream. pacman -Syu (sync + refresh + upgrade) is the single command that keeps an Arch system current. -Q queries the local database, -R removes, -U installs a local file. Arch also has the AUR (Arch User Repository), where community-maintained PKGBUILD scripts let users build packages from source in a reproducible way.

Repository trust model Every major package manager uses GPG (GNU Privacy Guard) signatures. The distro maintainers sign each package and the index. Your local package manager verifies these signatures before installing anything. This is the chain of trust: you trust the distro, the distro's key is in your keyring, and every package is signed by that key. Breaking this chain (e.g., with --no-check-certificate or --allow-unauthenticated) is a serious security decision.

# apt essentials
apt update                      # refresh package index
apt install build-essential     # install gcc, make, libc headers
apt show build-essential        # show metadata + dependencies
dpkg -L build-essential         # list files this package installed
dpkg -S $(which gcc)            # which package owns /usr/bin/gcc?

# pacman essentials  
pacman -Syu                     # full system upgrade
pacman -Ss 'build tools'        # search
pacman -Qi gcc                  # query installed package info
pacman -Ql gcc                  # list files in gcc package
pacman -Qo /usr/bin/gcc         # who owns this file?

When you run apt install foo, the package manager does not just install foo. It installs foo plus every package that foo depends on, recursively, while simultaneously ensuring no two installed packages conflict. This is a constraint satisfaction problem over a dependency graph, and it is harder than it looks.

Consider: package A depends on libssl >= 1.1, package B depends on libssl < 3.0, and you want both A and B. If your system has libssl 3.1, package B might be unsatisfiable. This is the classic "dependency hell" that package managers were invented to tame.

Dependency types (Debian model):

• Depends — hard requirement; must be installed
• Recommends — installed by default but can be skipped with --no-install-recommends
• Suggests — not installed by default; just informational
• Conflicts — cannot coexist with another package
• Provides — this package satisfies a virtual package name (e.g., mail-transport-agent)
• Breaks — installing this package breaks another

Why does apt sometimes say "the following packages will be removed"? Because installing X requires version 2 of library L, but Y (already installed) requires version 1 of L, and they conflict. The resolver found a solution but it involves removing Y. This is not a bug — it is correct behavior from the constraint solver.

apt-cache depends and apt-cache rdepends let you explore the graph manually. Understanding this graph is essential when you are trying to pin a specific kernel version for testing while keeping the rest of the system updated.

# Explore the dependency graph
apt-cache depends curl          # what does curl need?
apt-cache rdepends libssl3      # what needs libssl3?

# See exactly what would change before committing
apt install --dry-run vim

# Install without recommended packages (useful in containers)
apt install --no-install-recommends python3

# Hold a package at its current version (useful for kernel pinning)
apt-mark hold linux-image-generic
apt-mark showhold

When software is installed, where do the files go? The Linux Filesystem Hierarchy Standard (FHS) gives the answer, and understanding it prevents a lot of confusion when you mix distribution packages with hand-built software.

Key directories:

• /usr/bin — user binaries from the distro's package manager. Never put your own stuff here.
• /usr/lib — libraries from the distro.
• /usr/include — C/C++ header files from the distro.
• /usr/local/bin — binaries you installed yourself (not via the package manager). This is your sandbox.
• /usr/local/lib — your locally-built libraries.
• /usr/local/include — your locally-built headers.
• /opt — self-contained third-party software (each app gets its own subdirectory like /opt/google/chrome).
• /etc — system-wide configuration files.
• /var — variable data (logs, caches, databases).

Why the /usr/local separation matters: /usr/local has priority in $PATH over /usr/bin by default on most distros. So if you build a newer version of git from source and install it to /usr/local, it will shadow the distro's older /usr/bin/git. When the package manager later upgrades the distro's git, your local one still takes priority. This is powerful but can cause confusion — running which git is always worth doing to verify what you are actually executing.

The package manager owns /usr/bin and /usr/lib. It keeps a manifest of every file it has placed there. If you manually put files in those directories, the package manager does not know about them, and a future package upgrade might silently overwrite your work. Always use /usr/local for hand-built software.

For kernel development specifically: the kernel itself installs to /boot (vmlinuz, initramfs), /lib/modules/$(uname -r)/ (modules), and /usr/src/linux-headers-$(uname -r)/ (headers for out-of-tree module builds). These are managed by the package manager too.

# See the hierarchy in action
ls /usr/bin/git                 # distro git
which git                       # what actually runs

# After building git from source with prefix=/usr/local:
# /usr/local/bin/git takes priority in PATH

# Check PATH order
echo $PATH | tr ':' '\n'

# Find all files a package installed
dpkg -L git | grep bin

# What's in /usr/local (your hand-built stuff)
find /usr/local -maxdepth 3 -type f | head -20

The most common build process for C/C++ open-source software has three phases, each with a specific job.

Phase 1: ./configure This is a shell script (usually generated by GNU Autoconf) that probes your system: Does gcc exist? What version of libssl is installed? Does your kernel support feature X? It writes the results into config.h (a C header with #define statements like #define HAVE_OPENSSL 1) and Makefile (the actual build instructions, customized for your system).

Key options:

• --prefix=/usr/local — where to install. Default is /usr/local.
• --sysconfdir=/etc — where config files go.
• --disable-foo / --enable-foo — turn optional features on or off.
• --with-openssl=/path — tell configure where a dependency lives if it is in a non-standard location.

Run ./configure --help to see all options. This is not optional reading — for kernel-related libraries, the feature flags control what gets compiled in.

Phase 2: make Reads the Makefile and compiles everything. The Makefile expresses a dependency graph of build artifacts: foo.o depends on foo.c and foo.h; libfoo.so depends on all .o files; the final binary depends on libfoo.so. make only rebuilds what has changed since the last build — this is what makes make fast for iterative development.

make -j$(nproc) parallelizes the build across all CPU cores. You will use this constantly when building the kernel.

Phase 3: make install Copies the compiled artifacts to the installation prefix. Binaries go to $prefix/bin, libraries to $prefix/lib, headers to $prefix/include, man pages to $prefix/share/man. This step may need sudo if installing to /usr/local.

The DESTDIR trick: make install DESTDIR=/tmp/staging installs into a staging directory rather than the real prefix. This is how distro packagers work: they build software, stage it into a temp directory, then pack that directory into a .deb or .rpm. It means the package never actually touches the live system during the build process.

# Full workflow: build htop from source
git clone https://github.com/htop-dev/htop.git
cd htop
./autogen.sh               # generates ./configure from configure.ac
./configure --prefix=/usr/local --enable-unicode
make -j$(nproc)            # parallel build
make install               # installs to /usr/local
which htop                 # should be /usr/local/bin/htop

# To undo: the install manifest is in the source tree
cat install_manifest.txt   # cmake projects; autotools uses 'make uninstall'
make uninstall             # if the project supports it

When you compile a program that calls printf, the code for printf is not copied into your binary. Instead, your binary records a dependency on libc.so.6 (on Linux). At runtime, the dynamic linker (/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2) loads your binary, reads its list of required shared libraries, finds them, maps them into memory, and resolves all the symbol addresses. This is shared library (also called dynamically linked library or DSL) linking.

Why this matters for building from source: When you ./configure, it needs to find the development version of libraries: the headers (.h files) and the import stub (.so file without the version suffix). On Debian/Ubuntu, nearly every library comes in two packages:

• libssl3 — the runtime .so.3 shared library (needed to run programs)
• libssl-dev — the headers and .so symlink (needed to build programs against it)

Forgetting to install -dev packages is the single most common reason ./configure fails.

Library search path: The dynamic linker searches in order:

1. RPATH or RUNPATH embedded in the binary itself
2. $LD_LIBRARY_PATH environment variable
3. /etc/ld.so.cache (a binary cache built from /etc/ld.so.conf and its includes)
4. Default paths: /lib, /usr/lib

When you install a library to /usr/local/lib from source, you must either:

• Add /usr/local/lib to /etc/ld.so.conf.d/local.conf and run sudo ldconfig to rebuild the cache, or
• Set LD_LIBRARY_PATH=/usr/local/lib before running the program.

Running ldd /usr/bin/curl prints all shared library dependencies and where each one was found. If a library shows not found, the program will not run.

pkg-config: A tool that tells you what compiler and linker flags are needed to use a library. pkg-config --cflags --libs openssl might print -I/usr/include/openssl -lssl -lcrypto. Configure scripts use this heavily. If ./configure fails with "cannot find openssl", often pkg-config --exists openssl will show you what is wrong.

# Check what libraries a binary needs at runtime
ldd /usr/bin/curl

# Install a library from source, then register it
sudo cp libmylib.so.1.0 /usr/local/lib/
echo '/usr/local/lib' | sudo tee /etc/ld.so.conf.d/local.conf
sudo ldconfig              # rebuild cache
ldconfig -p | grep mylib   # verify it is found

# Use pkg-config to get build flags
pkg-config --cflags --libs zlib
# outputs: -lz   (just the linker flag for zlib)

# Install missing dev packages (Debian/Ubuntu)
sudo apt install libssl-dev zlib1g-dev

When you install a package from your distro, you are often not running the exact upstream source code. Distro maintainers routinely apply patches before building. Understanding why illuminates the entire distro ecosystem.

Reasons distros patch:

1. Security backports — A critical CVE is fixed in upstream version 2.5 but the distro ships 2.3 (because upgrading to 2.5 would break other things). The maintainer extracts just the security fix and applies it to 2.3. The version number gains a suffix: 2.3-4ubuntu5 means upstream 2.3, Debian revision 4, Ubuntu patch 5.

2. Filesystem hierarchy compliance — Upstream software might want to install to /usr/local or use non-standard paths. The distro patches the build system to comply with FHS and the distro's own conventions.

3. Integration with distro systems — A logging library gets patched to use systemd-journald instead of plain syslog. An init script gets replaced with a .service file.

4. Fixing bugs the upstream hasn't merged yet — Maintainers sometimes apply patches from upstream's bug tracker that are waiting for the next release.

5. Build reproducibility — Patches to remove embedded timestamps, non-deterministic behavior, or hardcoded paths.

How to see distro patches: On Debian/Ubuntu, apt source foo downloads the source package. You get the upstream tarball plus a debian/patches/ directory of .patch files applied on top. On Fedora, dnf download --source foo gives you an .src.rpm with the same structure. The quilt tool manages these patch series.

Why this matters for kernel development: Distro kernels are heavily patched. Ubuntu 22.04 ships a kernel derived from 5.15 but with hundreds of patches applied — for hardware support, security, performance, and Ubuntu-specific features. When you test a fix against the mainline kernel and a distro kernel, you may see different behavior because of this. Always test against linux-next or mainline for upstream submissions.

# See the distro's patches for a package (Debian/Ubuntu)
apt-get source curl            # downloads source + patches
ls curl-*/debian/patches/      # list all applied patches

# The version tells the story:
# curl 8.5.0-2ubuntu1
#  └── upstream: 8.5.0
#      └── Debian revision: 2
#          └── Ubuntu delta: 1

# Verify what kernel you are running vs upstream
uname -r                       # e.g., 6.8.0-41-generic
# compare to https://kernel.org — likely many patches behind latest

Building the kernel from source is a rite of passage and a practical necessity for kernel development. The process differs from autotools — the kernel uses its own Kconfig / Kbuild system — but the concepts are the same: configure, build, install.

Step 1: Get the source

git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux

For a shallower introduction, clone the stable tree (linux-stable) or the linux-next tree (for upcoming features).

Step 2: Configure The kernel has thousands of configuration options (drivers, filesystems, security features, debugging). make menuconfig opens a terminal UI. For a first build, the easiest start is to base your config on your running kernel:

cp /boot/config-$(uname -r) .config
make olddefconfig   # resolve any new options with defaults

Step 3: Build

make -j$(nproc)        # build everything: vmlinux, modules, bzImage
make -j$(nproc) M=drivers/net/ethernet/intel/e1000  # build just one module

A full build takes 10–60 minutes depending on hardware.

Step 4: Install

sudo make modules_install     # copies .ko files to /lib/modules/$(version)/
sudo make install             # copies bzImage + System.map to /boot/, updates grub

Prerequisites (the most common cause of build failures):

sudo apt install build-essential libncurses-dev bison flex libssl-dev libelf-dev

Missing these is where the package management knowledge from this module pays off immediately.

# Prerequisites on Debian/Ubuntu
sudo apt install -y \
  build-essential libncurses-dev bison flex \
  libssl-dev libelf-dev dwarves bc

# Clone and configure
git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
cp /boot/config-$(uname -r) .config
make olddefconfig

# Build (this takes a while!)
make -j$(nproc) 2>&1 | tee build.log

# Check the output
ls arch/x86/boot/bzImage      # the compressed kernel image
ls vmlinux                    # the uncompressed ELF (useful for debugging)

While autotools (./configure && make) is the classic and is used by the kernel's own toolchain dependencies, newer projects use CMake or Meson. As a kernel developer you will encounter both in the userspace tools you need to build.

CMake generates build files (Makefiles, or Ninja files) from a CMakeLists.txt. The workflow uses an out-of-source build directory — a clean pattern that keeps build artifacts separate from source:

mkdir build && cd build
cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)
make install

The -D flags are how you pass options (equivalent to ./configure --enable-foo).

Meson is newer, faster, and increasingly common in projects like glib, mesa, and modern kernel tooling. It always uses Ninja as its backend:

meson setup build --prefix=/usr/local
cd build
ninja -j$(nproc)
sudo ninja install

Key difference from autotools: Autotools runs feature detection in the source directory and modifies it. CMake and Meson use an out-of-source build directory. This means you can have multiple build directories (debug and release, or two different configurations) from the same source tree simultaneously — extremely useful when comparing a patched vs. unpatched build.

For kernel development specifically: perf (the kernel's performance analysis tool) and many kernel test tools use these modern build systems. The libbpf library and bpftool use a mix of Makefiles and CMake.

# CMake out-of-source build pattern
git clone https://github.com/nicowillis/someproject.git
cd someproject
mkdir build-release && cd build-release
cmake -DCMAKE_INSTALL_PREFIX=/usr/local \
      -DCMAKE_BUILD_TYPE=Release \
      ..
make -j$(nproc)
sudo make install

# Meson pattern (e.g., building a glib-based tool)
meson setup build --prefix=/usr/local --buildtype=release
cd build
ninja -j$(nproc)
sudo ninja install

make install directly into /usr/local works, but it has a problem: there is no easy way to uninstall later, because the install step scatters files across dozens of directories with no manifest. Over time, a system accumulates ghost files from old builds. Here are the tools to stay sane.

GNU Stow is a symlink farm manager. The idea: instead of installing to /usr/local, install to a per-package prefix like /usr/local/stow/htop-3.3.0/. Then run stow -v htop-3.3.0 from /usr/local/stow/, and Stow creates symlinks in /usr/local/bin, /usr/local/lib, etc., pointing into the stow directory. To uninstall: stow -D htop-3.3.0 removes all symlinks. Clean, reversible, no packaging required.

checkinstall intercepts the make install call, watches which files it would install, and creates a real .deb or .rpm package instead. You then install that package with your package manager. This means the package manager knows about every file, and you can uninstall cleanly with apt remove. For quick one-off builds on Debian systems, this is the most ergonomic solution.

Containers for build isolation: For kernel development, the cleanest approach is to use a container (Docker, Podman, or a systemd-nspawn container) as a reproducible build environment. You install all build dependencies inside the container, build there, and only copy out the artifacts. This keeps your host system immaculate and makes the build reproducible across machines.

Distrobox is a newer tool that creates a mutable container of any distro (Ubuntu, Fedora, Arch) on top of any host. You can have an Ubuntu 22.04 toolchain container on a Fedora 40 host, which is invaluable when a project requires a specific distribution's libraries.

# Using GNU Stow for clean installs
sudo apt install stow

# Build htop with a stow prefix
./configure --prefix=/usr/local/stow/htop-3.3.0
make -j$(nproc)
sudo make install           # installs to /usr/local/stow/htop-3.3.0/

# Activate (creates symlinks in /usr/local/)
cd /usr/local/stow
sudo stow htop-3.3.0
which htop                  # /usr/local/bin/htop -> stow/htop-3.3.0/bin/htop

# Deactivate later
sudo stow -D htop-3.3.0     # removes all symlinks, files remain intact

# Using checkinstall instead of make install
sudo apt install checkinstall
./configure --prefix=/usr/local
make -j$(nproc)
sudo checkinstall           # interactive: creates and installs a .deb
dpkg -l | grep htop         # package manager now tracks it

Your package manager's sources list tells it where to look for packages. Understanding repository management is essential because kernel developers often need newer toolchains (GCC 14, Clang 17, LLVM) or specific library versions than the distro provides.

Debian/Ubuntu repository structure: A repository URL contains four components: the base URL, the distribution codename (jammy, noble), the component (main = distro-supported; universe = community; multiverse = non-free), and architecture. /etc/apt/sources.list.d/ allows adding repositories modularly.

PPAs (Personal Package Archives) on Ubuntu Launchpad allow developers to publish .deb packages. add-apt-repository ppa:ubuntu-toolchain-r/test adds the Toolchain test PPA, which often has GCC and Clang builds newer than the distro default. This is how kernel developers get GCC 13 on Ubuntu 22.04. The trade-off: you trust an individual developer's packages, not the distro's security team.

COPR (Cool Other Package Repositories) is the Fedora equivalent of PPAs — community-hosted .rpm repositories.

Pinning / preferences: When you mix a PPA with the distro repository, you might not want all packages from the PPA — just gcc. APT's pinning mechanism (/etc/apt/preferences.d/) lets you assign priorities: default priority is 500; a pinned package at 1001 overrides even newer versions from other repos. This is how you say "give me GCC from the toolchain PPA, but everything else from Ubuntu main."

Security consideration: Every repository you add increases your attack surface. Malicious PPAs have been used to compromise systems. For kernel development, prefer official distro repositories plus the official LLVM/Clang apt.llvm.org repository (which is maintained by the LLVM project itself).

# Add the LLVM official repository for latest Clang (Debian/Ubuntu)
wget -qO- https://apt.llvm.org/llvm-snapshot.gpg.key | sudo tee /etc/apt/trusted.gpg.d/apt.llvm.org.asc
# Add the source line:
echo "deb http://apt.llvm.org/jammy/ llvm-toolchain-jammy-17 main" \
  | sudo tee /etc/apt/sources.list.d/llvm-17.list
sudo apt update
sudo apt install clang-17 llvm-17

# Pin: use LLVM repo ONLY for clang, not everything
cat /etc/apt/preferences.d/llvm <<'EOF'
Package: *
Pin: release o=apt.llvm.org
Pin-Priority: 100

Package: clang-17 llvm-17 lld-17
Pin: release o=apt.llvm.org
Pin-Priority: 900
EOF