Git Deep: The Kernel's Workflow

Everything Git stores lives in .git/objects/. There are exactly four object types: blob, tree, commit, and tag. Every object is stored by its SHA-1 hash (Git is transitioning to SHA-256; the logic is identical). The hash covers the object's type, size, a null byte, and then its content — so the same file content always produces the same hash, regardless of filename or location in the tree.

A blob stores raw file content — nothing else. No filename, no permissions. That information lives higher up. Two files with identical content share one blob object.

A tree is a directory listing: a sorted list of (mode, name, SHA) triples, where the SHA points to either a blob (file) or another tree (subdirectory). Trees are immutable; changing one file means creating a new blob, a new tree for its parent directory, a new tree for each ancestor directory up to the root — a chain of new objects.

A commit points to exactly one root tree (the state of the entire repository at that moment), zero or more parent commits (usually one, two for a merge, zero for an initial commit), and metadata: author, committer, timestamp, and the commit message. Because a commit's SHA covers its parent SHA(s), you cannot alter history without changing every descendant's hash — that is the integrity guarantee.

A tag object (annotated tag) wraps a commit SHA with a tagger name, timestamp, and message. Lightweight tags are just refs, not objects.

You can inspect any object directly:

# Look at the raw object database
ls .git/objects/

# Hash a string the same way git would (type=blob)
printf 'blob 5\0hello' | sha1sum
# same as:
echo -n 'hello' | git hash-object --stdin

# Inspect a commit
git cat-file -p HEAD
# tree 4b825dc642cb6eb9a060e54bf8d69288fbee4904
# parent a1b2c3d4...
# author Linus Torvalds <torvalds@linux-foundation.org> 1234567890 -0700
# committer ...
#
# treewide: Add SPDX-License-Identifier tags

# Inspect the tree it points to
git cat-file -p HEAD^{tree}
# 100644 blob 257cc5642... Makefile
# 040000 tree 9daeafb9e... arch
# 040000 tree c93be7e36... drivers
# ...

# See what type and size any object is
git cat-file -t HEAD    # commit
git cat-file -s HEAD    # e.g. 287

# Walk the whole object graph for a file
git ls-tree -r HEAD -- drivers/usb/core/hub.c

Git's history is a directed acyclic graph (DAG). Each commit node has directed edges pointing to its parent(s). The direction matters: commits point backward to their parents, never forward. The graph is acyclic: you cannot follow parent edges and return to your starting commit — that would mean a commit is its own ancestor, which would be a paradox (and would break the SHA integrity chain).

Why a graph and not a simple list? Because of merges. When you merge two branches, the resulting commit has two parents. Its SHA covers both parent SHAs, so it cryptographically records that it is the union of both lines of development. A rebase, by contrast, replays commits onto a new base, creating new commits (new SHAs) — the old commits still exist until garbage-collected.

The DAG has beautiful properties for distributed work. Two developers can clone a repo, make disjoint commits, and then merge with no central arbiter needed — because each commit is globally unique by its SHA. git pull is literally just git fetch (download the other DAG nodes) followed by git merge or git rebase (attach your local DAG to the downloaded one).

In the kernel, Linus's tree is the canonical tip of the DAG. Subsystem maintainers each have their own forks of the DAG, which Linus merges at rc1. Those maintainers pull from developer trees. The whole structure is a tree of DAGs, and git log --graph --oneline will draw it for you.

A critical concept: reachability. A commit X is reachable from commit Y if you can get from Y to X by following parent edges. git log Y shows all commits reachable from Y. git diff A B is the difference between the trees that A and B point to. git merge-base A B finds the most recent common ancestor in the DAG — the point where the two lines of development diverged — which is how Git knows what to merge.

# Visualize the DAG beautifully
git log --graph --oneline --decorate --all

# Find merge base (common ancestor) of two branches
git merge-base main feature/dma-rework

# Count commits reachable from HEAD but not from origin/main
git rev-list --count origin/main..HEAD

# Show all commits that touched a specific file, across the entire DAG
git log --follow --oneline -- drivers/net/ethernet/intel/e1000/e1000_main.c

# A merge commit has two parents; inspect them
git cat-file -p <merge-commit-sha>
# parent <sha1>   <- first parent (branch you merged INTO)
# parent <sha2>   <- second parent (branch you merged FROM)

# Show the symmetric difference: what's in A or B but not both
git log --oneline A...B

The object database is content-addressed and immutable, but humans need names. Refs are the naming layer: files under .git/refs/ (or packed in .git/packed-refs) that contain a SHA. A branch like main is simply .git/refs/heads/main containing a 40-character SHA. When you make a new commit on main, Git updates that file to the new commit's SHA. That is all a branch is — a mutable pointer to an immutable commit node in the DAG.

HEAD is the special ref at .git/HEAD. When you are on a branch, HEAD contains the string ref: refs/heads/main (a symbolic ref — a ref that points to another ref). When you check out a specific commit hash directly (detached HEAD state), HEAD contains that hash directly. Nearly every Git operation reads HEAD to know where you are.

The index (also called the staging area or cache) lives at .git/index. It is a binary file that represents the proposed next commit — a sorted list of (mode, SHA, stage, filename) entries. When you run git add, Git hashes the file into the object store (creating a blob) and updates the index entry. When you run git commit, Git writes the index as a tree object, then creates a commit pointing to that tree and to HEAD's current commit as parent, then advances the branch ref to the new commit SHA.

This three-stage model — working tree, index, object store — is the source of most Git confusion, and also its power. You can stage only part of a file (git add -p), you can stash the working tree while keeping the index, and you can inspect the index directly with git ls-files --stage. Understanding the index makes commands like git reset (which can move HEAD, the index, and/or the working tree independently) fully predictable rather than magical.

# What HEAD points to
cat .git/HEAD
# ref: refs/heads/main

# What main points to
cat .git/refs/heads/main
# a3f1bc9d...

# The index: every staged file, its mode and blob SHA
git ls-files --stage
# 100644 257cc5642... 0       Makefile
# 100644 9daeafb9e... 0       drivers/net/ethernet/intel/e1000/e1000_main.c

# Add only part of a file (interactive hunk staging)
git add -p drivers/usb/core/hub.c

# Understand git reset's three modes:
# --soft:  move branch ref only (index+worktree unchanged)
# --mixed: move branch ref + reset index  (worktree unchanged, this is the default)
# --hard:  move branch ref + reset index + reset worktree
git reset --soft HEAD~1   # undo last commit, keep changes staged
git reset --mixed HEAD~1  # undo last commit, unstage changes
git reset --hard HEAD~1   # undo last commit, DISCARD changes (dangerous)

# The reflog: git's undo history for refs
git reflog show main
# main@{0}: commit: net: fix null deref in e1000_probe
# main@{1}: reset: moving to HEAD~1
# main@{2}: commit: net: add e1000 driver skeleton

Because a branch is just a file containing a SHA, creating a branch in Git is O(1): write a file. This is why Git branches are so cheap compared to older VCSes like SVN, where branching meant copying a directory tree.

Merging takes two (or more) branch tips and creates a new commit that is a descendant of all of them. Git uses a three-way merge: it finds the merge base (common ancestor), diffs each branch tip against the merge base, and combines those diffs. If the same lines were changed differently on both sides, that is a merge conflict — Git cannot automatically resolve it and asks you to.

A fast-forward merge is a special case: if the branch you are merging into has not diverged from the branch you are merging in (i.e., the target is a direct ancestor of the source), Git simply advances the pointer. No merge commit is created. This is clean but loses the information that a branch existed. git merge --no-ff forces a merge commit even when fast-forward is possible — the kernel workflow often prefers this for feature branches so the integration point is visible in the DAG.

Merge strategies: Git has several. The default ort (previously recursive) handles renamed files and criss-cross merges. The octopus strategy merges more than two branches at once — Linus actually uses this when pulling from many subsystem maintainers simultaneously. Understanding which strategy is being used and why helps you predict (and debug) conflicts.

For kernel work, you almost never merge your feature branch into another developer's tree manually. Instead, you produce a patch series that the maintainer applies. But understanding merges is essential because you will see them everywhere in git log --graph of the mainline kernel tree, and you will need to rebase on top of merge commits.

# Create and switch to a new branch (modern syntax)
git switch -c fix/null-deref-e1000

# Make a commit
git add drivers/net/ethernet/intel/e1000/e1000_main.c
git commit -m 'e1000: fix null pointer dereference in e1000_probe'

# Switch back and merge (creates a merge commit explicitly)
git switch main
git merge --no-ff fix/null-deref-e1000

# Inspect the resulting merge commit
git log --graph --oneline -5

# Resolve a conflict manually:
# Git marks conflicts in the file like:
#   <<<<<<< HEAD
#   	if (!adapter)
#   =======
#   	if (!adapter || !adapter->hw)
#   >>>>>>> fix/null-deref-e1000
# Edit the file to keep the correct version, then:
git add drivers/net/ethernet/intel/e1000/e1000_main.c
git merge --continue

# Use a merge tool (e.g. vimdiff)
git mergetool

Rebase is Git's history-rewriting tool. Where merge joins two lines of history, rebase moves one line of history onto another — it replays each commit as a patch on top of a new base commit, creating new commits with new SHAs.

The mental model: imagine your patch series is a stack of Post-it notes. Rebasing lifts the stack off its current position and places it on top of a different commit. Each note gets a new SHA (because its parent changed) but its diff (the changes it records) is the same.

Why rebase for kernel patches? The kernel maintainers want a linear, clean history in submitted patches. If you branch off v6.9-rc3, do 5 commits, and then v6.9-rc4 comes out with a fix you depend on, you git rebase v6.9-rc4 to replay your 5 commits on top of the new base. The maintainer then sees a clean series of patches that apply to the current tip, not a tangle of merges.

Interactive rebase (git rebase -i) is your editing suite for a patch series. You can reorder commits, squash multiple commits into one, split one commit into many, edit commit messages, or drop commits entirely. This is how you polish a rough draft of patches into a clean, reviewable series before sending to the mailing list.

The golden rule: never rebase commits that have already been pushed to a shared public branch. Rebase rewrites history (new SHAs), so anyone who had the old SHAs now has diverged history. For your private branch before submitting, rebase freely. After sending patches to the list for review, a new version ([PATCH v2]) is a new submission, not an in-place rewrite.

Rebase vs merge: rebase produces a cleaner, linear history that is easier to read and bisect. Merge preserves the exact history of when branches diverged and rejoined. The kernel uses rebase for patch series preparation, and merge (via pull requests from maintainers) for integrating subsystem trees into mainline.

# Rebase your branch onto the latest upstream
git fetch origin
git rebase origin/main

# Interactive rebase: edit the last 4 commits
git rebase -i HEAD~4
# Editor opens with:
#   pick a1b2c3d e1000: add initial driver skeleton
#   pick b2c3d4e e1000: add interrupt handler
#   pick c3d4e5f fixup: fix typo in comment   <- we want to squash this
#   pick d4e5f6a e1000: add power management
#
# Change 'pick' to 'squash' (or 's') on the typo fix:
#   pick a1b2c3d e1000: add initial driver skeleton
#   pick b2c3d4e e1000: add interrupt handler
#   squash c3d4e5f fixup: fix typo in comment
#   pick d4e5f6a e1000: add power management

# Rebase a patch series onto a specific kernel tag
git rebase v6.9-rc4 --onto v6.9-rc4 v6.9-rc3

# If a rebase conflict occurs:
# Fix the conflict in the file, then:
git add <conflicted-file>
git rebase --continue
# Or abort entirely:
git rebase --abort

# Rebase preserving merge commits (for complex histories)
git rebase --rebase-merges origin/main

git format-patch is the bridge between Git's commit history and the Linux kernel's email-based review workflow. It takes commits from your branch and serializes each one as a .patch file in the mbox format — the same format that email clients use for storing messages. Each patch file contains the commit metadata (author, date, message) as email headers, followed by the diff in --- a/ +++ b/ unified diff format.

A patch series is a sequence of related patches submitted together, prefixed [PATCH 0/N] through [PATCH N/N]. Patch 0 is a cover letter: an overview email that explains the purpose of the series as a whole. Individual patches ([PATCH 1/N] etc.) each make one logical change with a commit message that explains why, not just what.

Kernel commit message conventions are strict and meaningful:

• Subject line format: subsystem: component: brief description (e.g., net: e1000: fix null pointer dereference in probe)
• Body: starts with a blank line, then explains the problem being solved, why the fix is correct, and any relevant context. Mention error messages, call traces, or how to reproduce.
• Signed-off-by: a DCO (Developer Certificate of Origin) line: Signed-off-by: Your Name <your@email>. This is required for all kernel patches. It certifies you have the right to submit the code under the kernel's license.
• Reviewed-by, Acked-by, Tested-by: tags added by reviewers and testers to indicate their sign-off.

The --cover-letter flag generates the patch 0. The -v2 flag (via --subject-prefix='PATCH v2') marks a revised submission. Always include --base= to specify which commit your patches are based on, so reviewers can apply them unambiguously.

# Format a single patch
git format-patch -1 HEAD
# Outputs: 0001-e1000-fix-null-pointer-dereference-in-probe.patch

# Format a series of 3 patches with a cover letter
git format-patch -3 --cover-letter --base=v6.9-rc4 -o patches/
# patches/0000-cover-letter.patch
# patches/0001-e1000-add-initial-driver-skeleton.patch
# patches/0002-e1000-add-interrupt-handler.patch
# patches/0003-e1000-add-power-management.patch

# Mark as version 2 of a previously sent series
git format-patch -3 --cover-letter --subject-prefix='PATCH v2' -o patches-v2/

# What a well-formed patch looks like (abbreviated):
cat patches/0001-e1000-fix-null-pointer-dereference-in-probe.patch
# From a1b2c3d4... Mon Sep 17 00:00:00 2001
# From: Jane Developer <jane@example.com>
# Date: Mon, 8 Jun 2026 14:30:00 +0530
# Subject: [PATCH 1/3] e1000: fix null pointer dereference in probe
#
# When e1000_probe() fails to allocate the adapter structure,
# a subsequent call to e1000_remove() dereferences adapter->hw
# without checking for NULL, causing a kernel oops.
#
# Fix by adding a NULL check before the dereference.
#
# Reported-by: Fuzzer Bot <fuzzer@example.com>
# Signed-off-by: Jane Developer <jane@example.com>
# ---
#  drivers/net/ethernet/intel/e1000/e1000_main.c | 5 +++--
#  1 file changed, 3 insertions(+), 2 deletions(-)
# diff --git a/drivers/net/ethernet/intel/e1000/e1000_main.c ...

# Validate your patch before sending (catches common mistakes)
scripts/checkpatch.pl patches/0001-*.patch

# Find the right maintainers to send to
scripts/get_maintainer.pl patches/0001-*.patch

git send-email sends your .patch files as real emails, with correct threading (each patch in a series is a reply to the cover letter, and review responses thread under their respective patches). This threading is critical: it keeps the entire discussion of a patch series in one visible thread on the mailing list archive, making it reviewable by the whole community.

Setting up git send-email requires configuring an SMTP server in .git/config or ~/.gitconfig. You can use Gmail's SMTP, a university server, or your ISP. The kernel community is particular about email formatting — HTML, base64 encoding, or line-wrapping that breaks the diff will cause your patch to be rejected immediately. git send-email avoids all of these by sending plain text.

Before sending to the real mailing list, always test by sending to yourself. Review the received email in a plain-text mail client (Mutt, NeoMutt, Thunderbird in plain-text mode, or the web archive at lore.kernel.org) and make sure the diff is intact and the formatting is clean.

The --to flag sets the primary recipient (usually the subsystem mailing list, like linux-kernel@vger.kernel.org or netdev@vger.kernel.org). The --cc flag adds the maintainers returned by scripts/get_maintainer.pl. The --in-reply-to flag is used for v2/v3 submissions to thread them under the original patch series thread.

After your patches are sent, watch the mailing list archive (lore.kernel.org) and your inbox for review feedback. If reviewers request changes, you submit a [PATCH v2] series addressing all feedback, with a changelog at the bottom of the cover letter (or after the --- separator in single patches) listing what changed between versions.

# Configure git send-email in ~/.gitconfig
# [sendemail]
#     smtpServer = smtp.gmail.com
#     smtpServerPort = 587
#     smtpEncryption = tls
#     smtpUser = jane@gmail.com
#     from = Jane Developer <jane@example.com>
#     confirm = always

# Dry run first: see what would be sent without actually sending
git send-email --dry-run patches/

# Send to yourself first (always test this)
git send-email --to=jane@example.com patches/

# Real submission: send to the list and cc maintainers
git send-email \
  --to=netdev@vger.kernel.org \
  --cc=linux-kernel@vger.kernel.org \
  --cc='Jeff Kirsher <jeffrey.t.kirsher@intel.com>' \
  --cc='Tony Nguyen <anthony.l.nguyen@intel.com>' \
  patches/

# Submit a v2 threaded under the original series
# (get the Message-ID from your first submission in the archive)
git send-email \
  --to=netdev@vger.kernel.org \
  --in-reply-to='<20260601123456.12345-1-jane@example.com>' \
  patches-v2/

# The changelog section (goes after '---' in cover letter or patch body)
# v2:
#   - Fixed sparse warning pointed out by reviewer (patch 1)
#   - Added missing unlock in error path (patch 2)
#   - Rebased onto v6.9-rc5

The Linux kernel contribution workflow is a tree of maintainers, each responsible for a subsystem, with Linus at the root. A patch flows upward through this hierarchy:

1. You write a patch, format it with git format-patch, run checkpatch.pl, identify maintainers with get_maintainer.pl, and send it via git send-email to the subsystem mailing list.
2. Community review happens in public on the mailing list. Anyone can comment. Maintainers (and senior contributors) may request changes, request tests, or give Reviewed-by/Acked-by tags.
3. The subsystem maintainer (e.g., David Miller for networking) pulls the patch into their subsystem tree (usually a branch of their fork of the kernel). They run further tests.
4. linux-next is an integration tree that combines all subsystem trees daily. Your patch will appear here typically within a week of being accepted by a maintainer. Build bots run across architectures; if something breaks, you will hear about it.
5. Merge window: at the start of each kernel cycle (right after the previous stable release), Linus opens a two-week window during which he pulls from subsystem maintainers. Your patch must be in a maintainer's tree before the merge window to make the cycle.
6. Mainline: your patch is in linux/master. It will appear in the next -rc kernels and eventually in a stable release.

This sounds slow, but for a bug fix, it can be days from submission to linux-next. For a new feature in a well-maintained subsystem, a few weeks is typical.

Key tools and resources:

• lore.kernel.org: the canonical mailing list archive; search for your thread here
• patchwork.kernel.org: tracks patch states (new, accepted, rejected, superseded)
• scripts/checkpatch.pl: mandatory style checker — a patch with checkpatch warnings will be rejected
• scripts/get_maintainer.pl: outputs the correct To/Cc list for any file
• The kernel's submitting-patches guide: Documentation/process/submitting-patches.rst

# The full workflow, step by step:

# 1. Set up a tracking branch from the subsystem maintainer's tree
git remote add netdev https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git
git fetch netdev
git switch -c fix/e1000-null-deref --track netdev/main

# 2. Write your fix
vim drivers/net/ethernet/intel/e1000/e1000_main.c
git add -p drivers/net/ethernet/intel/e1000/e1000_main.c
git commit  # Write a proper commit message!

# 3. Validate
scripts/checkpatch.pl --strict -g HEAD
# Must output: total: 0 errors, 0 warnings, N lines checked

# 4. Find maintainers
scripts/get_maintainer.pl -f drivers/net/ethernet/intel/e1000/e1000_main.c
# Jeff Kirsher <jeffrey.t.kirsher@intel.com> (maintainer)
# Tony Nguyen <anthony.l.nguyen@intel.com> (supporter)
# netdev@vger.kernel.org (open list)
# linux-kernel@vger.kernel.org (open list)

# 5. Format
git format-patch -1 --base=netdev/main -o patches/

# 6. Send
git send-email \
  --to=netdev@vger.kernel.org \
  --cc='Jeff Kirsher <jeffrey.t.kirsher@intel.com>' \
  patches/

# 7. Track your patch at:
# https://patchwork.kernel.org/project/netdevbpf/list/
# https://lore.kernel.org/netdev/

git bisect is one of the most powerful debugging tools in existence. Given a commit where a bug is present and a commit where it was absent, bisect performs a binary search through the commit DAG to find the exact commit that introduced the bug. The kernel has hundreds of thousands of commits; bisect finds the culprit in O(log N) — typically 15–20 steps even across 50,000 commits.

The workflow: mark a known-good commit (git bisect good v6.8) and a known-bad commit (git bisect bad HEAD). Git checks out the midpoint commit. You build the kernel, test, and tell Git whether it is good or bad. Git bisects again. Repeat until Git identifies the first bad commit.

Automated bisect is the real power: git bisect run <script> automates the entire process. You write a shell script that returns exit code 0 for good, 1 for bad, and 125 for "cannot test this commit" (Git will skip it). Git then runs your script at each midpoint without human intervention. For a kernel regression, your script might compile the kernel, install it in a VM, and run a reproducer — then return the result. Go to sleep; wake up with the culprit commit identified.

The kernel's Documentation/process/bisecting-a-bug.rst is the official guide. The key insight is that bisect is only useful if you have a reliable reproducer — a way to definitively say "yes, the bug is present" or "no, it is not." For performance regressions, flaky tests, or hardware-dependent issues, bisect still works but requires more care.

For kernel work, bisect is often more useful than any static analysis: regression reports from automated test systems (like 0day robot or syzkaller) come with a commit range, and bisect narrows it to one commit. The author of that commit is then the person to contact for a fix.

# Manual bisect session
git bisect start
git bisect bad HEAD                    # current kernel is broken
git bisect good v6.8                   # v6.8 release was fine
# Git checks out the midpoint commit (~25,000 commits ago)
# Build and test, then:
git bisect good    # or: git bisect bad
# Git moves to the next midpoint (binary search)
# ... repeat ~15 times ...
# Git announces: "a1b2c3d4 is the first bad commit"
git bisect reset   # returns to original HEAD

# Automated bisect with a test script
cat > test-regression.sh << 'EOF'
#!/bin/bash
# Build the kernel
make -j$(nproc) 2>/dev/null
if [ $? -ne 0 ]; then
    exit 125  # skip: build failed (not a good/bad test)
fi

# Install and test in QEMU (simplified)
./scripts/run-qemu-test.sh my-reproducer.c
if [ $? -eq 0 ]; then
    exit 0  # good: bug not present
else
    exit 1  # bad: bug present
fi
EOF
chmod +x test-regression.sh

git bisect start
git bisect bad HEAD
git bisect good v6.8
git bisect run ./test-regression.sh
# Git will run the script automatically at each bisect point
# and announce the first bad commit when done
git bisect log    # full transcript of the bisect session
git bisect reset  # always reset when done