The Shell & Command-Line Mastery

A shell is a program whose job is to read commands from you, run other programs in response to those commands, and report the results. Simple in concept; the mechanics reveal the entire Unix process model.

When you type ls -l, the shell does exactly three things:

1. fork() — it calls the fork system call, which creates an almost-identical copy of the shell process. This copy is called the child process. At this instant, two processes exist: the parent shell and the child shell.
2. exec() — the child calls execve("/bin/ls", ["ls", "-l"], envp). This replaces the child's entire memory image with the ls program. The child is now ls, not a shell. The PID (process ID) is unchanged.
3. wait() — the parent shell calls wait() to block until the child (now ls) exits. When ls finishes and exits, the kernel wakes up the parent, which then prints a new prompt and waits for your next command.

This fork-exec-wait cycle is the heartbeat of every interactive shell session. It explains several things that surprise beginners:

• Why cd cannot be an external program: if the shell fork+exec'd a cd binary, that child would change directory and then exit, leaving the parent shell in the original directory. cd must be a builtin that runs inside the shell process itself.
• Why environment variables set in a child do not propagate back to the parent. The child is a copy; changes to copies do not flow back.
• Why exit in a script terminates the whole script: the script runs in a child shell, and exit calls _exit() on that child.

You can watch this in real time: open two terminals and run watch -n 0.5 'ps --ppid $$ -o pid,comm' in one. Every command you run in the other terminal will briefly appear as a child of the shell.

/* Minimal shell in ~30 lines of C — see what Bash does under the hood */
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>
#include <string.h>

int main(void) {
    char line[256];
    while (1) {
        printf("sh> ");
        fflush(stdout);
        if (!fgets(line, sizeof line, stdin)) break;
        line[strcspn(line, "\n")] = '\0';  /* strip newline */

        /* tokenize (no pipes, quotes, etc. — bare minimum) */
        char *argv[64]; int argc = 0;
        char *tok = strtok(line, " ");
        while (tok) { argv[argc++] = tok; tok = strtok(NULL, " "); }
        argv[argc] = NULL;
        if (argc == 0) continue;

        /* built-in: cd */
        if (strcmp(argv[0], "cd") == 0) {
            if (chdir(argv[1] ? argv[1] : getenv("HOME")) < 0)
                perror("cd");
            continue;
        }

        pid_t pid = fork();
        if (pid == 0) {              /* child */
            execvp(argv[0], argv);   /* search PATH */
            perror(argv[0]);         /* only reached on error */
            _exit(127);
        } else {                     /* parent */
            int status;
            waitpid(pid, &status, 0);
        }
    }
    return 0;
}

Every process inherits three open file descriptors (FDs) from its parent:

• FD 0 — standard input (stdin): where the process reads input
• FD 1 — standard output (stdout): where normal output goes
• FD 2 — standard error (stderr): where error messages go

A file descriptor is just a small integer — an index into a per-process table of open files managed by the kernel. Files, pipes, sockets, devices, and terminals all look the same through this uniform interface. This uniformity is the Unix superpower.

Redirection tells the shell to replace one of those FDs before exec-ing the child. When you write ls -l > files.txt, the shell:

1. Opens files.txt for writing, getting FD 3.
2. Calls dup2(3, 1) — this makes FD 1 point to the same file as FD 3.
3. Closes FD 3 (now redundant).
4. execs ls -l.

ls never knows the difference — it writes to FD 1 as always. The kernel silently routes those bytes to the file.

Redirection operators:

• > — redirect stdout (truncate/create)
• >> — redirect stdout (append)
• < — redirect stdin
• 2> — redirect stderr
• 2>&1 — redirect stderr to wherever stdout currently points
• &> or >file 2>&1 — redirect both stdout and stderr

Pipes connect two processes in real time. ls | grep foo creates a kernel pipe object (a small in-memory ring buffer with a read end and a write end). The shell sets up ls's FD 1 to write into the pipe's write end, and grep's FD 0 to read from the pipe's read end. Both processes run concurrently — ls writes chunks and grep reads them as they arrive. The kernel wakes and sleeps each process as the pipe buffer fills and drains.

This is live IPC (inter-process communication) mediated by the kernel, happening every time you type a pipeline.

# Everything here uses redirection and pipes

# Write stdout to file; stderr still goes to terminal
ls /nonexistent > out.txt        # out.txt is empty; error on terminal

# Capture stderr too
ls /nonexistent > out.txt 2>&1   # IMPORTANT: order matters!

# Wrong order — stderr goes to original stdout, not the file:
ls /nonexistent 2>&1 > out.txt

# Discard all output
some-noisy-command > /dev/null 2>&1

# Pipe: stdout of find → stdin of xargs → stdin of grep
find . -name '*.c' | xargs grep -l 'kmalloc'

# Process substitution: treat command output as a file
diff <(sort file1) <(sort file2)

# Here-document: feed multiline string as stdin
cat <<'EOF'
line one
line two
EOF

When you type ls *.c, the shell — not ls — expands *.c into a list of matching filenames before calling ls. This is called globbing or pathname expansion. The program never sees the glob pattern; it sees a pre-expanded list of actual filenames.

Glob patterns:

• * — matches any string of characters (but not a leading ., and not /)
• ? — matches exactly one character
• [abc] — matches one character that is a, b, or c
• [a-z] — matches one character in the range
• [!abc] or [^abc] — matches any character NOT in the set
• ** — (Bash 4+ with shopt -s globstar) matches any path including /, enabling recursive globbing

Critical insight: if a glob matches nothing, Bash by default leaves the literal pattern in the argument list. This is a famous footgun. rm *.tmp when there are no .tmp files might become rm '*.tmp' — trying to delete a file literally named *.tmp. Use nullglob to make unmatched globs expand to nothing, or failglob to make them an error.

Globbing is purely filesystem-based — the shell reads the directory and matches names. It is different from regular expressions, which operate on arbitrary text. grep '*.c' does NOT grep for C files — * means 'zero or more of the preceding character' in regex. Confusing these two is one of the most common beginner errors.

Brace expansion {a,b,c} is a related but distinct mechanism that does NOT touch the filesystem — it generates combinations lexically: echo {foo,bar}.{c,h} produces foo.c foo.h bar.c bar.h regardless of whether any of those files exist.

# Enable globstar for recursive matching (Bash 4+)
shopt -s globstar nullglob

# Find all .h files anywhere under current directory
for f in **/*.h; do
    echo "$f"
done

# Match files like main_v1.c, main_v2.c but not main_vX.c
ls main_v[0-9].c

# Brace expansion — no filesystem access
touch file_{a,b,c}.txt   # creates file_a.txt file_b.txt file_c.txt
mkdir -p src/{drivers,fs,mm,net}  # create kernel-like directory tree

# Safe glob: if nothing matches, loop body doesn't run
shopt -s nullglob
for f in /tmp/stale_*.lock; do
    rm -- "$f"
done

Shell variables store strings (the shell has no native numeric type beyond arithmetic expansion). Assignment has no spaces around =:

name=Linus          # correct
name = Linus        # WRONG — shell tries to run a command named 'name'

To use a variable, prefix with $. Always double-quote variable expansions to prevent word splitting and globbing:

file="my file.txt"
cat $file       # BROKEN: shell word-splits into 'cat my file.txt' (two args)
cat "$file"     # CORRECT: single argument with the space

This is the most important quoting rule: double-quote every $variable unless you specifically want word-splitting.

Parameter expansion goes far beyond simple substitution. Some invaluable forms:

• ${var:-default} — use default if var is unset or empty
• ${var:?error message} — abort with message if var is unset
• ${#var} — length of the string
• ${var#prefix} — remove shortest prefix match
• ${var##prefix} — remove longest prefix match (greedy)
• ${var%suffix} — remove shortest suffix
• ${var%%suffix} — remove longest suffix
• ${var/pattern/replacement} — replace first match
• ${var//pattern/replacement} — replace all matches

Arrays (Bash extension):

arr=(alpha beta gamma)
echo "${arr[1]}"        # beta
echo "${arr[@]}"        # all elements (properly quoted)
echo "${#arr[@]}"       # number of elements

Arithmetic expansion: $(( expr )) evaluates integer arithmetic. $((2**10)) gives 1024.

Command substitution: $(command) runs command and inserts its stdout. The older backtick form `command` is equivalent but cannot nest cleanly — prefer $().

#!/usr/bin/env bash
set -euo pipefail   # exit on error, undefined vars, pipe failures

# Parameter expansion cookbook
path="/home/karthik/kernel/drivers/usb/core.c"

base="${path##*/}"        # core.c        (strip longest prefix up to /)
dir="${path%/*}"          # /home/karthik/kernel/drivers/usb
noext="${base%.c}"        # core          (strip .c suffix)

echo "file:   $base"
echo "dir:    $dir"
echo "stem:   $noext"

# Default values
BUILD_DIR="${BUILD_DIR:-/tmp/build}"

# Mandatory variable check
: "${KERNEL_SRC:?KERNEL_SRC must be set}"

# Array iteration — always quote ${arr[@]}
modules=(ext4 btrfs xfs tmpfs)
for mod in "${modules[@]}"; do
    echo "modprobe $mod"
done

# Arithmetic
total=$(( 1024 * 1024 * 4 ))   # 4 MiB in bytes
echo "4 MiB = $total bytes"

Every process has an environment — a set of key=value string pairs passed to it at birth. In C, main(int argc, char *argv[], char *envp[]) exposes this as a third argument, or you access it via getenv(). The shell lets you view and modify the environment interactively.

export marks a shell variable as part of the environment, so child processes inherit it:

FOO=bar          # shell variable, not in environment
export FOO       # now in environment — children see it
export BAR=baz   # shorthand

Without export, a variable is local to the current shell and invisible to programs you run.

PATH is the most important environment variable. It is a colon-separated list of directories. When you type gcc, the shell searches each directory in PATH from left to right, looking for an executable named gcc. First match wins. This is why which gcc tells you which one you get, and why prepending a directory to PATH (e.g., export PATH="$HOME/bin:$PATH") lets you override system commands.

Important environment variables to know:

• HOME — your home directory
• USER / LOGNAME — your username
• SHELL — path to current shell
• TERM — terminal type (used by ncurses programs)
• LANG / LC_ALL — locale settings (affect sort order, date formats)
• PS1 — your prompt string
• IFS — Internal Field Separator (default: space, tab, newline — controls word splitting)
• OLDPWD — previous directory (cd - uses this)

env with no arguments prints the full environment. env -i command runs a command with an empty environment — great for reproducibility testing. printenv VAR prints one variable.

A subtlety: when you ssh to a remote machine, only certain variables are forwarded (controlled by SendEnv in ~/.ssh/config). When you sudo, the environment is typically sanitized. Knowing this saves hours of debugging.

# See the full environment
env | sort

# Add ~/bin to PATH permanently (put in ~/.bashrc)
export PATH="$HOME/bin:$PATH"

# Run one command with a modified environment (doesn't affect your shell)
HTTPS_PROXY=proxy.example.com:3128 curl https://kernel.org

# env -i: see exactly what a fresh process gets
env -i PATH=/usr/bin:/bin bash --norc --noprofile -c 'echo $PATH; which ls'

# Inspect what a child actually inherits (requires /proc)
bash -c 'cat /proc/$$/environ' | tr '\0' '\n' | grep PATH

# IFS footgun: changing IFS affects word splitting everywhere
IFS=':'                            # now 'a:b:c' splits into three words
read -ra parts <<< "one:two:three"
echo "${parts[1]}"                 # two

The shell manages jobs — one or more processes that form a pipeline. At any time, one job is in the foreground (receiving your keyboard input) and zero or more are in the background.

Key operations:

• command & — run command in background; shell prints [1] 12345 (job number, PID)
• Ctrl-Z — suspend the foreground job (sends SIGTSTP)
• bg — resume a suspended job in the background
• fg — bring a background job to the foreground
• fg %2 — bring job number 2 to the foreground
• jobs — list all current jobs
• kill %1 — send SIGTERM to job 1
• wait — wait for all background jobs to finish

Signals are asynchronous notifications sent to processes. The shell translates keyboard shortcuts into signals:

• Ctrl-C → SIGINT (interrupt) — default action: terminate
• Ctrl-Z → SIGTSTP (terminal stop) — default action: suspend
• Ctrl-\ → SIGQUIT — default action: terminate with core dump
• kill -9 PID → SIGKILL — cannot be caught or ignored; kernel kills the process
• kill -15 PID → SIGTERM — default: terminate (can be caught for cleanup)

In scripts, trap lets you run cleanup code when a signal arrives:

trap 'rm -f /tmp/mylock; exit 1' INT TERM EXIT

Process groups and sessions: every process belongs to a process group. Sending a signal to a process group (negative PID: kill -INT -12345) signals all members. When you log out or close a terminal, SIGHUP is sent to the entire session's foreground process group. nohup command & makes the command ignore SIGHUP; disown %1 removes a job from the shell's job table (so SIGHUP is not forwarded when the shell exits).

screen/tmux solve the SIGHUP problem permanently: they maintain a persistent session that outlives your SSH connection.

#!/usr/bin/env bash
# Demonstrate job control and trap

TMPFILE=$(mktemp)

# Cleanup on exit OR interrupt
cleanup() {
    echo "Cleaning up $TMPFILE"
    rm -f "$TMPFILE"
}
trap cleanup EXIT INT TERM

echo "Working..." > "$TMPFILE"

# Start a background job
sleep 30 &
BG_PID=$!
echo "Background sleep PID: $BG_PID"

jobs   # shows [1]+ Running  sleep 30

# Wait for it (or Ctrl-C to trigger trap and cleanup)
wait "$BG_PID"
echo "Done"

A shell script is simply a text file containing shell commands, with #!/usr/bin/env bash as the first line (the shebang). Make it executable with chmod +x script.sh and run it with ./script.sh.

Always start scripts with:

#!/usr/bin/env bash
set -euo pipefail

• -e: exit on any command failure (nonzero exit status)
• -u: treat unset variables as errors
• -o pipefail: a pipeline fails if ANY command in it fails (not just the last)

Without these, scripts silently continue after errors. This is how real production bugs hide in shell scripts.

Exit status: every command returns a number 0-255. 0 = success, nonzero = failure. $? holds the last exit status. if, while, &&, || all work on exit status.

Conditionals: [[ ]] (Bash double-bracket) is preferred over [ ] (POSIX test) — it handles spaces in variables safely and supports regex with =~.

Loops:

for i in $(seq 1 10); do ...; done
while IFS= read -r line; do ...; done < file.txt   # correct way to read a file

Functions are first-class:

my_func() {
    local x="$1"   # 'local' is crucial — prevents variable leakage
    echo "arg: $x"
    return 0       # explicit return status
}
my_func "hello"

Without local, variables set inside functions pollute the global scope — a common, painful bug.

Script arguments: $1, $2, ... are positional parameters. $0 is the script name. $# is the count of arguments. "$@" is all arguments, properly quoted (always use this, not $*). shift consumes $1 and renumbers the rest.

#!/usr/bin/env bash
set -euo pipefail

# --- utility functions ---
die() { echo "ERROR: $*" >&2; exit 1; }
log() { echo "[$(date +%T)] $*"; }

# --- argument parsing ---
usage() { echo "Usage: $0 [-v] <kernel-src-dir>"; exit 1; }
VERBOSE=0
while getopts ':v' opt; do
    case $opt in
        v) VERBOSE=1 ;;
        *) usage ;;
    esac
done
shift $(( OPTIND - 1 ))
[[ $# -ge 1 ]] || usage

KERNEL_SRC="$1"
[[ -d "$KERNEL_SRC" ]] || die "Not a directory: $KERNEL_SRC"

# --- main logic ---
count_c_files() {
    local dir="$1"
    local count
    count=$(find "$dir" -name '*.c' | wc -l)
    echo "$count"
}

log "Scanning $KERNEL_SRC"
total=$(count_c_files "$KERNEL_SRC")
log "Found $total C source files"

[[ $VERBOSE -eq 1 ]] && find "$KERNEL_SRC" -name '*.c' -print

log "Done"

The Unix tradition is to have programs that do one thing well and communicate through text streams. Four tools form the backbone of text processing in the shell:

grep — search for lines matching a pattern

• grep pattern file — print lines matching regex
• -r — recursive (or use rg/ag for speed)
• -l — print filenames only
• -n — show line numbers
• -v — invert match (lines NOT matching)
• -E — extended regex (alternation |, +, ?)
• -F — fixed string, no regex (much faster)
• -c — count matching lines
• --include='*.c' — filter by filename pattern

Kernel developers use git grep inside the kernel tree — it respects .gitignore and is significantly faster.

sed — stream editor: transform text line by line

• sed 's/old/new/g' file — global substitution (the workhorse)
• sed -n '5,10p' — print only lines 5–10
• sed '/pattern/d' — delete matching lines
• sed -i 's/foo/bar/g' file — edit file in place (add backup: -i.bak)
• sed 's/^/ /' — indent every line by 4 spaces

awk — line-oriented pattern-action language Awk treats each line as a record and splits it into fields $1, $2, ..., $NF (last field). NR is the current line number. Awk is a full programming language — it has variables, arrays, arithmetic, and printf.

• awk '{print $2}' file — print second field
• awk -F: '{print $1}' /etc/passwd — set field separator to :
• awk '/pattern/ {sum += $3} END {print sum}' — sum column 3 for matching lines
• awk 'NR==5' — print line 5

find — search for files by attribute (name, type, size, time, permissions...)

• find . -name '*.c' — by name
• find . -type f -newer Makefile — files newer than Makefile
• find . -name '*.o' -delete — find and delete
• find . -name '*.c' -exec grep -l 'kmalloc' {} +' — find C files containing kmalloc
• -maxdepth 2 — don't recurse too deep
• -perm /u+x — files with execute bit

A key -exec idiom: {} is replaced by the filename; + batches many filenames into one grep call (efficient), while \; runs one invocation per file (sometimes necessary).

# Real-world text processing tasks

# 1. Find all functions that call kmalloc in kernel source
grep -rn 'kmalloc(' --include='*.c' linux/mm/ | head -20

# 2. Extract all #include'd headers from a C file
grep '^#include' main.c | sed 's/#include //; s/[<>"]//g'

# 3. Count syscall implementations in kernel (files named sys_*.c)
find linux/ -name 'sys_*.c' | wc -l

# 4. Print the 3rd column of /proc/meminfo (the unit)
awk '{print $3}' /proc/meminfo | sort -u

# 5. Sum all anonymous mapping sizes from /proc/$$/maps
awk '$5 == 0 {sum += strtonum("0x" $2) - strtonum("0x" substr($1,1,12))}
     END {printf "anon: %d MB\n", sum/1024/1024}' /proc/$$/maps

# 6. Rename all .txt files to .md (no external tool needed)
for f in *.txt; do
    mv -- "$f" "${f%.txt}.md"
done

# 7. Find C files > 1000 lines (potential complexity hotspots)
find . -name '*.c' -exec awk 'END{if(NR>1000)print NR, FILENAME}' {} \;

In 1978, Doug McIlroy articulated what became the Unix Philosophy in the Bell System Technical Journal:

Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.

This is not just a historical footnote — it is the architectural principle that explains why the Unix toolset is still useful 50 years later, and why you can compose tools that were written before you were born to solve problems their authors never imagined.

Do one thing well: wc counts lines, words, and bytes. It does not sort, filter, or format. This constraint is a feature — it makes wc reliable, fast, and auditable. When a tool does one thing, it is easy to understand, easy to test, and easy to combine.

Work together: The universal interface is bytes on a stream. grep does not care if its input came from a file, a network socket, another program, or a keyboard. Everything that can emit bytes on stdout and read bytes from stdin can be composed. This is why you can curl URL | jq '.items[]' | grep active | wc -l — four programs from four different authors, written in different decades, working together seamlessly.

Text streams: Text is the universal interchange format in Unix. It is human-readable, debuggable, and processable by every tool. Compare this to binary formats that require special decoders just to inspect.

The practical consequences:

• A complex task is decomposed into a pipeline of simple stages
• You can insert tee anywhere in a pipeline to inspect intermediate results without changing the flow
• You can replace any stage with a better implementation without changing the others
• Each stage can be independently tested

For kernel development specifically: the kernel's development process — email-based patches, git log --oneline, checkpatch.pl, scripts/get_maintainer.pl — is built entirely on this philosophy. Understanding it is not optional.

# The Unix philosophy in action: a realistic kernel workflow

# Question: which subsystems have the most commits in the last month?
git log --since='1 month ago' --format='%s' |
    grep -oP '^[a-z_/]+(?=:)' |   # extract 'subsystem:' prefix
    sort |                          # sort for uniq
    uniq -c |                       # count occurrences
    sort -rn |                      # sort numerically, descending
    head -10                        # top 10

# Each tool does ONE thing:
# git log   — enumerate commits
# grep -oP  — extract the subsystem label
# sort      — sort lines (required by uniq)
# uniq -c   — count consecutive duplicates
# sort -rn  — sort numerically
# head      — show top N

# Insert 'tee /tmp/debug.txt' anywhere to inspect intermediate output
# Replace 'git log' with 'cat saved-log.txt' to work offline
# The pipeline is modular, debuggable, and composable