Testing & Test-Driven Development
Testing is not just checking if code works — it is a formal discipline for specifying behavior, preventing regression, and building systems you can trust to change.
Overview
Testing & Test-Driven Development
Every programmer knows the gut-drop feeling of deploying code that "worked on their machine" — and then watching it fail in production. Testing is the engineering response to that problem: a systematic, repeatable discipline for specifying what your code should do, verifying that it does it, and protecting that guarantee as the codebase evolves.
But testing is far more than a sanity check. At its deepest, a test suite is a formal specification rendered in executable form. When you write a test before the code, you are forced to think precisely about behavior, edge cases, and contracts — before any implementation details can distract you. This is the insight behind Test-Driven Development (TDD), a methodology that emerged from Extreme Programming and is now practiced in every serious software shop from Google to SpaceX.
This module covers the full arc: from the micro-level discipline of writing a single unit test, through the architecture of the testing pyramid, to the theory of code coverage and property-based testing. We will also study mocking — how to isolate a component from the rest of the world — and integration and system testing, which verify that components work together as designed. Along the way, we will see how these ideas connect to formal methods, type theory, and the science of software reliability.
By the end, you will be able to design and implement a professional-grade test suite for a real system, justify every testing decision with principled reasoning, and understand why Google, Amazon, and every other major software organization treat testing infrastructure as a first-class engineering concern.
Software defects have caused planes to crash (Boeing MCAS), rockets to explode (Ariane 5 integer overflow), hospitals to overdose patients (Therac-25 radiation therapy), and stock markets to collapse (Knight Capital's $440 million loss in 45 minutes). These are not edge-case failures of exotic systems — they are failures of ordinary software that lacked sufficient testing. The cost of finding a bug in production is estimated to be 10–100x higher than finding it during development; at the system level, it can be literally incalculable.
At the software engineering level, tests are what make large-scale collaboration possible. When Google engineers make thousands of changes per day to a codebase with billions of lines of code, it is the automated test suite — running in CI/CD pipelines — that gives them the confidence to merge. Without that, no team larger than a handful of people can maintain velocity. Testing is therefore not just a craft skill but a systems-design concern: the architecture of your test suite determines the architecture of your ability to change your system safely over time.
The Testing Pyramid and CI Pipeline
The CI pipeline enforces the testing pyramid: fast unit tests gate the slower integration and E2E tests, so failures are caught as cheaply as possible. A failure at any level blocks the merge.
The concepts
01What a Test Actually Is: Assertions, Oracles, and the Testing Problem
A test is a program that exercises a unit of code under specific conditions and checks that the result matches an expected outcome. The expected outcome is called the oracle — the ground truth against which actual behavior is compared.
The testing problem is fundamentally unsolvable in the general case. Dijkstra's famous remark applies: testing can show the presence of bugs, but never their absence. For a function f: A → B, complete verification would require trying all inputs in A — which is infinite (or astronomically large) for any realistic type. Testing is therefore a sampling strategy: we choose inputs carefully to maximize coverage of behaviors, not inputs.
Every test has three phases, called Arrange–Act–Assert (AAA):
- Arrange: Set up the system state and inputs
- Act: Invoke the code under test
- Assert: Verify the output or side effect matches the oracle
A test that never fails is worthless. A good test is deterministic (same result every run), isolated (does not depend on other tests), fast (so it runs in CI), and meaningful (fails only when real behavior breaks).
import pytest
def add(a: int, b: int) -> int:
return a + b
# Arrange-Act-Assert pattern
def test_add_positive_numbers():
# Arrange
x, y = 3, 4
# Act
result = add(x, y)
# Assert
assert result == 7
def test_add_negative_numbers():
assert add(-1, -2) == -3
def test_add_zero_identity():
assert add(5, 0) == 5
assert add(0, 5) == 5
02Unit Testing: Testing in Isolation
A unit test exercises the smallest testable piece of code — a single function, method, or class — in complete isolation from its dependencies. The goal is to verify that the unit satisfies its contract: given certain inputs and preconditions, it produces the right outputs and side effects.
Why isolation matters: If your unit test for a shopping cart calls the real database, the real payment API, and the real email server, then a test failure could mean any of those things broke — not necessarily your cart logic. Isolation makes failures local and diagnostic.
Unit tests follow the F.I.R.S.T. principles:
- Fast: Run in milliseconds, thousands per second
- Independent: No shared state between tests
- Repeatable: Same result on any machine, any time
- Self-validating: Pass or fail — no human inspection needed
- Timely: Written before or alongside the code (TDD)
Good unit tests also act as living documentation. A reader of test_password_rejected_if_too_short understands the rule instantly, without reading the implementation.
Python's pytest, Java's JUnit, and JavaScript's Jest are the canonical frameworks. They discover test functions by naming convention, run them, and report failures with informative diffs.
# bank_account.py
class InsufficientFundsError(Exception):
pass
class BankAccount:
def __init__(self, balance: float = 0.0):
self._balance = balance
def deposit(self, amount: float) -> None:
if amount <= 0:
raise ValueError("Deposit must be positive")
self._balance += amount
def withdraw(self, amount: float) -> None:
if amount > self._balance:
raise InsufficientFundsError()
self._balance -= amount
@property
def balance(self) -> float:
return self._balance
# test_bank_account.py
import pytest
from bank_account import BankAccount, InsufficientFundsError
def test_initial_balance_is_zero():
acct = BankAccount()
assert acct.balance == 0.0
def test_deposit_increases_balance():
acct = BankAccount()
acct.deposit(100.0)
assert acct.balance == 100.0
def test_withdraw_decreases_balance():
acct = BankAccount(balance=200.0)
acct.withdraw(50.0)
assert acct.balance == 150.0
def test_withdraw_raises_on_overdraft():
acct = BankAccount(balance=10.0)
with pytest.raises(InsufficientFundsError):
acct.withdraw(100.0)
def test_deposit_rejects_nonpositive():
acct = BankAccount()
with pytest.raises(ValueError):
acct.deposit(-5.0)
03Test-Driven Development (TDD): Red–Green–Refactor
Test-Driven Development (TDD) inverts the usual order: you write the test before you write the implementation. The cycle is:
- Red: Write a failing test for the next small piece of behavior. It must fail because the feature doesn't exist yet. If it passes immediately, you wrote a bad test.
- Green: Write the minimum code needed to make the test pass. No more. Ugly code is fine here.
- Refactor: Clean up the code — extract functions, rename, simplify — while keeping all tests green.
This loop runs in minutes, not hours. Kent Beck, who popularized TDD in Extreme Programming Explained (2000), called it a way to "manage fear" — the fear of breaking things, the fear of the blank page.
Why does writing tests first produce better designs? Because writing a test forces you to use your API before you build it. If your test is awkward to write, your API is awkward to use. TDD gives you instant design feedback. Systems built TDD tend to have lower coupling and higher cohesion because tightly coupled components are hard to test in isolation.
A common misconception: TDD is not about 100% coverage or testing every line. It's about driving design and preventing regression. The tests are a by-product; the design improvement is the point.
Example workflow: Building a Roman numeral converter TDD-style:
- Test:
roman(1) == 'I'→ fails → implement → passes → refactor - Test:
roman(4) == 'IV'→ fails → implement → passes → refactor - ... and so on, edge cases emerging naturally
# TDD Example: Building a stack from scratch
# STEP 1 — RED: Write failing test first
# test_stack.py
from stack import Stack
def test_new_stack_is_empty():
s = Stack()
assert s.is_empty()
def test_push_makes_nonempty():
s = Stack()
s.push(42)
assert not s.is_empty()
def test_pop_returns_last_pushed():
s = Stack()
s.push(1)
s.push(2)
assert s.pop() == 2
def test_pop_on_empty_raises():
s = Stack()
import pytest
with pytest.raises(IndexError):
s.pop()
# STEP 2 — GREEN: Minimum implementation
# stack.py
class Stack:
def __init__(self):
self._items = []
def push(self, item):
self._items.append(item)
def pop(self):
if self.is_empty():
raise IndexError("pop from empty stack")
return self._items.pop()
def is_empty(self):
return len(self._items) == 0
# STEP 3 — REFACTOR: All tests still green,
# now clean up, add type hints, docstrings, etc.
04Mocking and Test Doubles: Replacing the World
When a unit depends on external systems — databases, HTTP APIs, clocks, random number generators — we replace those dependencies with test doubles that simulate their behavior without the real cost or nondeterminism.
There are five kinds of test doubles (Gerard Meszaros' taxonomy):
- Dummy: Objects passed but never used (fill parameter lists)
- Stub: Returns fixed pre-programmed values; does not verify calls
- Spy: Records calls so you can assert they happened correctly
- Mock: Pre-programmed with expectations about calls it must receive; fails if not met
- Fake: A working simplified implementation (e.g., an in-memory database)
In Python, unittest.mock.patch replaces a real object with a MagicMock for the duration of a test. In Java, Mockito is the canonical library.
When to mock: Mock things that are slow (network, disk), nondeterministic (random, time), have side effects (email, billing), or are third-party dependencies you don't control. Do not mock: the code under test itself, or value objects that are cheap to create.
A critical design insight: if a function is hard to mock, it usually has too many responsibilities or is poorly factored. Testability pressure reveals design problems.
# user_service.py
class UserService:
def __init__(self, db, email_sender):
self.db = db
self.email_sender = email_sender
def register(self, username: str, email: str) -> dict:
if self.db.user_exists(email):
raise ValueError("Email already registered")
user = self.db.create_user(username, email)
self.email_sender.send_welcome(email)
return user
# test_user_service.py
from unittest.mock import MagicMock
from user_service import UserService
def test_register_creates_user_and_sends_email():
# Arrange: inject mock dependencies
mock_db = MagicMock()
mock_db.user_exists.return_value = False
mock_db.create_user.return_value = {"id": 1, "username": "alice"}
mock_email = MagicMock()
svc = UserService(db=mock_db, email_sender=mock_email)
# Act
result = svc.register("alice", "alice@example.com")
# Assert behavior
assert result["username"] == "alice"
mock_db.create_user.assert_called_once_with("alice", "alice@example.com")
mock_email.send_welcome.assert_called_once_with("alice@example.com")
def test_register_raises_on_duplicate_email():
mock_db = MagicMock()
mock_db.user_exists.return_value = True # email taken
mock_email = MagicMock()
svc = UserService(db=mock_db, email_sender=mock_email)
import pytest
with pytest.raises(ValueError):
svc.register("alice", "alice@example.com")
# Welcome email must NOT be sent on failure
mock_email.send_welcome.assert_not_called()
05The Testing Pyramid: A Strategy for Test Suite Architecture
The testing pyramid (Mike Cohn, 2009) is a mental model for how to allocate testing effort across the three main levels:
/\
/ \ System / E2E tests (few, slow, expensive)
/----\
/ \ Integration tests (medium)
/--------\
/ \ Unit tests (many, fast, cheap)
/____________\
Unit tests (base): Fast (< 1ms each), isolated, many hundreds or thousands. Run on every save. Cover all business logic edge cases.
Integration tests (middle): Test that two or more components work together — your service layer + real database, your HTTP handler + real router, your parser + real file I/O. Slower (seconds), fewer, run in CI.
System / End-to-End (E2E) tests (apex): Test the entire deployed application through its real interface (browser, HTTP, CLI). Slow, brittle, expensive to write and maintain. A dozen to a few hundred. Run before release.
The pyramid shape reflects a fundamental economics: the higher you go, the more expensive a test is to write, run, and debug. If you flip the pyramid (many E2E, few unit tests), your CI pipeline takes 2 hours, failures are hard to diagnose, and developers stop running tests.
The ice-cream cone antipattern: No unit tests, a few integration tests, and relying entirely on manual QA. This is what most non-engineering organizations accidentally build, and it is catastrophically expensive.
Modern variants include the testing trophy (Kent C. Dodds) for frontend, which emphasizes integration tests over unit tests because UI components are hard to unit-test meaningfully in isolation.
# Integration test example: testing a real DB interaction
# Uses pytest-postgresql fixture for a real (temporary) Postgres instance
import pytest
from myapp.db import Database
from myapp.user_repo import UserRepository
@pytest.fixture
def db(postgresql): # pytest-postgresql spins up a real instance
database = Database(postgresql.dsn())
database.run_migrations()
yield database
database.drop_all()
def test_create_and_fetch_user(db):
repo = UserRepository(db)
repo.create(username="bob", email="bob@test.com")
user = repo.find_by_email("bob@test.com")
assert user.username == "bob"
def test_duplicate_email_raises(db):
repo = UserRepository(db)
repo.create(username="bob", email="bob@test.com")
with pytest.raises(Exception): # DB unique constraint
repo.create(username="bob2", email="bob@test.com")
06Code Coverage: What It Measures and What It Doesn't
Code coverage is a metric that measures which lines, branches, or paths of your code were executed during the test suite. It is the most widely used proxy for test quality — and the most widely misunderstood.
The four main coverage criteria, ordered by strength:
- Statement coverage: Every line executed at least once
- Branch coverage: Every branch (true/false of each
if) taken at least once - Condition coverage: Every boolean sub-expression evaluated both true and false
- Path coverage: Every distinct control-flow path through the function
Path coverage is theoretically complete but exponentially expensive (a function with 10 independent if statements has 2^10 = 1024 paths).
In practice, 80–90% line/branch coverage is a common goal. 100% coverage is achievable and valuable for critical systems (aerospace, medical devices) but can create perverse incentives: developers write tests that merely execute code rather than verify its behavior.
The crucial insight: Coverage measures which code is executed, not whether the results are correct. You can achieve 100% coverage with tests that have no assertions. Coverage is a necessary but not sufficient condition for good testing.
Mutation testing (next concept) is a stronger quality signal.
# Example showing branch coverage gap
def discount(price: float, is_member: bool, quantity: int) -> float:
"""Apply tiered discount."""
d = 0.0
if is_member: # branch 1
d += 0.10
if quantity >= 10: # branch 2
d += 0.05
return price * (1 - d)
# These two tests achieve 100% STATEMENT coverage
def test_member_bulk():
assert discount(100, True, 10) == 85.0 # both branches true
def test_nonmember_small():
assert discount(100, False, 1) == 100.0 # both branches false
# But 100% BRANCH coverage requires 4 tests:
# (True, small), (False, bulk), (True, bulk), (False, small)
# -- the first two tests MISS the (True, small) and (False, bulk) cases!
# Run coverage with:
# pytest --cov=mymodule --cov-report=term-missing
# Output shows lines not hit and branches not taken
07Property-Based Testing: Specify Laws, Not Examples
Traditional testing is example-based: you pick specific inputs and expected outputs by hand. Property-based testing is fundamentally different: you describe invariants or laws that must hold for all (random) inputs, and the framework generates hundreds of test cases automatically.
The idea comes from QuickCheck, invented by John Hughes and Koen Claessen in Haskell (1999), and has since been ported to every major language (Python's hypothesis, Java's jqwik, Scala's ScalaCheck).
The workflow:
- Identify a property your function should satisfy (e.g., "sorting is idempotent: sort(sort(x)) == sort(x)")
- Write the property as a parameterized test
- The framework generates random inputs, finds failing cases, and shrinks them to the smallest counter-example
Shrinking is the killer feature: if hypothesis finds that your function fails on a list of 1000 random integers, it will automatically reduce the input to the minimal failing case (often a 2-element list), making debugging trivial.
Good properties to look for:
- Round-trip:
decode(encode(x)) == x - Idempotence:
f(f(x)) == f(x) - Commutativity:
add(a,b) == add(b,a) - Invariants: Sorted output is always non-decreasing
- Oracle comparison: Compare a fast implementation against a slow but obviously-correct one
Property-based testing is especially powerful for finding edge cases humans never think of: empty strings, zero, None, extremely large numbers, unicode edge cases.
# Python property-based testing with hypothesis
from hypothesis import given, strategies as st
from hypothesis import settings
# System under test: our own sort
def my_sort(lst):
"""Insertion sort."""
lst = list(lst)
for i in range(1, len(lst)):
key = lst[i]
j = i - 1
while j >= 0 and lst[j] > key:
lst[j + 1] = lst[j]
j -= 1
lst[j + 1] = key
return lst
# Property 1: output is sorted
@given(st.lists(st.integers()))
def test_output_is_sorted(lst):
result = my_sort(lst)
for i in range(len(result) - 1):
assert result[i] <= result[i + 1]
# Property 2: same elements (no additions/deletions)
@given(st.lists(st.integers()))
def test_same_elements(lst):
assert sorted(my_sort(lst)) == sorted(lst)
# Property 3: idempotent
@given(st.lists(st.integers()))
def test_idempotent(lst):
assert my_sort(my_sort(lst)) == my_sort(lst)
# Property 4: agrees with Python's built-in sort (oracle)
@given(st.lists(st.integers()))
def test_matches_oracle(lst):
assert my_sort(lst) == sorted(lst)
08Integration and System Testing: Verifying Composition
Integration testing verifies that two or more components work correctly together. Units that pass all unit tests can still fail when composed — because they make incompatible assumptions about data formats, concurrency, error propagation, or resource lifecycle.
Classic integration failure scenarios:
- Service A expects JSON dates as ISO-8601 strings; Service B sends Unix timestamps
- Component A opens a database connection but never closes it; under load, the connection pool exhausts
- Two modules each assume they run single-threaded; together, they create a race condition
System testing (also called end-to-end testing) runs the entire deployed application as a black box — the test interacts only through real external interfaces: HTTP, CLI, browser, message queue. Tools include Selenium/Playwright (browser), Postman/requests (HTTP API), and Cypress (frontend).
Contract testing is a modern alternative for microservices: instead of spinning up all services for integration tests, each service publishes a contract (expected request/response shapes), and both sides verify their contract independently. Pact is the canonical tool.
The key tradeoff: Integration and system tests give high confidence that the full system works, but they are slow (seconds to minutes), brittle (any change anywhere can break them), and hard to debug (failures don't point to a specific line). Use them to verify critical user journeys, not to cover every business logic branch.
# System test: testing a REST API end-to-end with requests
import requests
import pytest
BASE = "http://localhost:8000"
@pytest.fixture(autouse=True)
def reset_db():
"""Reset test DB before each system test."""
requests.post(f"{BASE}/test/reset") # test-only admin endpoint
yield
def test_user_registration_and_login_flow():
# Register a new user
resp = requests.post(f"{BASE}/users", json={
"username": "alice",
"password": "s3cr3t!",
"email": "alice@test.com"
})
assert resp.status_code == 201
user_id = resp.json()["id"]
# Login and receive JWT token
resp = requests.post(f"{BASE}/auth/login", json={
"email": "alice@test.com",
"password": "s3cr3t!"
})
assert resp.status_code == 200
token = resp.json()["token"]
# Use token to access protected resource
resp = requests.get(f"{BASE}/users/{user_id}",
headers={"Authorization": f"Bearer {token}"})
assert resp.status_code == 200
assert resp.json()["username"] == "alice"
def test_protected_route_rejects_unauthenticated():
resp = requests.get(f"{BASE}/users/1")
assert resp.status_code == 401
09Mutation Testing: Are Your Tests Actually Testing Anything?
Mutation testing answers the question: "If I deliberately break my code, do my tests catch it?"
A mutation testing tool (Python's mutmut, Java's PIT, JavaScript's Stryker) automatically creates hundreds of mutants — slightly modified versions of your source code — and runs your test suite against each one. A mutant that causes at least one test to fail is killed. A mutant that passes all tests is a survivor — meaning your test suite has a gap.
Example mutations:
- Replace
+with- - Replace
>with>= - Replace
TruewithFalse - Delete a line
- Negate a return value
The mutation score = killed / total mutants. A mutation score of 90%+ indicates a genuinely strong test suite.
Mutation testing is expensive: running your full test suite once per mutant means O(N × tests) executions. Modern tools use incremental mutation and parallelism to make this tractable.
Why it matters: Coverage only tells you which lines ran. A test with no assertion on the return value achieves coverage but kills zero mutants. Mutation testing forces your tests to be precise about expected behavior.
This is a research-grade technique increasingly used in industry (Google runs mutation testing on their Android codebase).
# Demonstrating why coverage is insufficient — mutation catches what coverage misses
def max_of_two(a, b):
if a > b: # mutant: a >= b
return a
return b
# This test achieves 100% line coverage:
def test_max_coverage_only():
result = max_of_two(5, 3)
# Achieves coverage but doesn't assert anything about equal inputs!
assert result == 5 # Only tests a > b case
# The mutant (a >= b) STILL passes this test!
# Mutation testing would mark this as a SURVIVING mutant.
# This test KILLS the >= mutant:
def test_max_with_equal_inputs():
# a > b mutant returns a; a >= b mutant also returns a.
# We need a case where they differ: a == b
assert max_of_two(4, 4) == 4 # Both return 4 — mutant survives here too
assert max_of_two(3, 5) == 5 # a < b: tests the 'return b' branch
# With these 3 tests together, the >= mutant is killed:
# max_of_two(5,5) with (a>b): returns 5 (b), with (a>=b): returns 5 (a)
# identical — but max_of_two(3,5) kills the mutant if return a had been swapped
10Test Organization, Fixtures, and Parameterization
As a test suite grows, organization becomes critical. Good test organization follows three principles:
1. Mirror the source tree: src/myapp/user.py → tests/test_user.py. This makes it trivial to find tests for any module.
2. Use fixtures for setup/teardown: A fixture is a piece of test infrastructure that is set up before a test and torn down after. Pytest's @pytest.fixture decorator and its yield pattern are the standard; setUp/tearDown in JUnit/unittest are the Java/Python alternatives. Fixtures can be scoped: function (per test), class, module, or session (once per entire test run).
3. Parameterize to avoid repetition: When the same test logic should run with multiple input/output pairs, use parameterization. @pytest.mark.parametrize in pytest, @ParameterizedTest in JUnit 5. This avoids copy-paste tests and makes adding new cases trivial.
Test organization also affects test discoverability and parallel execution. Modern CI systems (GitHub Actions, CircleCI) run tests in parallel across workers; tests that share global mutable state will produce nondeterministic failures (flaky tests), the most demoralizing class of bug in a CI pipeline.
A conftest.py (pytest-specific) is a shared fixture module that pytest auto-discovers — it lets you share fixtures across multiple test files without importing them explicitly.
import pytest
# ---- Fixtures with setup/teardown ----
@pytest.fixture
def temp_file(tmp_path):
"""Provides a real temporary file, cleaned up after each test."""
f = tmp_path / "data.txt"
f.write_text("hello")
yield f # test runs here
# teardown: tmp_path is automatically removed by pytest
def test_reads_file(temp_file):
assert temp_file.read_text() == "hello"
# ---- Parameterized tests ----
@pytest.mark.parametrize("a, b, expected", [
(1, 2, 3),
(-1, 1, 0),
(0, 0, 0),
(100, -200, -100),
(1_000_000, 1, 1_000_001),
])
def test_add(a, b, expected):
assert a + b == expected
# ---- Grouped in a class (shares fixtures, no shared state) ----
class TestPasswordValidator:
def test_too_short(self):
from auth import validate_password
assert not validate_password("abc")
def test_valid(self):
from auth import validate_password
assert validate_password("correcthorsebatterystaple")
@pytest.mark.parametrize("pw", ["", "a", "ab"])
def test_short_passwords_rejected(self, pw):
from auth import validate_password
assert not validate_password(pw)
11Testing in C: Unit Testing Without a Framework
In systems programming (C, C++), the testing landscape is rawer but the principles are identical. The canonical C testing frameworks are Unity (embedded), Check, and Google Test (C++). But it is instructive to see unit testing without any framework, since that is often what systems programmers do.
In C, a unit test is simply a function that calls your code and asserts invariants using a macro. The assert() macro from <assert.h> aborts the program with a message on failure. For a test harness, you add a main() that calls all test functions and counts failures.
Key challenges in C testing:
- No exceptions: errors propagate via return codes or
errno; tests must check these - Memory management: tests must verify no leaks (use Valgrind or AddressSanitizer)
- No reflection: test discovery requires manually registering test functions
- Linking: you must compile the unit under test without
main(), and link it with the testmain()
Google Test (C++) is the gold standard for C++ projects and handles test discovery, parameterization, fixtures (TEST_F), and mocking (via Google Mock). Large C++ projects (LLVM, Chromium) have hundreds of thousands of Google Test cases.
/* stack.h */
typedef struct Stack Stack;
Stack* stack_new(void);
void stack_push(Stack*, int value);
int stack_pop(Stack*, int* out); /* returns 1 on success, 0 if empty */
void stack_free(Stack*);
/* test_stack.c — compile with: gcc test_stack.c stack.c -o test_stack */
#include <stdio.h>
#include <assert.h>
#include "stack.h"
static int failures = 0;
#define TEST(expr) do { \
if (!(expr)) { \
printf("FAIL: %s (%s:%d)\n", #expr, __FILE__, __LINE__); \
failures++; \
} else { \
printf("PASS: %s\n", #expr); \
} \
} while(0)
void test_new_stack_is_empty(void) {
Stack* s = stack_new();
int val;
TEST(stack_pop(s, &val) == 0); /* pop on empty returns 0 */
stack_free(s);
}
void test_push_pop_roundtrip(void) {
Stack* s = stack_new();
int val;
stack_push(s, 42);
TEST(stack_pop(s, &val) == 1);
TEST(val == 42);
stack_free(s);
}
void test_lifo_order(void) {
Stack* s = stack_new();
int val;
stack_push(s, 1);
stack_push(s, 2);
stack_push(s, 3);
stack_pop(s, &val); TEST(val == 3);
stack_pop(s, &val); TEST(val == 2);
stack_pop(s, &val); TEST(val == 1);
stack_free(s);
}
int main(void) {
test_new_stack_is_empty();
test_push_pop_roundtrip();
test_lifo_order();
printf("\n%s (%d failure(s))\n",
failures == 0 ? "ALL TESTS PASSED" : "SOME TESTS FAILED",
failures);
return failures > 0 ? 1 : 0;
}
12Continuous Integration and the Test Lifecycle
All the testing discipline in the world is worthless if tests are only run occasionally. Continuous Integration (CI) is the practice of automatically running the full test suite (and often building, linting, and deploying) on every commit pushed to the shared repository.
The CI pipeline typically looks like:
- Developer pushes a branch
- CI server (GitHub Actions, Jenkins, CircleCI, GitLab CI) detects the push
- Runs linter / static analysis (fast — seconds)
- Runs unit tests (fast — seconds to 1-2 min)
- Runs integration tests (medium — minutes)
- Runs system/E2E tests (optional, slow — minutes to hours)
- Reports pass/fail; blocks merge if anything fails
The golden rule of CI: the main branch must never be broken. A failing CI pipeline on main is treated as a P0 incident.
Test parallelism: Modern CI splits the test suite across multiple workers. This requires tests to be truly independent — no shared files, no shared databases, no port conflicts. Tools like pytest-xdist enable parallel test execution.
Flaky tests — tests that pass sometimes and fail sometimes without code changes — are the cancer of CI pipelines. They erode trust: developers start ignoring red builds. The discipline is to quarantine flaky tests immediately, fix or delete them, and never let them accumulate.
Pre-commit hooks (using the pre-commit tool) run a fast subset of tests locally before a commit is even made, catching trivial errors before they reach CI.
# .github/workflows/test.yml — GitHub Actions CI config
name: Test Suite
on:
push:
branches: [main, dev]
pull_request:
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install -e '.[dev]'
- run: pytest tests/unit/ --cov=src --cov-fail-under=80 -x
integration-tests:
needs: unit-tests # only run if unit tests pass
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16
env:
POSTGRES_PASSWORD: test
ports: ['5432:5432']
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: pip install -e '.[dev]'
- run: pytest tests/integration/ -v
env:
DATABASE_URL: postgresql://postgres:test@localhost:5432/test
Key insights
- ✦Tests are executable specifications: they define what correct behavior means, not just whether the current code happens to work.
- ✦TDD's Red-Green-Refactor cycle is primarily a design discipline — the tests are a beneficial side effect of being forced to specify behavior precisely before implementing it.
- ✦The testing pyramid exists because of economics: higher-level tests cost exponentially more to write, run, and debug. A pyramid-shaped suite gives maximum confidence per engineering hour.
- ✦Code coverage is a lower bound on test quality, not an upper bound. 100% coverage with no assertions proves nothing; 80% coverage with precise assertions is far stronger.
- ✦Mocking is not just a testing convenience — difficulty in mocking a component is a design smell indicating tight coupling or too many responsibilities.
- ✦Property-based testing finds bugs humans never think to test for, because humans are bad at generating adversarial edge cases; random search with shrinking is surprisingly powerful.
- ✦Mutation testing is the closest thing to a formal proof that your tests actually capture the behavior they claim to capture.
- ✦Flaky tests are an existential threat to CI culture: once developers learn to ignore red builds, the entire safety net disappears.
Common pitfalls
- ⚠Testing implementation details instead of behavior: tests that assert the internal state or call order of private methods break on every refactor, even correct ones. Test the public contract.
- ⚠Sharing mutable state between tests: global variables, shared database rows, or module-level side effects cause order-dependent test failures that are nightmarish to debug.
- ⚠Mock everything, including the thing under test: over-mocking produces tests that always pass but verify nothing about real behavior. Only mock what crosses a process or I/O boundary.
- ⚠Treating 100% coverage as the goal: coverage is a means, not an end. It creates incentives to write tests with no assertions just to hit the metric.
- ⚠Writing tests after the fact to hit a coverage target: these tests are written knowing the implementation, so they tend to test what the code does rather than what it should do.
- ⚠Ignoring flaky tests: a test that sometimes fails is treated as noise and ignored. Over time, the entire CI pipeline loses credibility and developers stop trusting it.
- ⚠Not testing error paths and edge cases: tests only for the happy path miss the bugs that actually matter — what happens with null, empty, zero, overflow, or concurrent access?
- ⚠Making tests too long and complex: a test that is 100 lines long and tests 10 things at once is itself buggy software. Each test should have one reason to fail.
Practice
- 01easy
Implement a `Calculator` class with `add`, `subtract`, `multiply`, and `divide` methods using strict TDD: write each test first, make it fail, then implement the minimum code to pass. Include tests for division by zero. Track how many Red-Green-Refactor cycles you complete.
- 02medium
Build a mock-based test suite for an `OrderService` that depends on an `InventoryDB` (checks/decrements stock) and an `EmailNotifier` (sends order confirmations). Write tests that verify: (a) successful order decrements inventory and sends email; (b) out-of-stock order raises an exception and sends no email; (c) email failure does not corrupt inventory. Use only mocks — no real DB or email.
- 03medium
Use the `hypothesis` library to find a bug in this implementation of binary search: write at least 3 properties (output is correct, no index errors, consistent with linear search), run it, and report the minimal failing case hypothesis finds.
- 04hard
Run `mutmut` (or `mutpy`) on the `BankAccount` class from the unit testing section. Report: (a) the mutation score; (b) two surviving mutants and explain what test you would add to kill each one; (c) add those tests and confirm the score improves.
- 05hard
Design a testing strategy for a microservice that (a) reads from a Postgres database, (b) calls an external payment API, (c) publishes events to a Kafka queue, and (d) exposes a REST API. For each layer of the testing pyramid, specify: what you would test, what you would mock/fake, and what tooling you would use. Then implement the unit-test layer.
Resources
The canonical TDD text by its inventor. Short, concrete, essential.
Definitive reference for test double taxonomy (mock/stub/spy/fake/dummy) and test smell catalog.
The canonical reference for Python's most-used test framework, including fixtures, parametrize, plugins.
Full docs for Python property-based testing. The 'How it works' section explains shrinking.
Chapters 11-14 cover unit testing, test doubles, larger testing, and flaky tests — the best practical guide to testing at scale.
The original 2000 paper by Claessen and Hughes that introduced property-based testing.
Ham Vocke's definitive article on martinfowler.com explaining the pyramid with concrete examples.
How to add tests to code that was not designed for testing. Essential for real-world codebases.
Checkpoints — you can now…
- ✓You can write a unit test from scratch using the Arrange-Act-Assert pattern and explain why each assertion targets behavior, not implementation.
- ✓You can execute a complete TDD Red-Green-Refactor cycle for a non-trivial function, writing the test before a single line of production code exists.
- ✓You can explain the difference between a mock, a stub, a spy, and a fake, and identify which is appropriate for a given dependency.
- ✓You can describe the testing pyramid and justify the shape with the economics of test speed, cost, and diagnostic value.
- ✓You can interpret a code coverage report, identify which branches are not covered, and write tests to cover them — while explaining why coverage alone is insufficient.
- ✓You can write a property-based test using hypothesis, identify at least two meaningful properties of a function, and explain how shrinking works.
- ✓You can design a CI pipeline configuration (e.g., GitHub Actions) that runs unit and integration tests on every PR and blocks merges on failure.