Hashing, MACs & Digital Signatures
Build the rigorous foundation of modern security: from one-way hash functions through MACs to digital signatures and the PKI chain of trust that secures every HTTPS connection.
Overview
Cryptographic hash functions, message authentication codes, and digital signatures form the bedrock of nearly every security protocol in use today — TLS, SSH, code signing, blockchain, password storage, and software package verification all rely on these primitives. Understanding them is not optional for a serious computer scientist; they are as fundamental to systems as sorting algorithms are to software engineering.
A hash function takes an arbitrarily long message and produces a fixed-size "fingerprint." The magic — and the deep mathematical difficulty — is in making that fingerprint behave like a one-way trapdoor: trivial to compute in one direction, practically impossible to reverse. This lecture builds from the intuitive notion of a "digital checksum" all the way to the rigorous security definitions (collision resistance, preimage resistance, second-preimage resistance) and how real constructions like SHA-256 and SHA-3 satisfy them.
Message Authentication Codes (MACs) layer in a secret key, turning a hash into a proof of both integrity and authenticity. They answer the question: "Was this message really sent by someone who knows the shared secret, and was it tampered with in transit?" We cover HMAC — the dominant construction — and explain why naive constructions like H(key || message) are dangerously broken.
Digital signatures go one step further: they achieve non-repudiation using asymmetric cryptography. Anyone can verify a signature using a public key, but only the holder of the private key could have produced it. We cover RSA-PKCS#1, ECDSA, and EdDSA, and show how X.509 certificates chain signatures into a Public Key Infrastructure (PKI). By the end of this module, you will understand why your browser trusts a website it has never visited before — and what can go wrong when that chain of trust breaks.
Hash functions, MACs, and signatures are not optional cryptographic ornaments — they are the structural steel of the internet. Every HTTPS connection you make today involves at least one hash function (in the handshake transcript hash), a MAC or AEAD cipher (to authenticate each record), and a digital signature (to verify the server's certificate). Remove any one of these, and the security model collapses entirely. A computer scientist who cannot reason about these primitives cannot reason about the security of any networked system.
Beyond the internet, the practical reach is enormous: git commit hashes, Docker image digests, package manager verification (npm, pip, apt), code-signing for app stores, firmware validation in embedded systems, password hashing for databases, and the proof-of-work / proof-of-stake mechanisms in distributed ledgers. Even mundane-seeming applications like "did this file download without corruption?" rest on hash functions. Fluency with these tools is the difference between building systems that are secure by construction and systems that are one clever attack away from total compromise.
TLS 1.3 Handshake: Hash, MAC, and Signature Interactions
Every major primitive from this module appears in a single TLS 1.3 handshake: SHA-256 as the transcript hash, HKDF (HMAC-based) for key derivation, digital signatures (ECDSA/RSA-PSS) in CertificateVerify, and HMAC in the Finished messages. Remove any one and the security model fails.
The concepts
01What Is a Cryptographic Hash Function?
A hash function h maps an arbitrary-length bitstring M (the message) to a fixed-length bitstring h(M) of n bits (the digest or hash). The word "cryptographic" adds five crucial properties that a mere checksum (like CRC32) does not have:
- Deterministic: same input always gives same output.
- Efficient: computing h(M) is fast for any M.
- Preimage resistance (one-wayness): given a digest d, it is computationally infeasible to find any M such that h(M) = d. Formally, for every probabilistic polynomial-time adversary A, Pr[A(h(M)) = M'] such that h(M') = h(M) is negligible.
- Second-preimage resistance: given M, it is infeasible to find M' ≠ M such that h(M') = h(M).
- Collision resistance: it is infeasible to find any pair (M, M') with M ≠ M' and h(M) = h(M').
Note the hierarchy: collision resistance implies second-preimage resistance (trivially — finding M' ≠ M with h(M')=h(M) is a collision), but the converse does not hold. Preimage resistance is conceptually independent of the other two.
The birthday paradox is the fundamental enemy of short digests. For a uniformly random function with n-bit output, you expect a collision after roughly 2^(n/2) evaluations — not 2^n. This is why SHA-256 with n=256 bits gives 128-bit collision-resistance security. A 64-bit hash would have only 32-bit collision security, which is trivially breakable on modern hardware.
Analogy: A good hash function behaves like a blender — you can easily put fruits in and get smoothie out, but you cannot reconstruct the original fruits from the smoothie, and it is practically impossible to find two different fruit combinations that produce identically-tasting smoothie.
import hashlib
msg = b"Hello, KernelPath!"
digest = hashlib.sha256(msg).hexdigest()
print(digest)
# 3b3f2a21a1e5...
# Avalanche effect: one-bit change → completely different digest
msg2 = b"Hello, KernelPath " # trailing space
print(hashlib.sha256(msg2).hexdigest())
# completely different — ~half the bits flip
02Security Definitions: Preimage, Second-Preimage, Collision Resistance
These three security notions are distinct and it is crucial to keep them separate. They correspond to three different adversarial scenarios:
Preimage resistance (one-wayness): The adversary is given h(M) for a randomly chosen M and must find any preimage. This protects password hashes — if passwords were stored as h(password), a preimage attacker tries to invert h to recover passwords.
Second-preimage resistance: The adversary is given a specific M and must find a different M' with the same hash. This protects file integrity — if I give you a binary and its hash, a second-preimage attacker tries to produce a malicious binary with the same hash.
Collision resistance: The adversary can choose both M and M' freely, only requiring M ≠ M' and h(M) = h(M'). This is the hardest to achieve because the adversary has the most freedom. It protects certificate signing — a CA signs a document D; if collisions exist, the attacker might construct (D, D') with a collision and get the CA to sign D (the legitimate-looking one), then use the signature on D' (the malicious one).
The birthday attack exploits collision resistance. With a hash of n bits:
- Preimage attack: expected 2^n evaluations (brute force)
- Collision attack: expected 2^(n/2) evaluations (birthday paradox)
For SHA-256 (n=256): collision security is 128 bits. For MD5 (n=128): only 64-bit collision security — and actual cryptanalytic attacks on MD5 find collisions in seconds on a laptop.
Formal security game (collision resistance):
Game COLL_h:
(M, M') ← A() # adversary outputs a pair
return (M ≠ M') ∧ (h(M) = h(M'))
Hash function h is collision resistant if Adv[A] = Pr[A wins] is negligible for all PPT adversaries A.
# Birthday paradox demo: how few evaluations to find a collision
# in a toy 32-bit hash (illustrative only, not a real hash)
import random
def toy_hash(x: int, bits: int = 32) -> int:
"""A terrible hash — just for counting collisions."""
# Use Python's hash() as a stand-in
return hash(x) & ((1 << bits) - 1)
def birthday_trial(bits: int = 32) -> int:
seen = {}
for i in range(10**7):
h = toy_hash(i, bits)
if h in seen:
return i # found collision after i evaluations
seen[h] = i
return -1
# Expected: ~sqrt(2^32) = ~65536 evaluations
print(birthday_trial(32)) # typically ~50,000–100,000
03Merkle-Damgård Construction and SHA-2
Most hash functions in practice (MD5, SHA-1, SHA-2) are built using the Merkle-Damgård construction. The idea is to reduce "hash an arbitrary-length message securely" to "build a secure fixed-input-length compression function f."
Merkle-Damgård step by step:
- Pad the message so its length is a multiple of the block size (e.g., 512 bits for SHA-256). Padding is: append a 1 bit, then enough 0 bits, then the original message length as a 64-bit integer. This length encoding is called MD-strengthening and is crucial for preventing length-extension attacks if omitted.
- Split the padded message into blocks B_1, B_2, ..., B_k.
- Set IV = a fixed initialization vector (built into the standard).
- Iterate: H_i = f(H_{i-1}, B_i), starting with H_0 = IV.
- Output H_k as the digest.
Security theorem (Merkle, Damgård, 1989): If the compression function f is collision resistant, then the full hash h is collision resistant. This reduces a hard problem (collide a variable-length hash) to a seemingly simpler one (collide a fixed-length function). The proof is constructive: any collision in h can be converted to a collision in f.
SHA-256 uses:
- Block size: 512 bits
- Output: 256 bits
- Internal state: 8 × 32-bit words (256 bits)
- 64 rounds per block, each mixing in message words and round constants derived from fractional parts of cube roots of primes
- Operations: bitwise AND, OR, XOR, NOT, ROTR (rotate right), addition mod 2^32
SHA-512 is analogous with 64-bit words, 1024-bit blocks, 80 rounds, 512-bit output.
Length-extension attack: Merkle-Damgård has a structural weakness. Given h(M) but not M, an attacker can compute h(M || padding || M') for any suffix M' without knowing M. This is why H(key || message) is insecure as a MAC — the attacker can extend the message. SHA-3 (Keccak) is immune to this attack by design.
# Merkle-Damgård structure (pseudocode — NOT a real compression function)
def merkle_damgard(message: bytes, iv: int, compress_f) -> int:
# 1. Pad: append 1-bit, zeros, then 64-bit length
L = len(message) * 8 # original length in bits
message += b'\x80' # 1-bit followed by 7 zeros
while (len(message) * 8) % 512 != 448:
message += b'\x00'
message += L.to_bytes(8, 'big') # MD-strengthening
# 2. Split into 512-bit (64-byte) blocks
blocks = [message[i:i+64] for i in range(0, len(message), 64)]
# 3. Iterate compression function
H = iv
for block in blocks:
H = compress_f(H, block)
return H
04SHA-3 / Keccak and the Sponge Construction
After the 2004–2005 breaks of MD5 and SHA-1, NIST ran a public competition (2007–2012) for a new hash standard. The winner was Keccak, standardized as SHA-3 in FIPS 202 (2015). SHA-3 is not a drop-in replacement for SHA-2 in terms of performance — it is intentionally different architecturally, providing security diversity.
The sponge construction replaces Merkle-Damgård:
- State: a 1600-bit (for SHA3-256) array organized as a 5×5 matrix of 64-bit lanes.
- Rate r and capacity c: the state is split into a rate (the "input/output" portion, r bits) and capacity (the "security" portion, c bits). For SHA3-256: r = 1088, c = 512, r + c = 1600.
- Absorbing phase: XOR each message block into the rate portion of the state, then apply the Keccak-f permutation.
- Squeezing phase: output r bits of state as digest, applying Keccak-f between outputs if more bits are needed.
Keccak-f[1600] is a 24-round permutation with five steps per round: θ (theta, column parity mixing), ρ (rho, lane rotations), π (pi, lane permutation), χ (chi, nonlinear row mixing — the only nonlinear step), ι (iota, round constant XOR).
Why sponges are better than Merkle-Damgård:
- No length-extension attacks — the capacity bits are never exposed, so an attacker cannot continue the computation.
- Flexible output length — squeeze as many bits as needed (SHA3-256, SHA3-512, SHAKE128, SHAKE256 — the last two are extendable output functions or XOFs).
- Built-in domain separation — different output lengths are achieved without additional hashing.
SHA-3's security margin is conservative: the capacity c gives 2^(c/2) collision security and 2^c preimage security. For SHA3-256, c=512 gives 256-bit collision security — twice as many bits of security as the output length would naively suggest.
import hashlib
# SHA-3 family — all from the sponge construction
print(hashlib.sha3_256(b"KernelPath").hexdigest()) # 256-bit fixed output
print(hashlib.sha3_512(b"KernelPath").hexdigest()) # 512-bit fixed output
# SHAKE: variable-length output (XOF)
shake = hashlib.shake_256(b"KernelPath")
print(shake.hexdigest(32)) # 32 bytes = 256 bits
print(shake.hexdigest(64)) # 64 bytes = 512 bits
# Same input, different output lengths — each is valid
05Message Authentication Codes (MACs)
A MAC (Message Authentication Code) is a keyed primitive: it takes a secret key K and a message M and produces a tag τ = MAC(K, M). The receiver, who also knows K, can verify: does MAC(K, M') == τ? If yes (and K is truly secret), the message is authentic and unmodified.
MACs provide two things that an unkeyed hash cannot:
- Authentication — only someone knowing K can compute valid tags.
- Integrity — any modification to M changes τ.
But MACs do not provide non-repudiation — both sender and receiver know K, so either could have computed the tag. (Signatures solve this.)
Formal security (EU-CMA — existential unforgeability under chosen-message attack): An adversary who can query MAC(K, ·) adaptively on messages of their choice must still be unable to forge a valid tag on any new message not previously queried. This is the gold-standard security notion.
Why H(key || message) is broken: This is the length-extension attack. Given MAC(K, M) = H(K || M) and the Merkle-Damgård structure, an attacker who knows |K| can compute H(K || M || pad || M') for any M' without knowing K. This is a catastrophic real-world vulnerability — never construct MACs this way.
Why H(message || key) is broken differently: Collisions in H directly give MAC forgeries — find M ≠ M' with H(M) = H(M'), then H(M||K) = H(M'||K).
HMAC (RFC 2104) is the correct construction:
HMAC(K, M) = H((K' ⊕ opad) || H((K' ⊕ ipad) || M))
where K' is K zero-padded to the block size, opad = 0x5c5c...5c, ipad = 0x3636...36. The double-hash with different XOR pads defeats the length-extension attack and has a clean security proof: if H is a PRF (pseudorandom function), HMAC is a PRF.
KMAC (from NIST SP 800-185): the SHA-3-based MAC, simpler and cleaner — just prefix the key and a length encoding, then use SHAKE. No nested hash needed because SHA-3's sponge design prevents length extension.
import hmac, hashlib
key = b"super-secret-key-32bytes-padded!"
message = b"Transfer $100 to Alice"
# Correct: HMAC-SHA256
tag = hmac.new(key, message, hashlib.sha256).hexdigest()
print(tag)
# Verification (constant-time comparison prevents timing attacks)
def verify_mac(key, message, received_tag):
expected = hmac.new(key, message, hashlib.sha256).digest()
received = bytes.fromhex(received_tag)
# hmac.compare_digest is constant-time!
return hmac.compare_digest(expected, received)
print(verify_mac(key, message, tag)) # True
print(verify_mac(key, b"Transfer $999 to Eve", tag)) # False
06Timing Attacks and Constant-Time Comparison
A subtlety that catches even experienced engineers: naive MAC verification is vulnerable to timing attacks.
Consider a byte-by-byte comparison:
def bad_verify(expected, received):
if len(expected) != len(received):
return False
for e, r in zip(expected, received):
if e != r:
return False # exits early!
return True
This leaks information: if the first byte is wrong, the function returns in ~1ns. If the first byte is right but the second is wrong, it takes ~2ns. An attacker who can measure timing with sufficient precision can brute-force the tag one byte at a time — 256 × 32 = 8,192 guesses instead of 256^32 ≈ 10^77. This is not theoretical — it has been demonstrated on real systems over networks.
Constant-time comparison ensures the function always takes the same time regardless of where the mismatch occurs:
def const_time_eq(a: bytes, b: bytes) -> bool:
if len(a) != len(b):
return False
result = 0
for x, y in zip(a, b):
result |= x ^ y # accumulate differences without branching
return result == 0
Every byte is always compared; no early exit. Python's hmac.compare_digest does this correctly. In C, CRYPTO_memcmp from OpenSSL or timingsafe_bcmp from BSD libc are the right tools.
The lesson: cryptographic correctness is not just about the algorithm — it is about the implementation details. Side channels (timing, power consumption, cache behavior, electromagnetic emissions) can break theoretically sound schemes. Real-world cryptography requires constant-time implementations, careful memory handling, and defense in depth.
import hmac
def const_time_eq_manual(a: bytes, b: bytes) -> bool:
"""Constant-time equality — always touches every byte."""
if len(a) != len(b):
# Length mismatch leaks length info, but that's usually OK
# (tag lengths are fixed and public)
return False
diff = 0
for x, y in zip(a, b):
diff |= x ^ y
return diff == 0
# In production, always use the library's implementation:
hmac.compare_digest(b"abc", b"abc") # True, constant-time
07Digital Signatures: Asymmetric Authentication
A digital signature scheme consists of three algorithms:
- KeyGen() → (pk, sk): generate a public key and private (signing) key.
- Sign(sk, M) → σ: produce a signature over message M using the private key.
- Verify(pk, M, σ) → {accept, reject}: check that σ is valid for M under pk.
The correctness property: Verify(pk, M, Sign(sk, M)) = accept always.
The security property — EU-CMA (Existential Unforgeability under Chosen-Message Attack): an adversary who can ask for signatures on arbitrary messages of their choice cannot forge a valid signature on any new message not previously signed. This is the standard notion and it implies both authentication and non-repudiation: the signer cannot later deny having signed M, because only the holder of sk could have produced σ.
RSA-PKCS#1 v1.5 (historical, widely deployed):
- KeyGen: choose large primes p, q; N = pq; e (typically 65537); d = e^{-1} mod φ(N).
- Sign: σ = H(M)^d mod N (with PKCS#1 padding — the padding is critical and has been attacked!).
- Verify: σ^e mod N == H(M).
- Security based on RSA problem hardness (factoring N). Key size ≥ 2048 bits required today.
RSA-PSS (RSA Probabilistic Signature Scheme): the modern RSA signature standard. Uses randomized padding that admits a tight security reduction to RSA. Prefer over PKCS#1 v1.5 for new systems.
ECDSA (Elliptic Curve Digital Signature Algorithm):
- Private key: random scalar k in Z_n
- Public key: point Q = kG on an elliptic curve
- Sign: choose random nonce r; compute (x, y) = rG; s = r^{-1}(H(M) + k·x) mod n; output (x mod n, s)
- Security critical point: the nonce r must be random and unique per signature. Sony's PlayStation 3 used a constant nonce in ECDSA, allowing extraction of the private key from just two signatures!
EdDSA (Edwards-curve Digital Signature Algorithm — RFC 8032):
- Used in Ed25519 (Curve25519) and Ed448.
- Deterministic: the nonce is derived as H(sk || M) — no random number generator needed. This eliminates the catastrophic nonce-reuse vulnerability of ECDSA.
- Constant-time by design, fast, small signatures (64 bytes for Ed25519).
- Strongly preferred for new systems over ECDSA.
# Ed25519 signatures using Python's cryptography library
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey
# Key generation
private_key = Ed25519PrivateKey.generate()
public_key = private_key.public_key()
message = b"I authorize payment of $1000 to Bob"
# Sign
signature = private_key.sign(message)
print(f"Signature ({len(signature)} bytes): {signature.hex()[:32]}...")
# Verify (raises InvalidSignature on failure)
from cryptography.exceptions import InvalidSignature
try:
public_key.verify(signature, message)
print("Signature valid!")
except InvalidSignature:
print("Tampered!")
# Tamper with message
try:
public_key.verify(signature, b"I authorize payment of $9999 to Eve")
except InvalidSignature:
print("Forgery detected — signature invalid")
08Hash-then-Sign: Why We Hash Before Signing
RSA and ECDSA cannot sign arbitrary-length messages directly — they operate on fixed-size integers. But there is a deeper reason to hash first, even for schemes that could handle long inputs: efficiency and security.
The hash-then-sign paradigm:
- Compute d = H(M) (a fixed-size digest, e.g., 32 bytes for SHA-256).
- Sign d: σ = Sign(sk, d).
- Verify: compute d' = H(M'), check Verify(pk, d', σ).
Signing a 32-byte hash instead of a 10MB file is 312,500× more efficient. But correctness requires that H is collision resistant — if an adversary finds M ≠ M' with H(M) = H(M'), they can get a signature on M and present it as a signature on M'. This is exactly the real-world attack:
The Flame malware attack (2012): Microsoft used MD5 in one of their certificate chains. Researchers (Sotirov, Stevens et al.) had already shown MD5 collision attacks in 2008. The Flame authors found a chosen-prefix collision in MD5 to forge a Microsoft code-signing certificate. They produced two documents: one a legitimate Microsoft terminal-services certificate (which Microsoft signed), and one a rogue CA certificate, with identical MD5 hashes. Microsoft's signature over the legitimate certificate was also a valid signature over the rogue CA cert. The attackers then used this to sign malware as if it were from Microsoft.
The lesson: the security of hash-then-sign is only as strong as the collision resistance of the hash. Never use MD5 or SHA-1 for signatures. Use SHA-256 or SHA-3 minimum.
# Hash-then-sign with RSA-PSS (the modern standard)
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import rsa, padding
from cryptography.hazmat.backends import default_backend
# Key generation (use >= 2048 bits in production)
private_key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
backend=default_backend()
)
public_key = private_key.public_key()
message = b"I, Alice, hereby transfer title to Bob."
# RSA-PSS: randomized padding with tight security reduction
signature = private_key.sign(
message,
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
hashes.SHA256() # <- hash-then-sign: library hashes M for us
)
# Verify
from cryptography.exceptions import InvalidSignature
try:
public_key.verify(signature, message,
padding.PSS(
mgf=padding.MGF1(hashes.SHA256()),
salt_length=padding.PSS.MAX_LENGTH
),
hashes.SHA256())
print("RSA-PSS signature valid")
except InvalidSignature:
print("Invalid")
09X.509 Certificates and Public Key Infrastructure (PKI)
A digital signature only proves that whoever holds the private key signed the message. But how does Alice know that a public key pk_bank actually belongs to her bank, and not to an attacker who sent her pk_evil claiming it is the bank?
This is the key distribution problem, and it is the central challenge of public-key cryptography in practice. X.509 certificates and PKI are the internet's (imperfect) solution.
An X.509 certificate is a data structure containing:
- Subject: "CN=www.google.com, O=Google LLC, ..."
- Subject Public Key: the RSA/ECDSA/EdDSA public key
- Issuer: the Certificate Authority (CA) that signed this certificate
- Validity period: notBefore, notAfter
- Extensions: Subject Alternative Names (SANs), key usage, basic constraints, etc.
- The Issuer's signature over all of the above (using SHA-256 with RSA or ECDSA)
The chain of trust:
- A Root CA (e.g., DigiCert, Let's Encrypt's ISRG Root X1) has a self-signed certificate. These are hardcoded into browsers/OS.
- The Root CA signs Intermediate CA certificates, delegating signing authority.
- The Intermediate CA signs Leaf (end-entity) certificates for domain owners.
- When you connect to google.com, the server presents its leaf cert + intermediate cert chain. Your browser verifies each signature back to a trusted root.
Certificate verification steps (simplified):
- Build the chain: leaf → intermediate → root.
- Verify each certificate's signature using the issuer's public key.
- Check validity periods.
- Check that the subject's CN or SAN matches the hostname.
- Check for revocation (CRL or OCSP).
- Check key usage extensions (e.g., is the leaf cert actually allowed to sign other certs? — it should not be).
Certificate Transparency (CT): a public, append-only log of all certificates ever issued. Browsers require CT proof for certificates to be trusted. This prevents CAs from secretly issuing rogue certificates (which has happened historically — DigiNotar CA was compromised in 2011 and issued fraudulent certificates for google.com that were used in Iran).
Let's Encrypt democratized HTTPS by providing free, automated, DV (Domain Validated) certificates via the ACME protocol (RFC 8555). It verifies domain ownership by having you serve a challenge at a well-known URL, then issues a 90-day certificate signed by the ISRG Root X1.
# Inspecting a TLS certificate in Python (real-world example)
import ssl, socket
def get_cert_info(hostname: str, port: int = 443) -> dict:
context = ssl.create_default_context()
with socket.create_connection((hostname, port)) as sock:
with context.wrap_socket(sock, server_hostname=hostname) as ssock:
cert = ssock.getpeercert()
return cert
cert = get_cert_info("github.com")
print("Subject:", dict(x[0] for x in cert['subject']))
print("Issuer: ", dict(x[0] for x in cert['issuer']))
print("Valid until:", cert['notAfter'])
print("SANs:", cert.get('subjectAltName'))
# Subject: {'commonName': 'github.com'}
# Issuer: {'commonName': 'DigiCert ...', ...}
# Valid until: ...
10Password Hashing: Bcrypt, scrypt, and Argon2
Cryptographic hash functions like SHA-256 are deliberately fast — that is their design goal for integrity checking. But fast is dangerous for password storage: a GPU can compute 10 billion SHA-256 hashes per second, making offline dictionary attacks trivial if passwords are stored as SHA-256(password).
Password hashing functions are designed to be deliberately slow and memory-hard:
Bcrypt (1999): based on the Blowfish cipher's key schedule. The cost parameter (work factor) doubles the computation time with each increment. Bcrypt(password, cost=12) takes ~250ms on modern hardware. Output includes the salt and cost factor, so verification just calls bcrypt on the stored hash.
scrypt (2009, Colin Percival): adds a memory-hard component. The algorithm fills a large memory buffer with pseudorandom data, then accesses it in a data-dependent (unpredictable) pattern. This forces attackers to pay for both time and memory — GPUs and FPGAs have limited RAM per core, making massive parallelism expensive. Parameters: N (memory), r (block size), p (parallelism).
Argon2 (winner of Password Hashing Competition 2015): the current best practice. Three variants:
- Argon2d: maximizes memory usage, best against GPU attacks, but vulnerable to side-channel attacks.
- Argon2i: uses data-independent memory access patterns, side-channel resistant.
- Argon2id: hybrid — recommended for password hashing (OWASP recommendation).
Parameters: time cost (iterations), memory cost (KB), parallelism (threads).
Salting is mandatory: a unique random salt per password ensures that two users with the same password get different hashes, defeating rainbow table attacks (precomputed hash → password lookup tables).
The workflow:
- Store: salt = random_bytes(16); hash = Argon2id(password, salt, params); store (salt, hash, params).
- Verify: recompute Argon2id(candidate_password, stored_salt, stored_params); compare constant-time.
# Argon2id password hashing (pip install argon2-cffi)
from argon2 import PasswordHasher
from argon2.exceptions import VerifyMismatchError
# OWASP recommended parameters (as of 2024):
# memory_cost=19456 KB (~19 MB), time_cost=2, parallelism=1
ph = PasswordHasher(
time_cost=2,
memory_cost=19456,
parallelism=1,
hash_len=32,
salt_len=16
)
# Store: hash includes salt, params, version (PHC string format)
hash_str = ph.hash("correct horse battery staple")
print(hash_str[:50], "...")
# $argon2id$v=19$m=19456,t=2,p=1$...
# Verify
try:
ph.verify(hash_str, "correct horse battery staple")
print("Login OK")
except VerifyMismatchError:
print("Wrong password")
# WRONG — never do this for passwords:
import hashlib
bad_hash = hashlib.sha256(b"password123").hexdigest() # crackable instantly
11Applications: Git, TLS, Code Signing, Blockchain
The primitives from this module appear everywhere in production systems. Understanding how they compose reveals both the elegance and the fragility of real-world security.
Git: every commit is identified by SHA-1(tree_hash || parent_hash || author || message). The commit hash is a cryptographic link — changing any byte of any file in history changes all descendant commit hashes. This makes history tampering detectable (though SHA-1 is now weak; GitHub rejects pushes exploiting the SHAttered collision). Git is migrating to SHA-256 object IDs. Git's trust model is weaker than signed commits — git commit --gpg-sign uses GnuPG to add a digital signature, providing actual non-repudiation.
TLS 1.3 handshake sketch:
- Client hello: send supported cipher suites, key share (ECDH public value).
- Server hello: choose cipher suite, send key share, send certificate chain.
- Client verifies certificate chain (chain of digital signatures back to trusted root).
- Both derive session keys using ECDH + HKDF (a hash-based key derivation function).
- Server sends CertificateVerify: a signature over the handshake transcript hash — proves possession of the private key corresponding to the certificate's public key.
- Both send Finished: HMAC over the entire handshake transcript — ensures nothing was tampered with. Every step uses our primitives: hash functions (SHA-256/384), MACs (HMAC in HKDF and Finished), signatures (ECDSA/RSA-PSS/EdDSA in CertificateVerify).
Package managers: pip install foo fetches a .whl file and a SHA-256 hash from PyPI. The hash is signed with the maintainer's private key (via sigstore / OIDC-based signing as of 2023). Your pip client verifies both the hash and the signature.
Bitcoin/Ethereum: transactions are signed with ECDSA (secp256k1). Blocks form a hash chain: each block header includes SHA-256(prev_block_header), making history immutable without redoing all subsequent proof-of-work. Merkle trees of transaction hashes allow SPV clients to verify individual transactions without downloading the full block.
# Git-style content-addressed storage using SHA-256
import hashlib, json
from dataclasses import dataclass
from typing import Optional
@dataclass
class Commit:
tree_hash: str
parent_hash: Optional[str]
message: str
author: str
def hash(self) -> str:
content = json.dumps({
"tree": self.tree_hash,
"parent": self.parent_hash,
"msg": self.message,
"author": self.author
}, sort_keys=True).encode()
return hashlib.sha256(content).hexdigest()
c1 = Commit("abc123", None, "Initial commit", "Alice")
c2 = Commit("def456", c1.hash(), "Add feature X", "Bob")
print(c1.hash()) # deterministic fingerprint
print(c2.hash()) # depends on c1's hash — chain!
Key insights
- ✦Collision resistance implies second-preimage resistance, but NOT preimage resistance — these three notions are logically distinct and correspond to different attack scenarios.
- ✦The birthday paradox halves the effective security bits for collision attacks: a 256-bit hash gives only 128-bit collision security, not 256-bit.
- ✦Merkle-Damgård's length-extension vulnerability means H(key || message) is never a secure MAC — HMAC exists precisely to fix this structural flaw.
- ✦ECDSA's nonce must be both random and unique per signature; reusing a nonce leaks the private key with just two observed signatures (the Sony PS3 disaster).
- ✦EdDSA derives its nonce deterministically from the private key and message, making nonce-reuse impossible by design — this is why it is preferred over ECDSA for new systems.
- ✦Password hashing functions (Argon2id) must be slow and memory-hard; fast hashes like SHA-256 are catastrophically wrong for passwords even with salting.
- ✦The security of hash-then-sign depends entirely on collision resistance of the hash — the Flame malware forged Microsoft certificates by finding an MD5 collision.
- ✦Certificate Transparency logs provide a public, auditable record of all issued TLS certificates, preventing silent issuance of rogue certificates by compromised CAs.
Common pitfalls
- ⚠Using H(key || message) or H(message || key) as a MAC instead of HMAC — the former is vulnerable to length extension, the latter to collision attacks on H.
- ⚠Using == (non-constant-time) to compare MACs or signature results, enabling byte-by-byte timing attacks that reduce brute-force complexity from 2^256 to 256×32.
- ⚠Reusing a nonce in ECDSA — this directly leaks the private key. Even a single nonce reuse across two signatures is sufficient for key recovery.
- ⚠Confusing collision resistance with preimage resistance when choosing a hash for password storage — a hash can be collision resistant yet perfectly invertible with a preimage attack.
- ⚠Using MD5 or SHA-1 for digital signatures or certificate fingerprints in new systems — both have known collision attacks; SHA-256 minimum.
- ⚠Storing passwords as plain SHA-256(password) — fast hashes make GPU dictionary attacks trivial; use Argon2id, bcrypt, or scrypt.
- ⚠Trusting the length of a signature rather than verifying it cryptographically — the Bleichenbacher 1998 attack exploited sloppy RSA signature verification that accepted signatures with garbage padding.
- ⚠Assuming that a self-signed certificate is secure because it is 'encrypted' — TLS encrypts the channel, but without a trusted certificate chain, there is no authentication and the connection is trivially MITM-able.
Practice
- 01medium
Implement a simplified Merkle tree in Python. Given a list of n data items, build a binary tree where each leaf is SHA-256(item) and each internal node is SHA-256(left_child || right_child). Return the root hash. Then write a function that produces a Merkle proof (the sibling hashes along the path from a leaf to the root) and a verifier that confirms a given item is in the tree using only the proof and the root hash. This is how Bitcoin SPV clients and Certificate Transparency logs work.
- 02hard
Demonstrate the length-extension attack on H(key || message) where H is SHA-256. Without knowing the key, write code that takes MAC = SHA-256(key || message) and the length of key, and computes a valid MAC for the extended message (message || padding || extra_data). Verify your forged MAC against the real SHA-256(key || message || padding || extra_data). (Hint: use the 'sha256-length-extension' concept — you need to restore the intermediate hash state from the output.)
- 03hard
Prove (informally but rigorously) that if hash function H is collision resistant, then HMAC-H is existentially unforgeable under chosen-message attack. Your proof should proceed by contradiction: assume there exists a PPT adversary A that forges HMAC-H; show how to use A to construct an adversary B that finds a collision in H.
- 04easy
Write a tool that downloads a Python package from PyPI, verifies its SHA-256 hash against the published hash, and reports whether the file was tampered with. Extend it to also check whether the package has a sigstore signature (packages signed after 2023 do). Handle the case where no signature is available gracefully.
- 05hard
The ECDSA private key recovery attack: suppose you observe two ECDSA signatures (r1, s1) and (r2, s2) over messages M1 and M2, and you know the signer reused the same nonce k (so r1 = r2 = r). Derive the formula to recover the private key d from these two signatures. Implement this in Python using the cryptography library's EC primitives, and verify your recovered key by signing a new message and checking the public key.
- 06medium
Build a minimal certificate verification function from scratch (without using ssl.match_hostname or pyOpenSSL). Given a server's certificate (as DER bytes) and an intermediate certificate, verify: (1) the intermediate's signature over the leaf, (2) that the leaf's subject matches a given hostname (including wildcard matching), (3) that the current date is within the validity period. Use only the cryptography library for ASN.1 parsing and signature verification.
Resources
MIT OCW 6.875 — rigorous treatment of hash functions, MACs, and signatures with formal security proofs.
The standard graduate cryptography textbook. Chapters 4 (hash functions), 5 (MACs), and 12–13 (signatures) are directly relevant. Rigorous game-based security definitions throughout.
Dan Boneh's legendary Stanford crypto course. Freely auditable. Covers collision resistance, HMAC, and signatures with remarkable clarity.
Accurate detailed description of the Merkle-Damgård construction, padding, and SHA-256/512 round functions with pseudocode.
The authoritative specification for SHA-1, SHA-224, SHA-256, SHA-384, SHA-512 and variants.
The authoritative specification for SHA-3 and SHAKE, including the sponge construction and Keccak-f permutation.
The original HMAC specification by Bellare, Canetti, and Krawczyk. Short and readable. Includes the security theorem.
Full specification of Edwards-curve signatures including Ed25519 and Ed448. Includes test vectors.
Practical and rigorous. Chapter 6 (hash functions) and chapter 7 (keyed hashing) are excellent. Written by a professional cryptographer.
Official explainer from the CT project team. Explains the append-only Merkle log design and why it prevents silent rogue certificate issuance.
The original Argon2 specification paper by Biryukov, Dinu, and Khovratovich.
Phillip Rogaway's essay on the societal implications of cryptography — broadens perspective on why this work matters beyond technical correctness.
Checkpoints — you can now…
- ✓You can state and distinguish the three security properties — preimage resistance, second-preimage resistance, and collision resistance — and give a concrete attack scenario for each.
- ✓You can explain why the birthday paradox means a 256-bit hash provides 128-bit collision security, and compute the expected number of evaluations to find a collision for a given output size.
- ✓You can describe the Merkle-Damgård construction step by step, explain what MD-strengthening is, and demonstrate the length-extension attack on H(key || message).
- ✓You can implement correct HMAC-based authentication in Python, using constant-time comparison for verification, and explain why both components are necessary.
- ✓You can explain the sponge construction at the level of rate/capacity/absorb/squeeze, and state why SHA-3 is immune to length-extension attacks while SHA-2 is not.
- ✓You can describe the EU-CMA security game for digital signatures, explain the difference between RSA-PSS and RSA-PKCS#1 v1.5, and state why ECDSA nonce reuse catastrophically leaks the private key.
- ✓You can trace a TLS 1.3 handshake and identify where hash functions, MACs, and digital signatures each appear and what security property each provides.
- ✓You can explain why SHA-256(password) is wrong for password storage, and configure Argon2id with appropriate parameters for a production login system.