Network & Systems Security
From TLS handshakes to sandboxed processes, learn how real systems defend themselves — and why defenders must be right every time while attackers need to be right only once.
Overview
Security is the discipline of adversarial reasoning: you must think like an attacker to build systems that survive attack. Unlike most of computer science — where you design for a cooperative, predictable environment — security forces you to reason about worst-case intelligent adversaries who will probe every assumption you make. This asymmetry is what makes security hard: a defender must close every gap, while an attacker only needs to find one.
This module covers the security stack from the ground up, beginning with how two strangers on the internet can establish a private channel (TLS), moving through the bureaucratic machinery that makes trust scalable (PKI), and examining how the OS enforces boundaries (access control, sandboxing). We also study the attacker's arsenal — malware taxonomy, lateral movement, privilege escalation — so you understand what you are defending against, not just abstract policies.
The ideas here are mathematically deep, rooted in complexity theory and number theory, but their applications are intensely practical. Every HTTPS connection you make, every time you ssh into a server, every container you launch — all of these are instantiations of the concepts in this module. Understanding them at the mechanism level turns you from a consumer of security into someone who can design it.
By the end of this module you will be able to read an RFC, trace a TLS 1.3 handshake packet by packet, explain why a misconfigured certificate chain is exploitable, build a minimal firewall ruleset, and reason about what a sandbox can and cannot guarantee. More importantly, you will have the mental model of a security engineer: assume breach, minimize trust, fail securely.
Network and systems security is not an add-on discipline — it is what makes distributed computing viable at all. Without TLS, e-commerce, online banking, and private communication collapse. Without access control models, multi-user operating systems are impossible. Without sandboxing, every browser tab would be able to exfiltrate your SSH keys. These are not theoretical concerns: the history of computing is littered with catastrophic failures from Heartbleed (a buffer over-read that leaked private keys from millions of servers) to the SolarWinds supply-chain attack (malware injected into a trusted software update mechanism) to Spectre/Meltdown (microarchitectural side channels that broke OS-level isolation). Each of these attacks exploited a failure in the concepts this module covers.
For a CS student entering the field, security literacy is now table stakes. Cloud computing, containerization, and distributed microservices have massively expanded the attack surface of modern systems. Developers write code that runs in adversarial environments (the public internet, shared cloud tenants, untrusted user input) and must make security decisions every day. A foundation in TLS, PKI, access control, and secure design principles is what separates engineers who build things that hold up from those who create the next breach headline.
TLS 1.3 Handshake and Certificate Chain Verification
TLS 1.3 one-RTT handshake: the client sends its ECDHE key share in ClientHello so the server can compute the shared secret immediately, then send its certificate and Finished in a single flight. Application data flows after exactly one round-trip.
The concepts
01TLS 1.3 — The Anatomy of a Secure Channel
Transport Layer Security (TLS) is the protocol that wraps almost all modern internet traffic in confidentiality, integrity, and authentication. Understanding TLS 1.3 (RFC 8446) is essential because it is simultaneously a masterclass in applied cryptography and a microcosm of secure protocol design.
What TLS must achieve
Three goals: (1) Confidentiality — an eavesdropper on the wire cannot read the traffic. (2) Integrity — an active attacker cannot tamper with bytes without detection. (3) Authentication — the client can verify it is talking to the real server, not an impersonator.
The TLS 1.3 handshake (simplified)
Client Server
|-- ClientHello (key_share, --------->|
| supported_versions, random) |
| |
|<-- ServerHello (key_share, ---------|
| random), {Certificate}, |
| {CertificateVerify}, |
| {Finished} |
| |
|-- {Finished} --------------------> |
|== Application Data ===============>|
In TLS 1.3, the client sends its Diffie-Hellman key share in the very first message (ClientHello), so the server can immediately compute the shared secret. This means application data can flow after one round-trip (1-RTT), compared to TLS 1.2's two. For resumed sessions, TLS 1.3 even supports 0-RTT early data using a pre-shared key (PSK) — though with the caveat that 0-RTT data is vulnerable to replay attacks.
Key exchange via ECDHE
The workhorse is Elliptic Curve Diffie-Hellman Ephemeral (ECDHE). Both sides pick ephemeral private keys a and b, exchange public values aG and bG on an elliptic curve, and each computes the shared secret abG. "Ephemeral" is the crucial word: a fresh key pair is generated for every session, providing forward secrecy — if the server's long-term private key is stolen years later, recorded past sessions remain undecipherable.
Record layer and AEAD
After the handshake, all data is encrypted with an Authenticated Encryption with Associated Data (AEAD) cipher — specifically AES-128-GCM or ChaCha20-Poly1305. AEAD simultaneously encrypts for confidentiality and authenticates for integrity in a single pass. A running sequence number is included in the MAC computation, preventing replay of old records.
Why TLS 1.2 had to die
TLS 1.2 accumulated 20 years of legacy: negotiable cipher suites that included RC4 and 3DES, optional RSA key exchange (no forward secrecy), CBC-mode ciphers vulnerable to padding oracle attacks (BEAST, POODLE), and renegotiation vulnerabilities. TLS 1.3 removed all of this, cutting the cipher suite space from hundreds of options to five, and banning RSA key transport entirely.
# Minimal TLS 1.3 client in Python — observe the negotiated cipher
import ssl, socket
ctx = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.minimum_version = ssl.TLSVersion.TLSv1_3
ctx.load_default_certs()
with socket.create_connection(("example.com", 443)) as raw:
with ctx.wrap_socket(raw, server_hostname="example.com") as tls:
print(tls.version()) # TLSv1.3
print(tls.cipher()) # ('TLS_AES_128_GCM_SHA256', ...)
print(tls.getpeercert()["subject"])
tls.sendall(b"GET / HTTP/1.0\r\nHost: example.com\r\n\r\n")
print(tls.recv(4096)[:200])
02Public Key Infrastructure (PKI) and the Web of Trust
TLS authenticates the server using a certificate — a signed data structure binding a public key to a domain name. But who signs the certificate, and why should the client trust that signature? This is the problem PKI solves.
The chain of trust
A Certificate Authority (CA) is an entity whose public key is pre-installed in your browser/OS as a root certificate — a trust anchor. When you visit https://bank.com, the server presents a certificate for bank.com signed by an intermediate CA, whose certificate is signed by a root CA. Your browser walks this certificate chain up to a root it already trusts.
Root CA (self-signed, in browser trust store)
└── Intermediate CA (signed by Root CA)
└── bank.com certificate (signed by Intermediate CA)
Why the intermediate? Root CA private keys are kept offline in hardware security modules in vaults. Intermediates are online and can be revoked if compromised.
X.509 certificate anatomy
An X.509 certificate contains: subject (who it belongs to), issuer (who signed it), validity period (not-before, not-after), subject public key, Subject Alternative Names (the actual domain names it covers), and the CA's signature over all of the above. Critical extensions include Basic Constraints (is this a CA cert?), Key Usage, and Extended Key Usage.
Revocation: CRL and OCSP
Certificates can be revoked before expiry — for instance, if the server's private key is leaked. Two mechanisms exist: Certificate Revocation Lists (CRLs) are signed lists of revoked serial numbers published periodically by CAs. Online Certificate Status Protocol (OCSP) allows the client to query a responder in real time for a specific certificate's status. Neither is perfect: CRLs are large and stale; OCSP adds latency and leaks browsing history to the CA. OCSP stapling fixes this by having the server pre-fetch and cache the signed OCSP response, stapling it to the TLS handshake.
Certificate Transparency (CT)
In 2011, DigiNotar CA was compromised and issued fraudulent certs for Google.com, enabling man-in-the-middle attacks on Iranian dissidents. Certificate Transparency (RFC 6962) was the response: all publicly-trusted CAs are now required to log every certificate they issue to publicly auditable append-only logs. Browsers reject certs not in the logs. This makes mis-issuance detectable even if not preventable.
Trust store politics
The browser trust store (Mozilla NSS, Windows CryptoAPI, Apple Keychain) is a political document. Each root CA therein represents an organization whose verification practices have been accepted by the browser vendor. The WebPKI is only as strong as the weakest trusted CA — and there are ~200 of them. Certificate pinning (HPKP, now deprecated; used in mobile apps) attempted to constrain which CAs could issue for a given domain, but caused too many self-inflicted outages.
DANE: DNS-based Authentication
DNS-Based Authentication of Named Entities (DANE, RFC 6698) uses DNSSEC-signed DNS records (TLSA records) to pin certificates directly in DNS, bypassing the CA system entirely for domains that have DNSSEC deployed.
# Inspect a certificate chain with Python
import ssl, socket
from cryptography import x509
from cryptography.hazmat.backends import default_backend
import datetime
def inspect_cert_chain(hostname, port=443):
ctx = ssl.create_default_context()
with socket.create_connection((hostname, port)) as raw:
with ctx.wrap_socket(raw, server_hostname=hostname) as tls:
der_certs = tls.getpeercert(binary_form=True)
cert = x509.load_der_x509_certificate(der_certs, default_backend())
print("Subject:", cert.subject)
print("Issuer :", cert.issuer)
print("Valid :", cert.not_valid_before_utc, "->", cert.not_valid_after_utc)
san = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName)
print("SANs :", san.value.get_values_for_type(x509.DNSName))
print("Days left:", (cert.not_valid_after_utc -
datetime.datetime.now(datetime.timezone.utc)).days)
inspect_cert_chain("github.com")
03Firewalls — Packet Filtering to Stateful Inspection
A firewall is a policy enforcement point: it decides which network traffic is allowed to pass and which is blocked. Understanding firewalls means understanding what they can and cannot see.
Packet-filter firewalls (Layer 3/4)
The simplest firewalls operate on individual packets, inspecting the IP header and TCP/UDP header fields: source IP, destination IP, source port, destination port, and protocol. Rules are evaluated in order; the first match wins.
iptables -A INPUT -p tcp --dport 22 -s 192.168.1.0/24 -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j DROP
iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
The third rule exemplifies stateful inspection: the firewall tracks connection state (SYN seen, SYN-ACK seen, established, etc.) so that return traffic for outbound connections is automatically allowed without writing symmetric rules.
Why pure packet-filtering fails
Packet filters see individual packets but not sessions. An attacker can fragment packets so that no single fragment triggers a rule. A packet filter also cannot distinguish "this port-80 traffic is HTTP" from "this is an SSH tunnel disguised as HTTP". It is trivially bypassed by tunneling prohibited traffic inside permitted protocols.
Deep Packet Inspection (DPI)
DPI firewalls (Next-Generation Firewalls, NGFWs) operate at Layer 7 — they reassemble TCP streams and parse application protocols. An NGFW can enforce policies like "allow HTTPS to *.google.com but block uploading files > 10MB" by inspecting the TLS SNI extension and tracking data volume. However, DPI cannot inspect the contents of properly encrypted TLS 1.3 sessions (only metadata like SNI and certificate).
Egress filtering — the overlooked half
Most organizations configure strict ingress rules but permissive egress. This is a mistake: malware almost always needs to phone home (command-and-control), exfiltrate data, or download payloads. Aggressive egress filtering — allowing only expected protocols to expected destinations — contains breaches dramatically. DNS-over-HTTPS (DoH) has made this harder because DNS queries are now hidden in encrypted HTTPS streams.
Firewall zones and DMZ
A demilitarized zone (DMZ) is a network segment that is less trusted than the internal network but more controlled than the internet. Web servers, mail servers, and DNS resolvers live in the DMZ. Traffic from the internet can reach DMZ hosts on specific ports; traffic from DMZ hosts cannot reach internal systems without inspection. This limits blast radius: a compromised web server cannot directly reach the internal database.
Network Address Translation (NAT) as accidental firewall
NAT rewrites source IP/port of outgoing packets and maintains a mapping to rewrite return traffic. It provides de facto ingress filtering (unsolicited inbound connections have no NAT mapping entry to follow), but this is a side effect, not a security policy. NAT can be bypassed by techniques like NAT traversal (used in WebRTC and VPNs).
# Conceptual stateful firewall in Python pseudocode
from collections import defaultdict
from enum import Enum
class State(Enum):
NEW = "new"
ESTABLISHED = "established"
RELATED = "related"
connection_table = {} # (src_ip, src_port, dst_ip, dst_port) -> State
POLICY = [
# (proto, dst_port, src_cidr, action)
("tcp", 22, "10.0.0.0/8", "ACCEPT"),
("tcp", 443, "0.0.0.0/0", "ACCEPT"),
("tcp", 80, "0.0.0.0/0", "ACCEPT"),
("*", None, "0.0.0.0/0", "DROP"),
]
def process_packet(pkt):
key = (pkt.src_ip, pkt.src_port, pkt.dst_ip, pkt.dst_port)
rev = (pkt.dst_ip, pkt.dst_port, pkt.src_ip, pkt.src_port)
# Stateful: allow return traffic for established sessions
if rev in connection_table and connection_table[rev] == State.ESTABLISHED:
return "ACCEPT"
# Policy match for new connections
for proto, dport, src_cidr, action in POLICY:
if matches(pkt, proto, dport, src_cidr):
if action == "ACCEPT" and pkt.flags == "SYN":
connection_table[key] = State.NEW
return action
return "DROP"
04Intrusion Detection and Prevention Systems (IDS/IPS)
A firewall enforces policy — what is allowed. An Intrusion Detection System (IDS) detects anomalous or malicious behavior within allowed traffic. An IPS extends this to block detected threats in real time.
Signature-based detection
The dominant approach: maintain a database of known attack patterns (signatures). Snort and Suricata operate this way. A signature might match on a specific byte sequence in a packet payload (e.g., the EternalBlue SMB exploit has a distinctive payload structure), a combination of flags and port, or an HTTP header pattern.
# Snort rule example (simplified)
alert tcp $EXTERNAL_NET any -> $HOME_NET 445 (
msg:"EternalBlue SMBv1 Exploit";
flow:to_server,established;
content:"|00 00 00 90|"; depth:4; # NetBIOS session length
content:"|FF 53 4D 42|"; distance:0; # SMB magic bytes
content:"|72 00|"; distance:52; # SMB command: Negotiate
sid:41978; rev:2;
)
Signature-based detection is fast and low-false-positive for known attacks, but blind to zero-days — novel attacks with no signature.
Anomaly-based detection
Establish a baseline of normal behavior (traffic volumes, protocol distributions, connection patterns, user activity rhythms), then alert on deviations. A machine learning classifier trained on network flows might flag: a workstation suddenly scanning 254 hosts (lateral movement), a server exfiltrating 10GB at 2am (data exfiltration), or a user logging in from two geographically impossible locations (credential stuffing or account takeover).
The fundamental tension: too sensitive → alert fatigue (analysts ignore everything); too permissive → miss real attacks. The base rate fallacy bites hard here: even a 99% accurate IDS, applied to a network where 0.01% of events are malicious, produces mostly false positives.
Host-based vs. network-based
Network IDS (NIDS) taps the wire (or a switch SPAN port) and analyzes traffic. It sees the network but not process internals. Host-based IDS (HIDS) runs on the endpoint: it watches file system changes (tripwire), system calls, process creation, and network sockets. HIDS survives encrypted traffic but requires an agent on every host.
SIEM and the kill chain
Modern Security Operations Centers (SOCs) aggregate logs from firewalls, IDS, endpoint agents, and cloud services into a Security Information and Event Management (SIEM) system (Splunk, Elastic SIEM). Analysts use the Lockheed Martin Cyber Kill Chain or MITRE ATT&CK framework to map detected events to attacker stages: Reconnaissance → Weaponization → Delivery → Exploitation → Installation → Command & Control → Actions on Objectives. Detecting early in the chain (e.g., at delivery) is far cheaper than detecting at the final stage.
# Simplified anomaly detection: flag hosts scanning many IPs
from collections import defaultdict
import time
SCAN_THRESHOLD = 20 # distinct dest IPs in WINDOW seconds
WINDOW = 60 # seconds
# contact_map[src_ip] = {(timestamp, dst_ip), ...}
contact_map = defaultdict(list)
def process_flow(src_ip, dst_ip, timestamp):
"""Call for each observed connection attempt."""
contacts = contact_map[src_ip]
# Prune old entries outside the sliding window
cutoff = timestamp - WINDOW
contact_map[src_ip] = [(t, d) for t, d in contacts if t > cutoff]
contact_map[src_ip].append((timestamp, dst_ip))
distinct_dsts = {d for _, d in contact_map[src_ip]}
if len(distinct_dsts) >= SCAN_THRESHOLD:
alert(f"Possible port scan / lateral movement from {src_ip}: "
f"{len(distinct_dsts)} distinct destinations in {WINDOW}s")
def alert(msg):
print(f"[ALERT {time.strftime('%H:%M:%S')}] {msg}")
05Malware — Taxonomy, Mechanisms, and Defense
Malware (malicious software) is the primary vector through which attackers compromise systems. Understanding its mechanisms is prerequisite to defending against it.
Taxonomy
- Virus: code that self-replicates by inserting itself into other executables or documents. Execution of the host triggers the virus. Classic example: CIH (Chernobyl), which overwrote the first megabyte of disk (BIOS flash + MBR) on April 26.
- Worm: self-replicates across networks without needing a host file. WannaCry (2017) used EternalBlue to propagate over SMBv1, encrypting files on each infected machine. Robert Morris's 1988 worm is the canonical example.
- Trojan: masquerades as legitimate software. Does not self-replicate. The attacker tricks users into installing it (phishing, supply-chain injection).
- Rootkit: achieves persistent, stealthy access, often by modifying the OS kernel or bootloader. A kernel rootkit can hide processes, files, and network connections by hooking system calls. Bootkit variants (e.g., Necurs) persist across OS reinstalls by infecting the MBR or UEFI.
- Ransomware: encrypts victim files and demands payment for the decryption key. WannaCry, NotPetya, REvil. NotPetya (2017) was particularly destructive because its decryption infrastructure was fake — it was a wiper disguised as ransomware, attributed to GRU.
- Spyware/Stalkerware: monitors user activity (keylogging, screenshotting, exfiltrating credentials) without visibility to the user.
- Botnet: a network of compromised machines (bots / zombies) controlled by a command-and-control (C2) server. Used for spam, DDoS, credential stuffing, cryptocurrency mining.
Common propagation techniques
- Phishing: email with malicious attachment (macro-enabled Office doc, PDF exploit) or link to credential-harvesting page.
- Drive-by download: victim visits a compromised web page; browser exploit (historically Flash, Java, JS engine bugs) installs malware silently.
- Supply-chain: compromise upstream software (build servers, update mechanisms). SolarWinds: attackers injected a backdoor into the Orion software build process, which was then signed and distributed to ~18,000 customers.
- Credential reuse / stuffing: use leaked username/password pairs from breached databases against other services.
Evasion techniques
- Polymorphism: malware that mutates its own code (XOR key changes, junk instructions inserted) with each replication, defeating signature-based AV.
- Metamorphism: full code rewriting — functionally equivalent but structurally different bytecode.
- Fileless malware: lives entirely in memory (injected into legitimate processes via Process Hollowing or reflective DLL injection), never touching disk. Evades file-system-based AV.
- Living off the Land (LotL): uses legitimate OS tools (PowerShell, WMI, certutil, mshta) to execute malicious payloads. No custom binary to detect.
Defense in depth
No single control stops all malware. Defense in depth layers: patching (close the vulnerability exploited for initial access), email filtering (reduce phishing surface), endpoint detection and response (EDR, behavioral analysis), application allowlisting (only signed, known-good executables run), least privilege (malware runs with the victim user's privilege level), segmentation (limits lateral movement), and offline backups (ransomware recovery).
# Simplified behavioral detection: flag use of LOLBins with suspicious args
import subprocess
import re
# Patterns associated with malicious PowerShell use
LOLBIN_RULES = [
("powershell", re.compile(
r"-enc|-EncodedCommand|-windowstyle hidden|-nop|-bypass",
re.IGNORECASE)),
("certutil", re.compile(
r"-urlcache|-decode", re.IGNORECASE)),
("mshta", re.compile(
r"http://|https://|javascript:", re.IGNORECASE)),
]
def check_process_creation(executable: str, args: str) -> bool:
"""Return True if the invocation looks suspicious."""
exe_lower = executable.lower()
for lolbin, pattern in LOLBIN_RULES:
if lolbin in exe_lower and pattern.search(args):
print(f"[ALERT] Suspicious {lolbin} invocation: {args[:120]}")
return True
return False
# Example: would flag this real-world malware pattern
check_process_creation(
"powershell.exe",
"-nop -windowstyle hidden -EncodedCommand JABzAD0ATgBlAHcALQBP..."
)
06Access Control Models — DAC, MAC, and RBAC
Access control is the mechanism by which a system decides whether a subject (process, user) is permitted to perform an operation on an object (file, socket, memory region). Three classical models form the theoretical foundation.
The reference monitor
All access control implementations must embody a reference monitor: a tamper-proof, always-invoked, verifiable mechanism that mediates every access. The Linux kernel's capable() checks, the Windows Security Reference Monitor, and the JVM's SecurityManager are all reference monitor implementations.
Discretionary Access Control (DAC)
In DAC, the owner of a resource decides who can access it. Unix permissions (rwxrwxrwx) are DAC: the file owner sets the permission bits. The conceptual model is an Access Control Matrix — rows are subjects, columns are objects, cells contain permitted operations. In practice, this is stored as an Access Control List (ACL, per-object) or Capability List (per-subject).
Weakness: if a process running as a user is compromised, it can access everything that user can access — including modifying permissions on their own files, potentially allowing malware to hide itself. DAC is purely voluntary: the OS enforces what the owner requests, not what is safe.
Mandatory Access Control (MAC)
In MAC, a system-wide policy overrides owner discretion. The classic formalization is the Bell-LaPadula model (designed for military classification):
- Simple Security Property ("no read up"): a subject at classification L cannot read objects at classification > L.
- ★-Property ("no write down"): a subject at classification L cannot write to objects at classification < L (prevents leaking classified info to lower levels).
SELinux and AppArmor implement MAC on Linux. SELinux assigns labels (contexts) to every process, file, and socket, and enforces a policy database specifying which source contexts can perform which operations on which target contexts. A compromised Apache process with label httpd_t cannot access /etc/shadow (labeled shadow_t) even if running as root.
Role-Based Access Control (RBAC)
RBAC (NIST standard) assigns permissions to roles, not directly to users. Users are assigned to roles. A hospital system might have roles: nurse, physician, pharmacist, admin. A nurse has read access to patient records but cannot prescribe. RBAC is practical for large organizations where individual-user ACLs would be unmanageable.
Extensions: ABAC (Attribute-Based Access Control) generalizes RBAC by using arbitrary attributes (department=cardiology, clearance=secret, time < 18:00) in policy rules, enabling fine-grained, context-sensitive access control — used in modern cloud IAM systems (AWS IAM policies).
The Principle of Least Privilege
Across all models, the fundamental design principle is least privilege: every subject should have the minimum access rights necessary for its legitimate function. A database backup process needs read access to the DB data directory — not root, not write access, not network access to arbitrary hosts. Violations of least privilege dramatically amplify the impact of any single compromise.
# SELinux policy excerpt (textual policy language)
# Allow httpd daemon to read its config files only
type httpd_t; # process domain for apache
type httpd_config_t; # type for /etc/httpd/ files
# httpd can read config files
allow httpd_t httpd_config_t : file { read getattr open };
# httpd CANNOT write config files (even as root — MAC overrides DAC)
# The following would be DENIED without explicit allow:
# allow httpd_t httpd_config_t : file { write };
# httpd can accept TCP connections on http_port_t (80, 443)
allow httpd_t http_port_t : tcp_socket { name_bind };
# httpd CANNOT connect outbound to arbitrary hosts
# (prevents SSRF-to-exfiltration)
# -- this rule is intentionally absent --
07Sandboxing — Confining Untrusted Code
A sandbox is an execution environment that restricts what a process can do, even if it is fully compromised. The goal is containment: limit the damage an attacker can cause from within a process by restricting its view of and access to system resources.
Why sandboxing is necessary
Modern applications process untrusted input: browsers render attacker-controlled HTML/JS/WebAssembly; PDF viewers parse attacker-controlled document structure; media players decode attacker-controlled audio/video. Any parsing of untrusted input in a memory-unsafe language (C, C++) is a potential remote code execution (RCE) vulnerability. Sandboxing ensures that even successful exploitation of a parsing bug yields minimal access.
OS-level mechanisms used by sandboxes
-
seccomp(Linux): filters system calls. A process can install a BPF (Berkeley Packet Filter) program that the kernel evaluates for every syscall. Chrome's renderer installs a whitelist of ~50 syscalls (mostly memory management and IPC); any other syscall is killed with SIGSYS. Attackers who achieve RCE in the renderer cannot exfiltrate files (noread), establish network connections (noconnect), or fork processes (nofork). -
Namespaces (Linux): isolate different OS resources per-process.
pidnamespace gives a process its own PID space (process 1 inside the namespace).netnamespace gives its own network stack.mntnamespace its own filesystem mount points.usernamespace maps user IDs. Docker containers are fundamentally collections of namespaces. -
cgroups(Linux): limit resource consumption — CPU, memory, disk I/O, number of processes. Prevents a container from forking-bombing the host. -
Pledge/Unveil (OpenBSD):
pledge()lets a process declare which system call categories it will use; after pledging, attempting an unpledged syscall kills the process.unveil()restricts the filesystem paths a process can see. -
Windows Job Objects + AppContainer: similar isolation primitives on Windows. UWP apps run in AppContainers with restricted capabilities.
Browser sandbox architecture
Chrome's multi-process architecture is the most-studied sandbox deployment. The browser process is privileged (GPU, filesystem, network). Each tab runs in a renderer process with a strict seccomp sandbox — it cannot make OS calls directly and communicates only via Chrome's IPC (Mojo). A GPU process handles rendering. This means exploiting a JavaScript engine bug (RCE in renderer) is insufficient; the attacker must also chain a sandbox escape — exploiting the IPC interface or a kernel bug — to reach the underlying OS.
Limitations of sandboxing
Sandboxes are not impenetrable: they rely on the correctness of the kernel (kernel exploits escape all sandboxes), the IPC boundary (logic bugs in privilege brokers), and the completeness of the policy (forgotten syscall, overly broad path unveil). They reduce attack surface but do not eliminate it. A good threat model quantifies what is still exposed even inside the sandbox.
// Minimal seccomp-BPF sandbox in C: allow only read/write/exit
#include <linux/seccomp.h>
#include <linux/filter.h>
#include <linux/audit.h>
#include <sys/prctl.h>
#include <sys/syscall.h>
#include <stddef.h>
static void install_sandbox(void) {
struct sock_filter filter[] = {
// Load syscall number from seccomp_data
BPF_STMT(BPF_LD | BPF_W | BPF_ABS,
offsetof(struct seccomp_data, nr)),
// Allow read(2)
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_read, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
// Allow write(2)
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_write, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
// Allow exit_group(2)
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, __NR_exit_group, 0, 1),
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
// Kill process for anything else
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),
};
struct sock_fprog prog = {
.len = sizeof(filter) / sizeof(filter[0]),
.filter = filter,
};
prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0);
syscall(__NR_seccomp, SECCOMP_SET_MODE_FILTER, 0, &prog);
}
08Secure System Design Principles
Jerome Saltzer and Michael Schroeder's 1975 paper "The Protection of Information in Computer Systems" laid out eight design principles that remain the gold standard for secure system design. Every one of these principles has been validated by decades of real-world failures.
1. Economy of mechanism (Keep it simple) Complex systems have more potential flaws. Simplicity is a security property. The principle explains why TLS 1.3 dropped legacy cipher suites, why SELinux policies are expressed in a restricted domain-specific language, and why capability-based designs beat ACL forests.
2. Fail-safe defaults (Default deny) Access decisions should default to denial. If a firewall rule is ambiguous or missing, traffic should be blocked, not allowed. AWS IAM starts with zero permissions; you add what you need. Most real-world breaches exploit systems where someone forgot to restrict access — fail-safe defaults make omission safe.
3. Complete mediation (Check every access) Every access to every object must be checked at the time of access — not just at object creation. The confused deputy problem (and CSRF attacks) occur when a check is performed by a trusted intermediary but the attacker can substitute the target without re-triggering the check.
4. Open design (Security through obscurity fails) Security should not depend on the secrecy of the mechanism — only on the secrecy of keys. Kerckhoffs's principle. AES is public; the algorithm is known to all. Security comes from the 256-bit key, not from hiding how AES works. Systems that rely on "security by obscurity" (hidden algorithm, hidden API endpoint) fail catastrophically when the obscured detail is discovered — and it always is.
5. Separation of privilege Require multiple conditions to be satisfied for access. Two-factor authentication is an instance: an attacker needs both your password and your physical device. Split-knowledge procedures for nuclear launch codes. Multi-party authorization for large financial transfers.
6. Least privilege (already discussed under access control)
7. Least common mechanism Minimize shared state between subjects with different trust levels. A shared library function used by both a privileged process and an unpredictable user process is a covert channel candidate. Microkernel designs and containerization pursue this principle by reducing shared OS machinery.
8. Psychological acceptability Security mechanisms must not be so burdensome that users route around them. Password policies that require 32-character random strings and rotation every 30 days cause users to write passwords on sticky notes. NIST SP 800-63B now recommends no mandatory rotation, length over complexity, and password managers. Unusable security is no security.
Defense in depth and threat modeling
Beyond Saltzer-Schroeder, modern secure design adds: defense in depth (no single control is trusted to hold; layer independent controls), threat modeling (STRIDE: Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege — systematically enumerate threats before writing code), and assume breach (design systems assuming the attacker is already inside some layer; limit blast radius).
# STRIDE threat model stub — data flow diagram annotated with threats
"""
System: User Login Service
DFD components:
User Browser --> [HTTPS] --> Auth Service --> [TLS] --> Database
STRIDE analysis:
S Spoofing:
- Attacker impersonates User: mitigate with passwords + 2FA
- Attacker spoofs Auth Service: mitigate with TLS + valid cert
T Tampering:
- Attacker modifies JWT token: mitigate with HMAC-SHA256 signature
- SQL injection in login form: mitigate with parameterized queries
R Repudiation:
- User denies performing action: mitigate with append-only audit log
(immutable, stored off-system)
I Information Disclosure:
- Password in DB plaintext: mitigate with bcrypt/Argon2 hashing
- Error message leaks username: use generic "invalid credentials"
D Denial of Service:
- Credential stuffing / brute: rate limiting + account lockout
- Slow-loris / large body: request timeout + body size limit
E Elevation of Privilege:
- JWT alg:none attack: explicitly validate and fix algorithm
- Mass assignment (user.is_admin in POST body): allowlist writable fields
"""
Key insights
- ✦TLS 1.3's forward secrecy means compromising a server's long-term private key today does not decrypt traffic recorded yesterday — ephemeral key pairs are generated fresh for every session.
- ✦The PKI trust model is only as strong as its weakest CA: any of ~200 trusted CAs can issue a certificate for any domain, which is why Certificate Transparency logs are essential for detecting mis-issuance.
- ✦Stateful firewalls track connection state so return traffic for outbound sessions is automatically permitted — but this also means an attacker who initiates an outbound connection gets unrestricted return traffic.
- ✦The base-rate fallacy makes anomaly-based IDS inherently prone to false positives: a 99% accurate detector on a network where 0.01% of events are malicious produces ~100 false alerts for every real one.
- ✦Malware authors use Living off the Land (LotL) techniques — abusing legitimate OS tools like PowerShell, WMI, and certutil — precisely because these binaries are trusted, signed, and whitelisted by most defenses.
- ✦MAC (SELinux/AppArmor) overrides DAC: a process running as root with the wrong SELinux context is denied access by the kernel before the DAC check even matters.
- ✦Sandboxes reduce but cannot eliminate attack surface — they shift the attacker's requirement from 'exploit the parser' to 'exploit the parser AND escape the sandbox,' dramatically raising the cost of a full compromise.
- ✦Fail-safe defaults (default deny) convert every omission into a safe outcome; most real-world breaches exploit systems configured permissively because someone forgot to restrict something.
Common pitfalls
- ⚠Confusing authentication (proving who you are) with authorization (what you are allowed to do): a valid TLS certificate proves server identity but says nothing about whether that server should be trusted to process your request.
- ⚠Treating TLS termination at a load balancer as sufficient: traffic between the load balancer and backend servers is often plaintext within the data center — an attacker with internal network access bypasses TLS entirely.
- ⚠Self-signed certificates in production: without a CA-signed cert, clients cannot distinguish the legitimate server from an impersonator; automated trust-all-certs code in scripts is a frequent source of MITM vulnerability.
- ⚠Firewall rules that allow outbound traffic promiscuously: egress filtering is as important as ingress — malware needs to phone home, and permissive egress lets it.
- ⚠Using ACL checks at object creation time (cache-then-use): the TOCTOU (Time Of Check To Time Of Use) race condition arises when an access check result is cached and re-used after conditions have changed.
- ⚠Conflating encryption with security: encrypted data can still be tampered with without an authentication tag (AEAD); unauthenticated encryption (pure AES-CBC) is vulnerable to padding oracle attacks and bit-flipping.
- ⚠Sandbox overconfidence: a seccomp filter blocks direct syscalls but a kernel vulnerability can still be triggered through an allowed syscall path; containment is not equivalence to isolation from kernel bugs.
- ⚠Password hashing with MD5 or SHA-1: these are fast hash functions designed for integrity, not password storage; use bcrypt, scrypt, or Argon2 which are deliberately slow and memory-hard.
Practice
- 01hard
Implement a TLS certificate chain validator in Python using the `cryptography` library. Given a list of DER-encoded certificates (leaf, intermediate, root), verify: (1) each certificate's signature is valid under the next certificate's public key, (2) the root certificate is self-signed, (3) none of the certificates have expired, (4) the leaf certificate contains a given hostname in its Subject Alternative Names. Do NOT use `ssl.match_hostname` or any high-level validation — implement the chain walk yourself.
- 02medium
Using Linux `iptables` (or `nftables`), design a firewall ruleset for a server that: (a) runs a public HTTPS web server, (b) is administered via SSH from a single management subnet 10.0.0.0/24, (c) needs to make outbound DNS queries and HTTPS requests for Let's Encrypt ACME challenges, and (d) should drop all other traffic. Write the complete rule set with a default DROP policy on all chains. Explain why order matters for each rule.
- 03medium
Threat-model a simple REST API that handles user authentication and stores personal data. Draw a data flow diagram (DFD) with at least 4 components (client, API server, database, cache). Apply STRIDE to each edge and component, identify at least 8 distinct threats, and for each threat propose a concrete mitigation. For two of the threats, implement the mitigation in code (e.g., parameterized queries for SQL injection, Argon2 hashing for credential storage).
- 04hard
Write a minimal sandboxed subprocess launcher in Python that: (a) forks a child process, (b) uses `prctl(PR_SET_NO_NEW_PRIVS)` and installs a seccomp-BPF filter (via the `pyseccomp` library or raw `ctypes`) that allows only `read`, `write`, `exit_group`, and `brk`, (c) then exec's a provided command. Verify that the sandboxed process cannot open files (`open` returns EPERM / the process is killed by the kernel). Discuss: what class of attacks does this stop? What remains possible?
- 05medium
Research and explain the 2014 Heartbleed vulnerability (CVE-2014-0160) at the source-code level. (a) Find and read the original vulnerable code in OpenSSL's `t1_lib.c`. (b) Explain precisely which memory safety invariant was violated and why the TLS record layer design allowed it. (c) Explain why proper AEAD encryption did NOT protect against Heartbleed — what security property does encryption provide that Heartbleed bypassed? (d) Describe the minimal correct fix.
- 06easy
Implement a simple Role-Based Access Control (RBAC) engine in Python. Define: (a) a set of resources (files, APIs), (b) a set of permissions (read, write, delete, execute), (c) a set of roles (admin, editor, viewer) with assigned permissions, (d) a set of users with assigned roles. Write an `authorize(user, resource, permission)` function. Then extend it to support role inheritance (editor inherits all viewer permissions) and implement `audit_log` that records every access decision with timestamp, subject, object, action, and decision. Verify your RBAC correctly rejects escalation attempts.
Resources
The primary source. Read sections 2 (Goals), 4 (Handshake), and Appendix C (Implementation Notes). Dense but authoritative.
The most comprehensive academic textbook on computer security. Covers access control models, cryptography, network security, and formal methods. The reference for courses at UC Davis and many others.
Saltzer and Schroeder 1975. The original statement of the eight security design principles. Short, rigorous, and still directly applicable 50 years later.
Full MIT course with lecture notes, labs (buffer overflows, TLS, sandboxing, web security), and reading list. The labs are the best way to develop hands-on intuition.
Standard undergraduate textbook used at hundreds of universities. Thorough coverage of TLS, PKI, firewalls, and intrusion detection.
Exhaustive taxonomy of adversary tactics and techniques with real-world malware examples. The standard reference for threat modeling and IDS rule development.
The most practical and current treatment of TLS deployment, certificate management, and the WebPKI. Essential for anyone deploying TLS in production.
Official kernel documentation for seccomp BPF. Includes the exact filter semantics and a worked example. Reference this alongside the Chrome sandbox design doc.
Authoritative US government guidance on password and authenticator requirements. The source of modern password policy recommendations (no mandatory rotation, length over complexity).
Invaluable practical complement to the theory. Covers SQL injection, XSS, CSRF, authentication flaws, and session management attacks with worked examples.
Checkpoints — you can now…
- ✓You can trace a TLS 1.3 handshake step by step, naming each message, the cryptographic operation it performs, and what the output of each step is used for in subsequent steps.
- ✓You can explain why TLS 1.3's use of ephemeral Diffie-Hellman key exchange provides forward secrecy, and give a concrete scenario where forward secrecy prevents compromise of past traffic.
- ✓You can walk a certificate chain from a leaf certificate to a root CA, verifying each signature, and explain what Certificate Transparency logs add beyond chain validation.
- ✓You can write a stateful iptables/nftables ruleset for a multi-service server, explaining the reasoning behind the ordering of each rule and the consequence of getting it wrong.
- ✓You can apply the STRIDE threat model to a simple system's data flow diagram and propose concrete mitigations for at least six distinct threat categories.
- ✓You can compare DAC, MAC, and RBAC, give a real OS example of each, and explain a scenario where MAC prevents an attack that DAC cannot — even for a root-level process.
- ✓You can explain the security guarantees and limitations of a seccomp sandbox: what system calls it restricts, how it interacts with kernel vulnerabilities, and why sandbox escape is harder but not impossible.
- ✓You can articulate all eight Saltzer-Schroeder principles by name, give a concrete example of each from real systems, and identify which principle was violated in a described historical breach.