Modern Networking

Wi-Fi is a shared medium: every device in a cell competes for the same radio channel, and unlike a wire, you cannot sense a collision while it is happening (because your transmitter drowns out the weak incoming signal). This is why Wi-Fi uses CSMA/CA — Carrier Sense Multiple Access with Collision Avoidance rather than detection.

The CSMA/CA dance:

1. Before transmitting, the station senses the channel. If it is idle for at least a DIFS (DCF Inter-Frame Space, ~34 µs in 802.11g), it enters the backoff phase.
2. The station draws a random backoff counter from [0, CW−1] where CW is the contention window (starts at 15, doubles on collision, up to 1023).
3. The counter decrements only when the channel is idle; it freezes when the channel is busy. When it hits zero, transmission starts.
4. The receiver, if it successfully receives the frame, sends an ACK after a SIFS (Short IFS, ~16 µs). SIFS < DIFS ensures the ACK wins channel access over new transmissions.
5. If no ACK arrives, the sender assumes a collision, doubles CW, and repeats.

Hidden node problem: Station A and C are both in range of B but not each other. A senses the channel as clear (it cannot hear C) and transmits while C is already transmitting to B — a collision at B that neither A nor C detects. RTS/CTS (Request to Send / Clear to Send) solves this: A sends a tiny RTS to B; B broadcasts a CTS that C hears; C defers.

802.11 physical channels: The 2.4 GHz band has 14 overlapping 22 MHz channels but only 3 non-overlapping (1, 6, 11 in North America). The 5 GHz band has ~24 non-overlapping 20 MHz channels, which is why 5 GHz Wi-Fi is far less congested. 802.11ax (Wi-Fi 6) adds OFDMA — Orthogonal Frequency Division Multiple Access — borrowed from LTE, allowing the AP to subdivide a channel into Resource Units and serve multiple clients simultaneously, like a cellular base station.

Beamforming and MIMO: 802.11n/ac/ax APs use Multiple Input Multiple Output (MIMO) with 2–8 antennas. Spatial multiplexing sends independent data streams on separate spatial paths; maximum throughput scales linearly with min(transmit antennas, receive antennas). MU-MIMO (Multi-User MIMO) lets a single AP beam different streams to different clients simultaneously.

# Simulate exponential backoff for 802.11 CSMA/CA
import random

CW_MIN, CW_MAX = 15, 1023

def transmit(frame_id: int, max_retries: int = 7) -> int:
    """Returns number of attempts needed. Raises if all retries exhausted."""
    cw = CW_MIN
    for attempt in range(max_retries + 1):
        backoff = random.randint(0, cw)
        # Simulate: collision occurs if another station picks backoff <= ours
        # (simplified: independent Bernoulli collision with prob p=0.3)
        collision = random.random() < 0.3 if attempt > 0 else False
        if not collision:
            print(f"Frame {frame_id}: transmitted on attempt {attempt+1}, backoff={backoff}")
            return attempt + 1
        cw = min(cw * 2 + 1, CW_MAX)  # double CW on collision
    raise RuntimeError(f"Frame {frame_id}: max retries exceeded")

for i in range(5):
    transmit(i)

Cellular networking solves a harder version of the Wi-Fi problem: you are moving at highway speed, the cell boundaries are kilometers apart, there are thousands of concurrent users per tower, and you need seamless handoff between cells and between network generations. The solution is a beautiful combination of radio engineering, distributed systems, and protocol design.

LTE (4G) architecture has three domains:

• UE (User Equipment): your phone.
• E-UTRAN (Evolved UTRAN): the radio access network, composed of eNodeBs (base stations) that communicate directly with each other over the X2 interface (enabling fast handover) and with the core via the S1 interface.
• EPC (Evolved Packet Core): the 'network brain'. Key nodes: MME (Mobility Management Entity) tracks UE location and manages attach/detach; S-GW (Serving Gateway) is the data-plane anchor during handover; P-GW (PDN Gateway) is the internet exit point; HSS (Home Subscriber Server) is the authentication database.

OFDMA and SC-FDMA: LTE divides spectrum into 15-kHz subcarriers. The downlink (tower→phone) uses OFDMA: the scheduler at the eNodeB assigns Resource Blocks (180 kHz × 1 ms time slots) to each UE based on their channel quality reports (CQI). A UE with a good channel gets 64-QAM; one near the cell edge gets QPSK. The uplink uses SC-FDMA (Single Carrier FDMA), which has lower peak-to-average power ratio — important for battery life.

Handover procedure:

1. UE measures reference signal strength (RSRP) from neighbor cells.
2. UE reports measurements to serving eNodeB.
3. Serving eNodeB sends Handover Request to target eNodeB (X2 interface).
4. Target eNodeB admits the request, sends Handover Command back.
5. UE detaches from old cell, connects to new cell — the old eNodeB forwards buffered packets to the new one during the brief outage.
6. Path switch: target eNodeB notifies MME, which updates S-GW to send packets to the new eNodeB.

Total X2-based handover time: ~50ms. From user's perspective, latency blips once.

5G New Radio (NR): 5G introduces massive MIMO (up to 256 antennas per base station), mmWave bands (28/39 GHz with centimeter wavelength and extreme bandwidth but short range), network slicing (virtual networks with different SLA guarantees carved out of shared infrastructure via SDN/NFV), and ultra-low latency URLLC (< 1ms) for industrial IoT. The 5G core moves to a Service-Based Architecture (SBA): instead of point-to-point interfaces between fixed nodes, all functions (AMF, SMF, UPF, etc.) expose REST/HTTP APIs and communicate via a service bus.

Spectrum as a resource: cellular operators pay billions of dollars per MHz of licensed spectrum. The entire discipline of wireless network design is essentially about extracting the theoretical Shannon capacity C = B·log₂(1 + SNR) from every Hz and every millisecond, under mobility, interference, and fairness constraints.

# Compute Shannon capacity for a cellular link
import math

def shannon_capacity_mbps(
    bandwidth_mhz: float,
    snr_db: float
) -> float:
    """Shannon-Hartley theorem: C = B * log2(1 + SNR)"""
    snr_linear = 10 ** (snr_db / 10)
    capacity_bps = bandwidth_mhz * 1e6 * math.log2(1 + snr_linear)
    return capacity_bps / 1e6  # Mbps

# LTE: 20 MHz channel, excellent SNR (near tower)
print(f"LTE near tower (SNR=30dB): {shannon_capacity_mbps(20, 30):.1f} Mbps")
# LTE: 20 MHz, cell edge
print(f"LTE cell edge  (SNR=5dB):  {shannon_capacity_mbps(20, 5):.1f} Mbps")
# 5G mmWave: 400 MHz channel, very good SNR (short range)
print(f"5G mmWave      (SNR=25dB): {shannon_capacity_mbps(400, 25):.1f} Mbps")

The core insight of a CDN is geographic proximity. Light in fiber travels at ~200,000 km/s, so a round-trip from New York to London takes at minimum ~70ms. For a Netflix stream, every TCP segment requires acknowledgment, so this RTT directly limits throughput. CDNs solve this by placing copies of content (and increasingly, origin logic) in hundreds or thousands of Points of Presence (PoPs) worldwide, so most users are within 10–20ms of a CDN server.

CDN request routing: How does your browser get routed to the nearest CDN node? Two main mechanisms:

1. DNS-based routing: The CDN's authoritative DNS server looks at the resolver's IP address (approximating your location) and returns the IP of the nearest PoP. TTL is kept short (30–60s) to allow rapid failover.
2. Anycast routing: The CDN advertises the same IP block from all PoPs via BGP. Your ISP's routing tables naturally route packets to the topologically closest PoP. Used by Cloudflare, Google's 1.1.1.1, etc.

Consistent hashing for cache distribution: A CDN PoP may have dozens of cache servers. When a request arrives, you need to pick a server that likely already has the object (to maximize cache hit rate) without keeping a central directory. Consistent hashing solves this:

Map both servers and objects onto a ring of 2^32 positions (using SHA-1 or similar). For each request, hash the URL and walk clockwise until you hit a server. If a server is added or removed, only ~1/N of keys need to be remapped (N = number of servers), versus N-1/N for modular hashing.

Practical improvement: use virtual nodes — each physical server occupies K positions on the ring. This achieves more uniform load distribution even with heterogeneous server capacities.

Tiered caching: Modern CDNs (Fastly, Cloudflare, Akamai) use 2–3 tier hierarchies: edge servers close to users, regional aggregation nodes, and origin shield nodes that protect the origin from the thundering herd. Cache miss propagates up the hierarchy; hit at any tier is a win.

CDN for dynamic content: Not all CDN work is static file caching. Modern CDNs run application logic at the edge (Cloudflare Workers, Fastly Compute, Lambda@Edge). These V8 isolate-based execution environments can run JavaScript/Wasm at each PoP, enabling server-side rendering, A/B testing, authentication, and even database queries from globally distributed edge nodes — a paradigm called the "edge computing" model.

import hashlib
from bisect import insort, bisect_right

class ConsistentHashRing:
    def __init__(self, virtual_nodes: int = 100):
        self.vn = virtual_nodes
        self.ring: list[int] = []      # sorted hash positions
        self.pos_to_server: dict[int, str] = {}

    def _hash(self, key: str) -> int:
        return int(hashlib.md5(key.encode()).hexdigest(), 16)

    def add_server(self, server: str):
        for i in range(self.vn):
            pos = self._hash(f"{server}#{i}")
            insort(self.ring, pos)
            self.pos_to_server[pos] = server

    def remove_server(self, server: str):
        for i in range(self.vn):
            pos = self._hash(f"{server}#{i}")
            self.ring.remove(pos)
            del self.pos_to_server[pos]

    def get_server(self, key: str) -> str:
        if not self.ring:
            raise ValueError("No servers in ring")
        pos = self._hash(key)
        idx = bisect_right(self.ring, pos) % len(self.ring)
        return self.pos_to_server[self.ring[idx]]

ring = ConsistentHashRing(virtual_nodes=150)
for s in ["edge-us-east", "edge-eu-west", "edge-ap-southeast"]:
    ring.add_server(s)

for url in ["/video/cats.mp4", "/img/logo.png", "/api/trending"]:
    print(f"{url:30s} -> {ring.get_server(url)}")

The traditional network has its control plane (the logic that decides where to forward packets) and data plane (the hardware that actually does the forwarding) fused together inside each individual router. Every router runs OSPF, BGP, or similar protocols independently, converging on a global routing table through distributed message passing. This worked fine for decades but created problems at scale: network changes require touching every device, bugs are hard to diagnose because state is fragmented across hundreds of routers, and optimizing global traffic is nearly impossible when each router makes local decisions.

SDN's key insight: separate the control and data planes. A logically centralized controller maintains a global view of the network topology and programs the forwarding tables of all switches. Each switch becomes a simple, fast forwarding device — a "dumb pipe" with a programmable match-action table.

OpenFlow protocol: The seminal SDN interface (now largely supplanted by P4 and vendor APIs, but foundational). An OpenFlow switch has a flow table: each entry is a (match, action, priority, counters) tuple. Match fields can include any combination of L2 (MAC, VLAN), L3 (IP src/dst), L4 (TCP/UDP ports), or even tunneling headers. Actions: forward out port(s), drop, modify header fields, send to controller.

When a packet arrives and no flow entry matches, it is sent to the controller (PacketIn). The controller installs a new rule (FlowMod) and potentially injects a forwarding decision (PacketOut). For bulk traffic, the controller proactively installs rules; for rare traffic, it handles reactively.

OpenFlow message types:

• OFPT_FEATURES_REQUEST/REPLY: controller learns switch capabilities.
• OFPT_PACKET_IN: switch → controller, unknown packet.
• OFPT_FLOW_MOD: controller → switch, install/delete/modify flow entry.
• OFPT_PACKET_OUT: controller → switch, send this packet.
• OFPT_STATS_REQUEST/REPLY: telemetry (byte/packet counters per flow).

SDN controllers: Production SDN deployments use controllers like ONOS (Open Network Operating System), OpenDaylight, or cloud-vendor proprietary systems (Google's Orion for their WAN, Facebook's FBOSS). These are distributed systems themselves — they must be highly available and must handle controller failover without dropping traffic.

East-West APIs and network apps: The controller exposes a northbound API to network applications: a traffic engineering app might compute optimal ECMP weights using a global topology graph and push them down; a security app might detect a DDoS flow and install a drop rule across the entire network in seconds; a telemetry app might mirror all SYN packets to an IDS. This is the real power of SDN: network behavior becomes programmable software.

Limitations of logical centralization: The controller is a potential bottleneck and single point of failure. In practice, controllers are deployed as distributed clusters with consistent state (using Raft or Paxos). The 'logical centralization' means a unified view, not a single physical node. Also, flow table capacity on real hardware (TCAM) is limited — typically 4K–16K entries — which constrains how granular the flow policies can be.

# Pseudocode for a simple SDN reactive controller loop
# (matches the RYU/POX controller model)

class SimpleL2Controller:
    def __init__(self):
        # mac_table[switch_id][mac_addr] = port_number
        self.mac_table: dict[int, dict[str, int]] = {}

    def on_packet_in(self, switch_id: int, in_port: int,
                     src_mac: str, dst_mac: str, payload: bytes):
        # Learn: src MAC arrived on in_port of this switch
        self.mac_table.setdefault(switch_id, {})
        self.mac_table[switch_id][src_mac] = in_port

        if dst_mac in self.mac_table[switch_id]:
            out_port = self.mac_table[switch_id][dst_mac]
            # Install a proactive rule so future packets bypass the controller
            self.install_flow(
                switch_id,
                match={"eth_dst": dst_mac},
                action={"output": out_port},
                idle_timeout=30
            )
            self.packet_out(switch_id, payload, out_port)
        else:
            # Destination unknown — flood all ports except in_port
            self.packet_out(switch_id, payload, port=OFPP_FLOOD)

    def install_flow(self, switch_id, match, action, idle_timeout):
        # Sends OFPT_FLOW_MOD to the switch via OpenFlow channel
        openflow_send(switch_id, FlowMod(
            match=match, action=action,
            idle_timeout=idle_timeout, priority=10
        ))

    def packet_out(self, switch_id, payload, port):
        openflow_send(switch_id, PacketOut(payload=payload, port=port))

OpenFlow fixed the set of header fields that could be matched (Ethernet, IP, TCP/UDP). But what if you want to match on VXLAN tunnel IDs, or implement a custom telemetry protocol, or run in-network machine learning inference? P4 (Programming Protocol-independent Packet Processors) is a domain-specific language that lets you program the data plane itself — defining custom packet headers, parsers, and match-action pipelines that run at line rate on programmable switch ASICs (like Intel's Tofino) or FPGAs.

A P4 program has four components:

1. Header definitions: describe the packet format — which fields exist, in what order, how wide.
2. Parser: a state machine that extracts headers from the raw byte stream.
3. Ingress/Egress match-action pipelines: a sequence of tables, each with a match type (exact, ternary, LPM), a key, and a set of actions.
4. Deparser: reassembles the (possibly modified) headers back into a byte stream.

P4 enables in-network computing: instead of sending packets to servers for processing, you perform computation at the switch — counting distinct flows (hyperloglog in P4), load balancing with power-of-two-choices, implementing RDMA over Converged Ethernet (RoCE) retransmission at the switch, or running In-band Network Telemetry (INT) where packets collect latency and queue depth stamps at each hop.

// Minimal P4 example: parse Ethernet+IPv4, forward by IPv4 LPM
header ethernet_t {
    bit<48> dst;  bit<48> src;  bit<16> ether_type;
}
header ipv4_t {
    bit<4>  version; bit<4>  ihl;  bit<8>  tos;
    bit<16> len;     bit<16> id;   bit<3>  flags;
    bit<13> frag_off; bit<8> ttl;  bit<8>  proto;
    bit<16> checksum; bit<32> src; bit<32> dst;
}
struct headers_t { ethernet_t eth; ipv4_t ipv4; }

parser MyParser(packet_in pkt, out headers_t hdr) {
    state start {
        pkt.extract(hdr.eth);
        transition select(hdr.eth.ether_type) {
            0x0800: parse_ipv4;
            default: accept;
        }
    }
    state parse_ipv4 { pkt.extract(hdr.ipv4); transition accept; }
}

control MyIngress(inout headers_t hdr,
                  inout standard_metadata_t sm) {
    action ipv4_forward(bit<9> port) {
        sm.egress_spec = port;
        hdr.ipv4.ttl = hdr.ipv4.ttl - 1;
    }
    action drop() { mark_to_drop(sm); }

    table ipv4_lpm {
        key = { hdr.ipv4.dst: lpm; }
        actions = { ipv4_forward; drop; }
        default_action = drop();
    }
    apply { if (hdr.ipv4.isValid()) ipv4_lpm.apply(); }
}

Traditionally, a carrier's or enterprise's network is studded with dedicated hardware appliances: firewalls (Palo Alto, Checkpoint), load balancers (F5 BIG-IP), WAN optimizers (Riverbed Steelhead), IDS/IPS (Snort on Sourcefire), DPI boxes, NATs, and VPN concentrators. Each cost $10,000–$1,000,000, had a 5-7 year procurement cycle, and had to be racked and cabled in specific locations.

NFV replaces these hardware middleboxes with software Virtual Network Functions (VNFs) running on commodity x86 servers in a data center. The original ETSI NFV whitepaper (2012), authored by 13 of the world's major carriers, argued that virtualizing these functions would reduce capex and opex by 10x while enabling on-demand provisioning, scaling, and geographic flexibility.

NFV architecture (ETSI NFV-MANO):

• VNF: the software implementing a network function (e.g., a software firewall VM).
• NFVI (NFV Infrastructure): the physical compute/storage/networking on which VNFs run — servers, hypervisors, and the SDN-controlled network fabric.
• MANO (Management and Orchestration): three subsystems:
  • VIM (Virtualized Infrastructure Manager): manages compute/storage/network resources (typically OpenStack).
  • VNFM (VNF Manager): lifecycle management of individual VNFs (scaling, healing, upgrades).
  • NFV Orchestrator (NFVO): composes VNFs into end-to-end service chains (SFCs).

Service Function Chaining (SFC): Traffic must traverse VNFs in a specific order: e.g., traffic ingress → DPI → firewall → NAT → egress. SFC uses NSH (Network Service Header, RFC 8300): a 32-bit header prepended to packets that encodes the service path ID and the index of the next function to visit. SDN-controlled forwarders read the NSH and steer packets to the correct VNF instance.

Performance challenges: Software VNFs on general-purpose hardware initially suffered a 10–100x performance gap versus custom ASICs. This drove the development of kernel-bypass networking: DPDK (Data Plane Development Kit, Intel) maps NIC DMA rings directly into userspace, polls instead of using interrupts, and achieves 10–100 Mpps on commodity NICs. SR-IOV (Single Root I/O Virtualization) allows a single physical NIC to expose multiple virtual functions directly to VMs, bypassing the hypervisor's virtual switch entirely.

Containerized VNFs: The industry has further moved from VMs to containers (Docker/Kubernetes) for VNF deployment, reducing overhead and enabling faster autoscaling. Projects like CNCF's Network Service Mesh and OpenShift CNF (Cloud-Native Network Functions) define the container-native NFV stack.

# Sketch: a minimal userspace VNF firewall using raw sockets
# (In production this would use DPDK or AF_XDP for performance)
import socket, struct

# Blocked source IPs (statically configured; real VNF gets policy from MANO)
BLOCKED_IPS = {"192.168.1.100", "10.0.0.5"}

def parse_ip_src(raw_packet: bytes) -> str:
    """Extract source IP from raw IPv4 packet (skipping 14-byte Ethernet header)."""
    ip_header = raw_packet[14:34]
    src = struct.unpack_from('!4s', ip_header, 12)[0]
    return socket.inet_ntoa(src)

def vnf_firewall(iface_in: str, iface_out: str):
    sock_in  = socket.socket(socket.AF_PACKET, socket.SOCK_RAW,
                              socket.htons(0x0800))
    sock_out = socket.socket(socket.AF_PACKET, socket.SOCK_RAW,
                              socket.htons(0x0800))
    sock_in.bind((iface_in, 0))
    sock_out.bind((iface_out, 0))

    while True:
        pkt, _ = sock_in.recvfrom(65535)
        src_ip = parse_ip_src(pkt)
        if src_ip in BLOCKED_IPS:
            print(f"DROP: {src_ip}")   # meter this in production
        else:
            sock_out.send(pkt)          # forward to next VNF / egress

A hyperscale data center might house 100,000–1,000,000 servers in a single building. The network connecting them is one of the most carefully engineered systems in existence. The requirements are unprecedented: bisection bandwidth must be full (any server must be able to communicate with any other at full NIC speed), fault tolerance must be extreme (a link failure should affect almost no traffic), and the cost per port must be minimized.

Why traditional topologies fail: A classic tree network has a single root switch. If all 100,000 servers try to communicate across the root, bandwidth is catastrophically bottlenecked. The root switch would need to be a single device with 100,000-port capacity — physically impossible and economically absurd.

Fat-tree topology (Al-Fares et al., SIGCOMM 2008): The key insight is to use many identical commodity switches (with k ports each) arranged in a three-layer structure to achieve full bisection bandwidth:

• Core layer: (k/2)² switches, fully meshed with the aggregation layer.
• Aggregation layer: k pods, each with k/2 switches.
• Edge layer: k pods, each with k/2 switches directly connected to servers.
• Servers: each pod has (k/2)² servers. Total servers = k³/4.

For k=48 (a 48-port commodity switch): 576 core switches, 1152 aggregation, 1152 edge, 27,648 servers — all with full bisection bandwidth! Every path between any two servers has the same length and the same capacity. You can build this entire network from the same commodity switch, ordered in bulk at drastically lower cost than a custom high-radix router.

ECMP (Equal-Cost Multi-Path): In a fat-tree, there are (k/2)² equal-cost paths between any two servers. ECMP spreads load across them. Standard ECMP hashes the 5-tuple (src IP, dst IP, src port, dst port, protocol) to select a path deterministically for each flow (so packets in a flow stay ordered). This is per-flow load balancing — if flows are many and roughly equal, load is well distributed.

Elephant and mice flows: In practice, a small fraction of flows (elephants, >1 MB, e.g., MapReduce shuffles) carry the vast majority of bytes, while most flows (mice, <10 KB, e.g., RPC calls) are small and latency-sensitive. ECMP can create hash collisions where two elephants share a link, causing congestion. Solutions: CONGA (congestion-aware load balancing), MPTCP (multipath TCP that splits a single connection across multiple paths), and Hedera (OpenFlow-based dynamic elephant flow scheduling).

Beyond fat-tree: Google's Jupiter network uses a multi-stage Clos topology (a generalization of fat-tree). Facebook's Fabric uses a similar approach. Microsoft's Azure uses a 4-layer Clos. Meta's Pluto research explores optical circuit switching as a reconfigurable overlay on top of the packet-switched fabric — extremely high bandwidth optical links that are reconfigured daily based on measured traffic demand.

# Fat-tree switch-count formula and bandwidth analysis

def fat_tree_stats(k: int, link_speed_gbps: float = 10.0):
    """k = port count per switch (must be even)."""
    assert k % 2 == 0, "k must be even"
    n_core  = (k // 2) ** 2
    n_agg   = k * (k // 2)
    n_edge  = k * (k // 2)
    n_hosts = k ** 3 // 4
    n_pods  = k

    # Bisection bandwidth: cut the network in half.
    # Each pod's uplinks to core = (k/2)*(k/2) links.
    # With k pods and cutting in half:
    bisection_bw_tbps = (n_core * link_speed_gbps * 2) / 1000
    # Each core switch has k links, k/2 to each half.

    print(f"Fat-tree k={k} @ {link_speed_gbps}Gbps/link:")
    print(f"  Pods: {n_pods}")
    print(f"  Core switches:        {n_core:>6}")
    print(f"  Aggregation switches: {n_agg:>6}")
    print(f"  Edge switches:        {n_edge:>6}")
    print(f"  Total servers:        {n_hosts:>6}")
    print(f"  Bisection bandwidth: ~{bisection_bw_tbps:.1f} Tbps")
    print(f"  ECMP paths per pair:  {(k//2)**2}")

fat_tree_stats(k=8)
fat_tree_stats(k=48)

A data center runs thousands of tenants (in a public cloud) or hundreds of applications (in a private cloud), each with its own private network address space (often RFC 1918). How do you give each tenant the illusion of an isolated L2 network stretching across potentially thousands of physical servers, without managing millions of VLAN IDs (VLANs only have 12-bit IDs = 4096 VLANs) or flooding the physical network with broadcasts?

VXLAN (Virtual Extensible LAN, RFC 7348) is the dominant solution. It is an overlay encapsulation: virtual machines communicate as if they are on the same Ethernet segment; the physical underlay carries their traffic in UDP packets.

VXLAN packet format: The original Ethernet frame is prepended with:

• Outer Ethernet header (VTEP → VTEP MAC)
• Outer IP header (VTEP → VTEP IP)
• Outer UDP header (src=ephemeral, dst=4789)
• VXLAN header: 8 bytes, including a 24-bit VNI (VXLAN Network Identifier — allows 16 million virtual networks, solving the 4096 VLAN limit)
• Original inner Ethernet frame

VTEP (VXLAN Tunnel Endpoint): typically the hypervisor's virtual switch (Open vSwitch on Linux) or a physical ToR (Top-of-Rack) switch. The VTEP encapsulates outgoing frames and decapsulates incoming packets.

Control plane for VTEP discovery: Early VXLAN used multicast for BUM (Broadcast, Unknown unicast, Multicast) traffic, which doesn't scale. Production deployments use BGP EVPN (Ethernet VPN, RFC 7432): VTEPs advertise their connected MAC/IP bindings as BGP routes, so every VTEP learns where every VM lives without flooding. SDN controllers (like VMware NSX or Juniper Contrail) manage the EVPN control plane.

Network virtualization in cloud: AWS VPC, Azure VNet, and GCP VPC are all implemented using overlay networking (variants of VXLAN or proprietary equivalents). When your EC2 instance in us-east-1 pings another in the same VPC, the traffic is VXLAN-encapsulated by the hypervisor, traverses the fat-tree physical fabric, and is decapsulated at the destination hypervisor — the physical network sees only UDP traffic between hypervisor IPs and has no knowledge of tenant addressing.

# Illustrate VXLAN encapsulation structure using Scapy
# (install: pip install scapy)

from scapy.all import Ether, IP, UDP, VXLAN

def build_vxlan_packet(
    outer_src_ip: str, outer_dst_ip: str,
    inner_src_mac: str, inner_dst_mac: str,
    inner_src_ip: str, inner_dst_ip: str,
    vni: int = 12345
) -> bytes:
    """Build a VXLAN-encapsulated packet between two VMs."""
    inner_frame = (
        Ether(src=inner_src_mac, dst=inner_dst_mac) /
        IP(src=inner_src_ip, dst=inner_dst_ip)
    )
    vxlan_pkt = (
        IP(src=outer_src_ip, dst=outer_dst_ip) /
        UDP(sport=45678, dport=4789) /
        VXLAN(vni=vni, flags='Instance') /
        inner_frame
    )
    raw = bytes(vxlan_pkt)
    print(f"Outer IP: {outer_src_ip} -> {outer_dst_ip}")
    print(f"VNI: {vni}  (supports up to {2**24:,} virtual networks)")
    print(f"Inner IP: {inner_src_ip} -> {inner_dst_ip}")
    print(f"Total encapsulated size: {len(raw)} bytes")
    print(f"Overhead vs raw IP: +{len(raw) - 20} bytes")
    return raw

build_vxlan_packet(
    outer_src_ip="10.0.0.1", outer_dst_ip="10.0.0.2",
    inner_src_mac="aa:bb:cc:dd:ee:01",
    inner_dst_mac="aa:bb:cc:dd:ee:02",
    inner_src_ip="192.168.100.5",
    inner_dst_ip="192.168.100.6",
    vni=42
)

Data-center workloads are strikingly different from WAN or internet workloads. Traffic is highly bursty, latency-sensitive RPCs (mice flows) and enormous bulk transfers (elephant flows) coexist, incast is a dominant pathology, and link utilization must be high (you are paying for the hardware). This has driven a wave of new congestion-control algorithms specifically designed for data centers.

Incast collapse: In MapReduce-style workloads, a worker sends requests to hundreds of storage servers simultaneously. All hundreds respond at the same time, causing a synchronized burst that fills switch buffers and drops packets. Even a single packet drop triggers a TCP retransmission timeout (~200ms), wasting the entire RTO window. For a 1ms target latency, this is catastrophic. Solutions: smaller RTO (1–5ms), larger switch buffers (not always possible), or application-level request spreading.

DCTCP (Data Center TCP, SIGCOMM 2010, RFC 8257): Standard TCP reacts to ECN (Explicit Congestion Notification) marks as a binary signal — mark = congestion. DCTCP makes this proportional: the sender tracks the fraction of ACKs that carry ECN marks (α) and reduces the window by (1 − α/2) instead of halving. This keeps queue occupancy low and stable at ~K packets (a configurable threshold), giving low latency to mice while still achieving high throughput for elephants.

RoCE and RDMA: For storage and ML training workloads, even the CPU overhead of TCP is unacceptable. RDMA over Converged Ethernet (RoCE v2) bypasses the kernel network stack entirely: the NIC DMA-reads directly from application memory and writes received data directly to the destination application's memory, with no CPU involvement. Latency drops from ~50µs to ~2µs. This requires a lossless fabric (Priority Flow Control, PFC) because RDMA has no retransmission mechanism. Managing PFC-induced congestion (deadlock from cyclic buffer dependencies) and RDMA congestion control (DCQCN, which applies ECN to RDMA send rates) is an active research and engineering challenge at every hyperscaler.

RDMA and distributed ML: GPU clusters for training large language models use RDMA (via InfiniBand or RoCE) for All-Reduce collective operations (used in data-parallel training). With 1000 GPUs, a ring All-Reduce requires 2×(N−1)/N bandwidth overhead at each GPU link; with a fat-tree and ECMP, the All-Reduce pattern creates specific traffic matrices that stress certain bisections — a known challenge in cluster networking.

# DCTCP alpha estimation — how the sender tracks ECN fraction
# RFC 8257 Section 3.2

class DCTCPSender:
    def __init__(self, g: float = 0.0625):
        """
        g: EWMA weight for alpha update (RFC 8257 recommends 1/16).
        alpha: estimated fraction of ACKs with ECN marks.
        """
        self.g = g
        self.alpha = 0.0
        self.bytes_acked = 0
        self.bytes_marked = 0  # ECN-marked bytes in current window

    def on_ack(self, acked_bytes: int, ecn_marked: bool):
        self.bytes_acked += acked_bytes
        if ecn_marked:
            self.bytes_marked += acked_bytes

    def end_of_window(self, cwnd: float) -> float:
        """Called once per RTT. Returns new cwnd."""
        # Update alpha: EWMA of ECN fraction
        if self.bytes_acked > 0:
            f = self.bytes_marked / self.bytes_acked
            self.alpha = (1 - self.g) * self.alpha + self.g * f

        # Reduce window proportionally (not halving like standard TCP)
        new_cwnd = cwnd * (1 - self.alpha / 2)

        print(f"  ECN fraction={self.alpha:.3f}, "
              f"cwnd: {cwnd:.0f} -> {new_cwnd:.0f} MSS")

        # Reset per-window counters
        self.bytes_acked = self.bytes_marked = 0
        return max(new_cwnd, 1.0)

sender = DCTCPSender()
# Simulate 3 RTTs with 30% ECN marking
for rtt in range(3):
    for _ in range(100):
        sender.on_ack(1460, ecn_marked=(_ < 30))
    cwnd = sender.end_of_window(cwnd=100 if rtt == 0 else cwnd)